Oct 30 12:42:20.307862 ip-10-0-143-38 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 12:42:20.307876 ip-10-0-143-38 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 12:42:20.307887 ip-10-0-143-38 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 12:42:20.308262 ip-10-0-143-38 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 12:42:30.482085 ip-10-0-143-38 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 12:42:30.482103 ip-10-0-143-38 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 7f0d79fccd4741beb9d63dc3a093680c --
Oct 30 12:44:24.062545 ip-10-0-143-38 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 12:44:24.522172 ip-10-0-143-38 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:44:24.522172 ip-10-0-143-38 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 12:44:24.522172 ip-10-0-143-38 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:44:24.522172 ip-10-0-143-38 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:44:24.527987 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.527852    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:44:24.527987 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.527987    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:44:24.527987 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.527991    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:44:24.527987 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.527995    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:44:24.528115 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.527998    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:44:24.528115 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528002    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:44:24.528115 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528005    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:44:24.528115 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528009    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:44:24.528115 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528012    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:44:24.528115 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528014    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:44:24.528115 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528017    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:44:24.528115 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528020    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:44:24.528115 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528023    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:44:24.528115 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528025    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:44:24.528115 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528028    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:44:24.528115 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528030    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:44:24.528115 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528033    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:44:24.528115 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528036    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:44:24.528115 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528039    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:44:24.528115 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528042    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:44:24.528115 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528044    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:44:24.528115 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528047    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:44:24.528115 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528050    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:44:24.528115 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528053    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:44:24.528594 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528055    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:44:24.528594 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528058    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:44:24.528594 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528060    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:44:24.528594 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528063    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:44:24.528594 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528065    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:44:24.528594 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528069    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:44:24.528594 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528071    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:44:24.528594 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528083    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:44:24.528594 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528086    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:44:24.528594 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528089    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:44:24.528594 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528091    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:44:24.528594 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528094    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:44:24.528594 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528097    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:44:24.528594 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528101    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:44:24.528594 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528104    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:44:24.528594 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528106    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:44:24.528594 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528109    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:44:24.528594 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528112    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:44:24.528594 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528115    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:44:24.529038 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528118    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:44:24.529038 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528121    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:44:24.529038 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528124    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:44:24.529038 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528127    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:44:24.529038 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528130    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:44:24.529038 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528133    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:44:24.529038 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528135    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:44:24.529038 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528140    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:44:24.529038 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528144    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:44:24.529038 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528147    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:44:24.529038 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528151    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:44:24.529038 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528153    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:44:24.529038 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528156    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:44:24.529038 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528159    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:44:24.529038 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528161    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:44:24.529038 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528164    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:44:24.529038 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528169    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:44:24.529038 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528174    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:44:24.529481 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528177    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:44:24.529481 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528180    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:44:24.529481 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528183    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:44:24.529481 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528186    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:44:24.529481 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528189    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:44:24.529481 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528191    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:44:24.529481 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528194    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:44:24.529481 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528197    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:44:24.529481 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528200    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:44:24.529481 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528203    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:44:24.529481 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528205    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:44:24.529481 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528208    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:44:24.529481 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528210    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:44:24.529481 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528213    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:44:24.529481 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528216    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:44:24.529481 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528232    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:44:24.529481 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528235    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:44:24.529481 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528238    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:44:24.529481 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528241    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:44:24.529481 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528243    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:44:24.529958 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528246    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:44:24.529958 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528249    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:44:24.529958 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528252    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:44:24.529958 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528255    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:44:24.529958 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528258    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:44:24.529958 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528261    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:44:24.529958 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528264    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:44:24.529958 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528267    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:44:24.529958 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528269    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:44:24.529958 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528272    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:44:24.529958 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528274    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:44:24.529958 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528277    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:44:24.529958 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528279    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:44:24.529958 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528282    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:44:24.529958 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528284    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:44:24.529958 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528288    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:44:24.529958 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528292    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:44:24.529958 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528295    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:44:24.529958 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528298    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:44:24.529958 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528300    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:44:24.530440 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528304    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:44:24.530440 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528306    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:44:24.530440 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528309    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:44:24.530440 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528312    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:44:24.530440 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.528314    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:44:24.530440 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529139    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:44:24.530440 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529146    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:44:24.530440 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529149    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:44:24.530440 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529152    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:44:24.530440 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529155    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:44:24.530440 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529157    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:44:24.530440 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529160    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:44:24.530440 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529163    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:44:24.530440 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529167    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:44:24.530440 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529170    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:44:24.530440 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529172    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:44:24.530440 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529176    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:44:24.530440 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529179    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:44:24.530440 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529181    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:44:24.530440 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529184    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:44:24.530920 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529187    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:44:24.530920 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529189    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:44:24.530920 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529192    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:44:24.530920 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529194    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:44:24.530920 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529197    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:44:24.530920 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529200    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:44:24.530920 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529202    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:44:24.530920 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529205    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:44:24.530920 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529208    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:44:24.530920 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529211    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:44:24.530920 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529214    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:44:24.530920 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529234    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:44:24.530920 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529238    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:44:24.530920 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529241    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:44:24.530920 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529244    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:44:24.530920 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529247    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:44:24.530920 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529249    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:44:24.530920 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529252    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:44:24.530920 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529254    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:44:24.530920 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529257    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:44:24.531409 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529260    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:44:24.531409 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529263    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:44:24.531409 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529265    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:44:24.531409 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529268    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:44:24.531409 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529271    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:44:24.531409 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529273    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:44:24.531409 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529281    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:44:24.531409 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529284    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:44:24.531409 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529287    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:44:24.531409 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529290    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:44:24.531409 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529292    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:44:24.531409 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529295    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:44:24.531409 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529298    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:44:24.531409 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529300    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:44:24.531409 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529302    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:44:24.531409 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529305    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:44:24.531409 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529308    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:44:24.531409 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529311    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:44:24.531409 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529313    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:44:24.531409 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529316    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:44:24.531891 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529319    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:44:24.531891 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529321    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:44:24.531891 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529326    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:44:24.531891 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529330    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:44:24.531891 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529333    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:44:24.531891 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529336    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:44:24.531891 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529340    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:44:24.531891 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529343    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:44:24.531891 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529345    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:44:24.531891 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529349    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:44:24.531891 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529351    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:44:24.531891 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529354    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:44:24.531891 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529357    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:44:24.531891 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529360    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:44:24.531891 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529363    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:44:24.531891 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529365    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:44:24.531891 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529368    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:44:24.531891 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529371    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:44:24.531891 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529374    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:44:24.532366 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529378    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:44:24.532366 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529380    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:44:24.532366 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529383    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:44:24.532366 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529386    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:44:24.532366 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529388    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:44:24.532366 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529391    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:44:24.532366 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529394    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:44:24.532366 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529396    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:44:24.532366 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529400    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:44:24.532366 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529402    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:44:24.532366 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529405    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:44:24.532366 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529408    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:44:24.532366 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529411    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:44:24.532366 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529414    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:44:24.532366 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529416    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:44:24.532366 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529419    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:44:24.532366 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529421    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:44:24.532366 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529424    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:44:24.532366 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529426    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:44:24.532814 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529429    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:44:24.532814 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529431    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:44:24.532814 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529434    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:44:24.532814 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529437    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:44:24.532814 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529440    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:44:24.532814 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529445    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:44:24.532814 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529448    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:44:24.532814 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529451    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:44:24.532814 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529454    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:44:24.532814 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529457    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:44:24.532814 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529460    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:44:24.532814 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529462    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:44:24.532814 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.529465    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:44:24.532814 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530016    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:44:24.532814 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530023    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:44:24.532814 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530026    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:44:24.532814 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530029    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:44:24.532814 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530032    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:44:24.532814 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530035    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:44:24.532814 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530038    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:44:24.533326 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530041    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:44:24.533326 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530044    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:44:24.533326 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530047    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:44:24.533326 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530051    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:44:24.533326 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530055    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:44:24.533326 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530059    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:44:24.533326 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530062    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:44:24.533326 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530064    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:44:24.533326 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530067    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:44:24.533326 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530070    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:44:24.533326 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530072    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:44:24.533326 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530075    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:44:24.533326 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530078    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:44:24.533326 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530080    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:44:24.533326 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530083    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:44:24.533326 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530085    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:44:24.533326 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530088    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:44:24.533326 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530090    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:44:24.533326 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530093    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:44:24.533771 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530096    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:44:24.533771 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530098    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:44:24.533771 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530101    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:44:24.533771 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530103    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:44:24.533771 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530106    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:44:24.533771 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530108    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:44:24.533771 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530111    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:44:24.533771 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530115    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:44:24.533771 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530117    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:44:24.533771 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530120    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:44:24.533771 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530123    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:44:24.533771 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530125    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:44:24.533771 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530128    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:44:24.533771 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530130    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:44:24.533771 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530133    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:44:24.533771 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530136    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:44:24.533771 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530138    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:44:24.533771 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530141    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:44:24.533771 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530145    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:44:24.533771 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530148    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:44:24.534254 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530150    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:44:24.534254 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530153    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:44:24.534254 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530155    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:44:24.534254 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530158    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:44:24.534254 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530161    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:44:24.534254 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530164    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:44:24.534254 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530166    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:44:24.534254 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530169    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:44:24.534254 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530171    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:44:24.534254 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530173    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:44:24.534254 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530176    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:44:24.534254 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530178    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:44:24.534254 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530181    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:44:24.534254 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530183    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:44:24.534254 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530186    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:44:24.534254 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530190    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:44:24.534254 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530193    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:44:24.534254 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530195    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:44:24.534254 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530199    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:44:24.534763 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530202    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:44:24.534763 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530205    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:44:24.534763 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530208    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:44:24.534763 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530210    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:44:24.534763 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530213    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:44:24.534763 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530230    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:44:24.534763 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530235    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:44:24.534763 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530240    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:44:24.534763 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530244    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:44:24.534763 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530248    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:44:24.534763 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530253    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:44:24.534763 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530258    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:44:24.534763 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530261    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:44:24.534763 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530263    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:44:24.534763 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530266    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:44:24.534763 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530269    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:44:24.534763 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530271    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:44:24.534763 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530274    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:44:24.534763 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530277    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:44:24.534763 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530280    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:44:24.535239 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530283    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:44:24.535239 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530286    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:44:24.535239 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530288    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:44:24.535239 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530291    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:44:24.535239 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530294    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:44:24.535239 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530297    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:44:24.535239 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530299    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:44:24.535239 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530302    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:44:24.535239 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530304    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:44:24.535239 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530307    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:44:24.535239 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530309    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:44:24.535239 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530312    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:44:24.535239 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530315    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:44:24.535239 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530317    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:44:24.535239 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530321    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:44:24.535239 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530324    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:44:24.535239 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530326    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:44:24.535239 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530329    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:44:24.535239 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530331    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:44:24.535681 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530334    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:44:24.535681 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.530336    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:44:24.535681 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530447    2580 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 12:44:24.535681 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530455    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 12:44:24.535681 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530462    2580 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 12:44:24.535681 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530468    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 12:44:24.535681 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530472    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 12:44:24.535681 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530476    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 12:44:24.535681 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530481    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 12:44:24.535681 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530485    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 12:44:24.535681 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530489    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 12:44:24.535681 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530492    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 12:44:24.535681 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530496    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 12:44:24.535681 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530499    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 12:44:24.535681 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530503    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 12:44:24.535681 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530506    2580 flags.go:64] FLAG: --cgroup-root=""
Oct 30 12:44:24.535681 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530509    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 12:44:24.535681 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530512    2580 flags.go:64] FLAG: --client-ca-file=""
Oct 30 12:44:24.535681 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530515    2580 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 12:44:24.535681 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530518    2580 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 12:44:24.535681 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530522    2580 flags.go:64] FLAG: --cluster-domain=""
Oct 30 12:44:24.535681 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530525    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 12:44:24.535681 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530528    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 12:44:24.536349 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530532    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 12:44:24.536349 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530535    2580 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 12:44:24.536349 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530540    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 12:44:24.536349 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530543    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 12:44:24.536349 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530546    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 12:44:24.536349 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530551    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 12:44:24.536349 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530555    2580 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 12:44:24.536349 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530558    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 12:44:24.536349 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530563    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 12:44:24.536349 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530567    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 12:44:24.536349 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530570    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 12:44:24.536349 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530575    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 12:44:24.536349 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530578    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 12:44:24.536349 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530581    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 12:44:24.536349 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530592    2580 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 12:44:24.536349 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530597    2580 flags.go:64] FLAG: --enable-server="true"
Oct 30 12:44:24.536349 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530601    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 12:44:24.536349 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530607    2580 flags.go:64] FLAG: --event-burst="100"
Oct 30 12:44:24.536349 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530610    2580 flags.go:64] FLAG: --event-qps="50"
Oct 30 12:44:24.536349 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530613    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 12:44:24.536349 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530616    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 12:44:24.536349 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530619    2580 flags.go:64] FLAG: --eviction-hard=""
Oct 30 12:44:24.536349 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530624    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 12:44:24.536349 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530627    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 12:44:24.536940 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530630    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 12:44:24.536940 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530634    2580 flags.go:64] FLAG: --eviction-soft=""
Oct 30 12:44:24.536940 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530637    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 12:44:24.536940 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530640    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 12:44:24.536940 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530643    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 12:44:24.536940 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530646    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 12:44:24.536940 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530649    2580 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 12:44:24.536940 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530652    2580 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 12:44:24.536940 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530655    2580 flags.go:64] FLAG: --feature-gates=""
Oct 30 12:44:24.536940 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530662    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 12:44:24.536940 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530665    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 12:44:24.536940 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530668    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 12:44:24.536940 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530672    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 12:44:24.536940 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530675    2580 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 12:44:24.536940 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530680    2580 flags.go:64] FLAG: --help="false"
Oct 30 12:44:24.536940 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530683    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-143-38.ec2.internal"
Oct 30 12:44:24.536940 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530688    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 12:44:24.536940 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530691    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 12:44:24.536940 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530694    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 12:44:24.536940 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530698    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 12:44:24.536940 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530702    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 12:44:24.536940 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530705    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 12:44:24.536940 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530708    2580 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 12:44:24.538525 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530711    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 12:44:24.538525 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530714    2580 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 12:44:24.538525 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530717    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 12:44:24.538525 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530721    2580 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 12:44:24.538525 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530724    2580 flags.go:64] FLAG: --kube-reserved=""
Oct 30 12:44:24.538525 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530727    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 12:44:24.538525 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530730    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 12:44:24.538525 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530733    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 12:44:24.538525 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530736    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 12:44:24.538525 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530739    2580 flags.go:64] FLAG: --lock-file=""
Oct 30 12:44:24.538525 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530743    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 12:44:24.538525 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530746    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 12:44:24.538525 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530749    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 12:44:24.538525 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530754    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 12:44:24.538525 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530757    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 12:44:24.538525 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530760    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 12:44:24.538525 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530764    2580 flags.go:64] FLAG: --logging-format="text"
Oct 30 12:44:24.538525 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530767    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 12:44:24.538525 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530771    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 12:44:24.538525 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530774    2580 flags.go:64] FLAG: --manifest-url=""
Oct 30 12:44:24.538525 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530777    2580 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 12:44:24.538525 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530782    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 12:44:24.538525 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530786    2580 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 12:44:24.538525 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530790    2580 flags.go:64] FLAG: --max-pods="110"
Oct 30 12:44:24.538525 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530794    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 12:44:24.539135 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530799    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 12:44:24.539135 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530802    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 12:44:24.539135 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530805    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 12:44:24.539135 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530808    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 12:44:24.539135 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530811    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 12:44:24.539135 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530814    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 12:44:24.539135 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530823    2580 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 12:44:24.539135 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530826    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 12:44:24.539135 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530829    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 12:44:24.539135 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530832    2580 flags.go:64] FLAG: --pod-cidr=""
Oct 30 12:44:24.539135 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530835    2580 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 12:44:24.539135 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530838    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 12:44:24.539135 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530842    2580 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 12:44:24.539135 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530845    2580 flags.go:64] FLAG: --port="10250"
Oct 30 12:44:24.539135 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530848    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 12:44:24.539135 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530851    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01f1782ea84ad690c"
Oct 30 12:44:24.539135 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530854    2580 flags.go:64] FLAG: --qos-reserved=""
Oct 30 12:44:24.539135 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530858    2580 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 12:44:24.539135 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530861    2580 flags.go:64] FLAG: --register-node="true"
Oct 30 12:44:24.539135 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530864    2580 flags.go:64] FLAG: --register-with-taints=""
Oct 30 12:44:24.539135 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530868    2580 flags.go:64] FLAG: --registry-burst="10"
Oct 30 12:44:24.539135 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530871    2580 flags.go:64] FLAG: --registry-qps="5"
Oct 30 12:44:24.539135 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530874    2580 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 12:44:24.539135 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530877    2580 flags.go:64] FLAG: --reserved-memory=""
Oct 30 12:44:24.539135 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530881    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 12:44:24.539135 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530886    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 12:44:24.539763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530889    2580 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 12:44:24.539763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530892    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 12:44:24.539763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530895    2580 flags.go:64] FLAG: --runonce="false"
Oct 30 12:44:24.539763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530898    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 12:44:24.539763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530901    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 12:44:24.539763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530904    2580 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 12:44:24.539763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530909    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 12:44:24.539763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530912    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 12:44:24.539763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530916    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 12:44:24.539763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530919    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 12:44:24.539763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530922    2580 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 12:44:24.539763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530926    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 12:44:24.539763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530929    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 12:44:24.539763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530932    2580 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 12:44:24.539763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530936    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 12:44:24.539763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530939    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 12:44:24.539763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530942    2580 flags.go:64] FLAG: --system-cgroups=""
Oct 30 12:44:24.539763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530945    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 12:44:24.539763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530951    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 12:44:24.539763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530954    2580 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 12:44:24.539763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530957    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 12:44:24.539763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530961    2580 flags.go:64] FLAG: --tls-min-version=""
Oct 30 12:44:24.539763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530964    2580 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 12:44:24.539763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530967    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 12:44:24.539763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530971    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 12:44:24.540414 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530974    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 12:44:24.540414 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530977    2580 flags.go:64] FLAG: --v="2"
Oct 30 12:44:24.540414 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530981    2580 flags.go:64] FLAG: --version="false"
Oct 30 12:44:24.540414 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530986    2580 flags.go:64] FLAG: --vmodule=""
Oct 30 12:44:24.540414 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530990    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 12:44:24.540414 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.530993    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 12:44:24.540414 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531136    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:44:24.540414 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531140    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:44:24.540414 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531144    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:44:24.540414 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531147    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:44:24.540414 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531150    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:44:24.540414 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531152    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:44:24.540414 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531155    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:44:24.540414 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531159    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:44:24.540414 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531163    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:44:24.540414 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531166    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:44:24.540414 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531168    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:44:24.540414 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531171    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:44:24.540414 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531174    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:44:24.540414 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531177    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:44:24.540414 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531179    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:44:24.540943 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531182    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:44:24.540943 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531185    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:44:24.540943 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531187    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:44:24.540943 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531190    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:44:24.540943 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531193    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:44:24.540943 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531195    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:44:24.540943 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531199    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:44:24.540943 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531201    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:44:24.540943 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531204    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:44:24.540943 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531207    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:44:24.540943 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531209    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:44:24.540943 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531213    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:44:24.540943 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531231    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:44:24.540943 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531237    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:44:24.540943 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531241    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:44:24.540943 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531246    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:44:24.540943 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531250    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:44:24.540943 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531254    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:44:24.540943 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531257    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:44:24.540943 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531259    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:44:24.541427 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531262    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:44:24.541427 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531265    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:44:24.541427 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531268    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:44:24.541427 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531270    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:44:24.541427 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531275    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:44:24.541427 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531278    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:44:24.541427 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531281    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:44:24.541427 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531284    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:44:24.541427 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531287    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:44:24.541427 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531289    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:44:24.541427 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531292    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:44:24.541427 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531295    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:44:24.541427 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531299    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:44:24.541427 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531303    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:44:24.541427 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531306    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:44:24.541427 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531309    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:44:24.541427 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531312    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:44:24.541427 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531314    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:44:24.541427 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531317    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:44:24.541887 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531319    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:44:24.541887 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531322    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:44:24.541887 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531324    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:44:24.541887 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531327    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:44:24.541887 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531330    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:44:24.541887 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531333    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:44:24.541887 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531336    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:44:24.541887 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531338    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:44:24.541887 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531341    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:44:24.541887 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531343    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:44:24.541887 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531347    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:44:24.541887 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531350    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:44:24.541887 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531352    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:44:24.541887 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531355    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:44:24.541887 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531358    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:44:24.541887 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531360    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:44:24.541887 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531363    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:44:24.541887 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531367    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:44:24.541887 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531370    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:44:24.541887 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531372    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:44:24.541887 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531376    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:44:24.542410 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531379    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:44:24.542410 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531381    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:44:24.542410 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531384    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:44:24.542410 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531387    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:44:24.542410 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531389    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:44:24.542410 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531393    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:44:24.542410 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531395    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:44:24.542410 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531398    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:44:24.542410 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531400    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:44:24.542410 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531403    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:44:24.542410 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531406    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:44:24.542410 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531408    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:44:24.542410 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531411    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:44:24.542410 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531414    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:44:24.542410 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531417    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:44:24.542410 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531419    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:44:24.542410 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531422    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:44:24.542410 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531425    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:44:24.542410 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531428    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:44:24.542874 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531430    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:44:24.542874 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531433    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:44:24.542874 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531436    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:44:24.542874 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531439    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:44:24.542874 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531442    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:44:24.542874 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531444    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:44:24.542874 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531446    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:44:24.542874 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531449    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:44:24.542874 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531452    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:44:24.542874 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531455    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:44:24.542874 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531458    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:44:24.542874 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531461    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:44:24.542874 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.531488    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:44:24.542874 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531532    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:44:24.542874 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531537    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:44:24.542874 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531541    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:44:24.543270 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531544    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:44:24.543270 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531547    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:44:24.543270 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531550    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:44:24.543270 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531552    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:44:24.543270 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531555    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:44:24.543270 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531557    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:44:24.543270 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531560    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:44:24.543270 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531563    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:44:24.543270 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531565    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:44:24.543270 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531568    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:44:24.543270 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531571    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:44:24.543270 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531573    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:44:24.543270 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531576    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:44:24.543270 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531579    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:44:24.543270 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531581    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:44:24.543270 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531584    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:44:24.543270 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531586    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:44:24.543270 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531589    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:44:24.543270 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531594    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:44:24.543270 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531596    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:44:24.543270 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531599    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:44:24.543760 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531602    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:44:24.543760 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531605    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:44:24.543760 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531607    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:44:24.543760 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531610    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:44:24.543760 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531612    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:44:24.543760 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531615    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:44:24.543760 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531618    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:44:24.543760 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531620    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:44:24.543760 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531623    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:44:24.543760 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531626    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:44:24.543760 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531629    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:44:24.543760 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531632    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:44:24.543760 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531635    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:44:24.543760 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531637    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:44:24.543760 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531640    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:44:24.543760 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531643    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:44:24.543760 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531645    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:44:24.543760 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531648    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:44:24.543760 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531651    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:44:24.544213 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531653    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:44:24.544213 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531656    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:44:24.544213 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531659    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:44:24.544213 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531661    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:44:24.544213 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531664    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:44:24.544213 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531666    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:44:24.544213 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531669    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:44:24.544213 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531671    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:44:24.544213 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531674    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:44:24.544213 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531676    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:44:24.544213 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531681    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:44:24.544213 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531683    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:44:24.544213 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531686    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:44:24.544213 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531688    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:44:24.544213 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531691    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:44:24.544213 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531694    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:44:24.544213 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531696    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:44:24.544213 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531699    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:44:24.544213 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531702    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:44:24.544689 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531705    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:44:24.544689 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531707    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:44:24.544689 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531710    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:44:24.544689 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531712    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:44:24.544689 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531716    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:44:24.544689 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531719    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:44:24.544689 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531721    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:44:24.544689 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531724    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:44:24.544689 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531726    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:44:24.544689 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531729    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:44:24.544689 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531731    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:44:24.544689 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531734    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:44:24.544689 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531737    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:44:24.544689 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531739    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:44:24.544689 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531742    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:44:24.544689 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531744    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:44:24.544689 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531747    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:44:24.544689 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531749    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:44:24.544689 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531752    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:44:24.545130 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531754    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:44:24.545130 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531757    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:44:24.545130 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531759    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:44:24.545130 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531763    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:44:24.545130 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531769    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:44:24.545130 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531772    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:44:24.545130 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531775    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:44:24.545130 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531778    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:44:24.545130 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531780    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:44:24.545130 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531783    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:44:24.545130 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531786    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:44:24.545130 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531789    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:44:24.545130 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531792    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:44:24.545130 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531794    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:44:24.545130 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531797    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:44:24.545130 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531799    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:44:24.545130 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531802    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:44:24.545130 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531805    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:44:24.545130 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531808    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:44:24.545592 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531810    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:44:24.545592 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531813    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:44:24.545592 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531815    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:44:24.545592 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531818    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:44:24.545592 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531820    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:44:24.545592 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.531826    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:44:24.545592 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.539437    2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 12:44:24.545592 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.539462    2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 12:44:24.545592 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.539477    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 12:44:24.545592 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.539483    2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 12:44:24.545592 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539546    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:44:24.545592 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539552    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:44:24.545592 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539556    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:44:24.545592 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539559    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:44:24.545592 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539562    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:44:24.545592 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539565    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:44:24.545592 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539568    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:44:24.545592 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539571    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:44:24.546062 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539573    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:44:24.546062 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539577    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:44:24.546062 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539579    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:44:24.546062 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539582    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:44:24.546062 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539584    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:44:24.546062 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539587    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:44:24.546062 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539590    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:44:24.546062 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539592    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:44:24.546062 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539595    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:44:24.546062 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539597    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:44:24.546062 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539600    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:44:24.546062 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539603    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:44:24.546062 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539605    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:44:24.546062 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539608    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:44:24.546062 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539611    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:44:24.546062 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539613    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:44:24.546062 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539616    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:44:24.546062 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539618    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:44:24.546062 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539621    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:44:24.546062 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539624    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:44:24.546544 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539626    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:44:24.546544 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539639    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:44:24.546544 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539642    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:44:24.546544 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539645    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:44:24.546544 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539648    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:44:24.546544 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539650    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:44:24.546544 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539653    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:44:24.546544 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539656    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:44:24.546544 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539658    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:44:24.546544 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539661    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:44:24.546544 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539663    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:44:24.546544 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539666    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:44:24.546544 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539669    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:44:24.546544 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539671    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:44:24.546544 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539674    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:44:24.546544 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539677    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:44:24.546544 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539679    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:44:24.546544 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539682    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:44:24.546544 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539685    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:44:24.546544 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539688    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:44:24.547020 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539691    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:44:24.547020 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539694    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:44:24.547020 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539696    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:44:24.547020 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539699    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:44:24.547020 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539702    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:44:24.547020 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539704    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:44:24.547020 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539707    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:44:24.547020 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539710    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:44:24.547020 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539713    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:44:24.547020 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539715    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:44:24.547020 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539718    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:44:24.547020 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539720    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:44:24.547020 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539723    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:44:24.547020 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539726    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:44:24.547020 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539743    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:44:24.547020 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539749    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:44:24.547020 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539752    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:44:24.547020 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539754    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:44:24.547020 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539757    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:44:24.547020 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539760    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:44:24.547634 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539762    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:44:24.547634 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539765    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:44:24.547634 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539768    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:44:24.547634 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539770    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:44:24.547634 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539774    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:44:24.547634 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539776    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:44:24.547634 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539779    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:44:24.547634 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539782    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:44:24.547634 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539785    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:44:24.547634 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539788    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:44:24.547634 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539791    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:44:24.547634 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539793    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:44:24.547634 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539796    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:44:24.547634 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539798    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:44:24.547634 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539801    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:44:24.547634 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539804    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:44:24.547634 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539807    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:44:24.547634 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539810    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:44:24.547634 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539814    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:44:24.548439 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539818    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:44:24.548439 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539821    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:44:24.548439 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539824    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:44:24.548439 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539827    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:44:24.548439 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539830    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:44:24.548439 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539832    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:44:24.548439 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539835    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:44:24.548439 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539838    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:44:24.548439 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539846    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:44:24.548439 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539850    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:44:24.548439 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539853    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:44:24.548439 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539855    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:44:24.548439 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539858    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:44:24.548439 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539861    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:44:24.548439 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539863    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:44:24.548439 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539866    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:44:24.548439 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539869    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:44:24.548439 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539871    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:44:24.548439 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.539874    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:44:24.549162 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.539906    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:44:24.549162 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540117    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:44:24.549162 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540122    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:44:24.549162 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540127    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:44:24.549162 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540131    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:44:24.549162 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540134    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:44:24.549162 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540138    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:44:24.549162 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540141    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:44:24.549162 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540143    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:44:24.549162 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540146    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:44:24.549162 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540149    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:44:24.549162 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540152    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:44:24.549162 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540154    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:44:24.549162 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540157    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:44:24.549162 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540160    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:44:24.549162 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540162    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:44:24.549722 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540165    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:44:24.549722 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540168    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:44:24.549722 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540170    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:44:24.549722 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540173    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:44:24.549722 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540176    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:44:24.549722 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540179    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:44:24.549722 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540188    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:44:24.549722 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540191    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:44:24.549722 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540194    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:44:24.549722 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540196    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:44:24.549722 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540199    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:44:24.549722 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540202    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:44:24.549722 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540205    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:44:24.549722 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540207    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:44:24.549722 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540210    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:44:24.549722 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540213    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:44:24.549722 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540215    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:44:24.549722 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540233    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:44:24.549722 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540237    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:44:24.550232 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540242    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:44:24.550232 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540247    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:44:24.550232 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540249    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:44:24.550232 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540252    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:44:24.550232 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540255    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:44:24.550232 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540257    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:44:24.550232 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540260    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:44:24.550232 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540263    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:44:24.550232 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540265    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:44:24.550232 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540268    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:44:24.550232 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540270    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:44:24.550232 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540273    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:44:24.550232 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540275    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:44:24.550232 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540278    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:44:24.550232 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540280    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:44:24.550232 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540283    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:44:24.550232 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540286    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:44:24.550232 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540289    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:44:24.550232 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540292    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:44:24.550232 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540294    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:44:24.550746 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540307    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:44:24.550746 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540311    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:44:24.550746 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540313    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:44:24.550746 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540316    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:44:24.550746 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540319    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:44:24.550746 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540321    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:44:24.550746 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540324    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:44:24.550746 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540326    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:44:24.550746 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540329    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:44:24.550746 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540332    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:44:24.550746 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540334    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:44:24.550746 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540337    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:44:24.550746 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540340    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:44:24.550746 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540342    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:44:24.550746 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540345    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:44:24.550746 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540347    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:44:24.550746 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540350    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:44:24.550746 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540353    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:44:24.550746 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540355    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:44:24.551277 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540358    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:44:24.551277 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540360    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:44:24.551277 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540363    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:44:24.551277 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540372    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:44:24.551277 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540375    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:44:24.551277 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540377    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:44:24.551277 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540380    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:44:24.551277 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540382    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:44:24.551277 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540385    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:44:24.551277 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540388    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:44:24.551277 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540391    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:44:24.551277 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540393    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:44:24.551277 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540396    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:44:24.551277 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540399    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:44:24.551277 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540407    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:44:24.551277 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540411    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:44:24.551277 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540413    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:44:24.551277 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540416    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:44:24.551277 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540421    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:44:24.551749 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540424    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:44:24.551749 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540427    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:44:24.551749 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540430    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:44:24.551749 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540432    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:44:24.551749 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540435    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:44:24.551749 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540437    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:44:24.551749 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540440    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:44:24.551749 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540443    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:44:24.551749 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540445    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:44:24.551749 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540448    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:44:24.551749 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540450    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:44:24.551749 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540453    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:44:24.551749 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540456    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:44:24.551749 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540459    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:44:24.551749 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.540483    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:44:24.551749 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540522    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:44:24.552134 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540526    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:44:24.552134 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540529    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:44:24.552134 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540532    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:44:24.552134 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540534    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:44:24.552134 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540537    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:44:24.552134 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540540    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:44:24.552134 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540542    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:44:24.552134 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540545    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:44:24.552134 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540548    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:44:24.552134 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540551    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:44:24.552134 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540554    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:44:24.552134 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540556    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:44:24.552134 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540565    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:44:24.552134 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540568    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:44:24.552134 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540572    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:44:24.552134 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540574    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:44:24.552134 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540577    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:44:24.552134 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540579    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:44:24.552134 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540582    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:44:24.552603 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540584    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:44:24.552603 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540587    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:44:24.552603 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540589    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:44:24.552603 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540592    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:44:24.552603 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540595    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:44:24.552603 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540597    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:44:24.552603 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540601    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:44:24.552603 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540605    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:44:24.552603 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540608    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:44:24.552603 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540610    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:44:24.552603 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540613    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:44:24.552603 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540615    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:44:24.552603 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540618    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:44:24.552603 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540621    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:44:24.552603 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540624    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:44:24.552603 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540626    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:44:24.552603 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540629    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:44:24.552603 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540631    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:44:24.552603 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540634    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:44:24.552603 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540637    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:44:24.553077 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540640    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:44:24.553077 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540642    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:44:24.553077 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540645    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:44:24.553077 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540650    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:44:24.553077 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540653    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:44:24.553077 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540656    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:44:24.553077 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540668    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:44:24.553077 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540671    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:44:24.553077 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540674    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:44:24.553077 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540676    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:44:24.553077 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540679    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:44:24.553077 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540681    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:44:24.553077 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540684    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:44:24.553077 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540687    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:44:24.553077 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540690    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:44:24.553077 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540692    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:44:24.553077 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540695    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:44:24.553077 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540697    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:44:24.553077 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540700    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:44:24.553581 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540702    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:44:24.553581 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540705    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:44:24.553581 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540707    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:44:24.553581 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540710    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:44:24.553581 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540712    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:44:24.553581 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540715    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:44:24.553581 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540718    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:44:24.553581 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540720    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:44:24.553581 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540723    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:44:24.553581 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540726    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:44:24.553581 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540728    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:44:24.553581 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540731    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:44:24.553581 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540734    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:44:24.553581 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540736    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:44:24.553581 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540739    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:44:24.553581 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540742    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:44:24.553581 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540744    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:44:24.553581 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540747    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:44:24.553581 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540750    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:44:24.553581 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540752    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:44:24.554069 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540765    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:44:24.554069 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540768    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:44:24.554069 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540771    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:44:24.554069 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540773    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:44:24.554069 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540776    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:44:24.554069 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540779    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:44:24.554069 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540781    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:44:24.554069 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540784    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:44:24.554069 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540786    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:44:24.554069 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540789    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:44:24.554069 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540791    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:44:24.554069 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540794    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:44:24.554069 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540797    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:44:24.554069 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540799    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:44:24.554069 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540802    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:44:24.554069 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540804    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:44:24.554069 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540807    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:44:24.554069 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540809    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:44:24.554069 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540812    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:44:24.554541 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540814    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:44:24.554541 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540817    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:44:24.554541 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540823    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:44:24.554541 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540826    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:44:24.554541 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540830    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:44:24.554541 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540832    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:44:24.554541 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540835    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:44:24.554541 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:24.540838    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:44:24.554541 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.540929    2580 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 12:44:24.554541 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.543407    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 12:44:24.554541 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.544583    2580 server.go:1014] "Starting client certificate rotation"
Oct 30 12:44:24.554541 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.544692    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 12:44:24.554541 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.544756    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 12:44:24.554541 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.548376    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 12:44:24.554541 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.550552    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 12:44:24.560360 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.560335    2580 log.go:25] "Validated CRI v1 runtime API"
Oct 30 12:44:24.564588 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.564426    2580 log.go:25] "Validated CRI v1 image API"
Oct 30 12:44:24.565483 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.565458    2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 12:44:24.568349 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.568324    2580 fs.go:135] Filesystem UUIDs: map[207b088b-cfd2-47c4-a126-f0a601e2084d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e259b42c-307e-4cbc-84c2-9dbd799b6da3:/dev/nvme0n1p4]
Oct 30 12:44:24.568439 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.568347    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 12:44:24.573879 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.573730    2580 manager.go:217] Machine: {Timestamp:2026-10-30 12:44:24.572152033 +0000 UTC m=+0.377514692 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097904 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2eceb28fd290b835c4179577a50dd6 SystemUUID:ec2eceb2-8fd2-90b8-35c4-179577a50dd6 BootID:7f0d79fc-cd47-41be-b9d6-3dc3a093680c Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f9:4e:d3:89 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f9:4e:d3:89 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:2a:9d:8f:80:3e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 12:44:24.573879 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.573873    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 12:44:24.574039 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.574027    2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 12:44:24.575439 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.575403    2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 12:44:24.575609 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.575442    2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-38.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 12:44:24.575657 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.575615    2580 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 12:44:24.575657 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.575623    2580 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 12:44:24.575707 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.575659    2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 12:44:24.575707 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.575672    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 12:44:24.576968 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.576954    2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 12:44:24.581608 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.581589    2580 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 12:44:24.581761 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.581751    2580 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 12:44:24.582613 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.582602    2580 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 12:44:24.582649 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.582620    2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 12:44:24.582649 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.582632    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 12:44:24.582725 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.582657    2580 kubelet.go:405] "Adding apiserver pod source"
Oct 30 12:44:24.582725 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.582667    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 12:44:24.583080 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.583067    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 12:44:24.585901 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.585870    2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 12:44:24.595757 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.595730    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 12:44:24.596782 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.596767    2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 12:44:24.596847 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.596799    2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 12:44:24.596968 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.596957    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 12:44:24.596998 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.596977    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 12:44:24.596998 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.596984    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 12:44:24.596998 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.596990    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 12:44:24.596998 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.596995    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 12:44:24.597096 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.597001    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 12:44:24.597816 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.597807    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 12:44:24.597816 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.597817    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 12:44:24.597873 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.597825    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 12:44:24.597873 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.597831    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 12:44:24.597873 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.597840    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 12:44:24.597873 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.597850    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 12:44:24.599070 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.599059    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 12:44:24.599106 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.599072    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 12:44:24.603469 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.603446    2580 server.go:1263] "Started kubelet"
Oct 30 12:44:24.603625 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.603558    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 12:44:24.603681 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.603652    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 12:44:24.603839 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.603796    2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 12:44:24.604543 ip-10-0-143-38 systemd[1]: Started Kubernetes Kubelet.
Oct 30 12:44:24.605267 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.605202    2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 12:44:24.605896 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.605883    2580 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 12:44:24.611060 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:24.611035    2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 12:44:24.612890 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.612875    2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 12:44:24.612971 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.612951    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 12:44:24.613137 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:24.613103    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-38.ec2.internal\" not found"
Oct 30 12:44:24.613922 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.613900    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 12:44:24.614010 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.613986    2580 factory.go:55] Registering systemd factory
Oct 30 12:44:24.614010 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.614002    2580 factory.go:223] Registration of the systemd container factory successfully
Oct 30 12:44:24.614099 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.614080    2580 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 12:44:24.614099 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.614095    2580 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 12:44:24.614191 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.614165    2580 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 12:44:24.614191 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.614174    2580 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 12:44:24.614299 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.614254    2580 factory.go:153] Registering CRI-O factory
Oct 30 12:44:24.614299 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.614267    2580 factory.go:223] Registration of the crio container factory successfully
Oct 30 12:44:24.614387 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.614327    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 12:44:24.614387 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.614346    2580 factory.go:103] Registering Raw factory
Oct 30 12:44:24.614387 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.614360    2580 manager.go:1199] Started watching for new ooms in manager
Oct 30 12:44:24.615553 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.615536    2580 manager.go:319] Starting recovery of all containers
Oct 30 12:44:24.629774 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.629752    2580 manager.go:324] Recovery completed
Oct 30 12:44:24.633957 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.633933    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:44:24.635965 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.635942    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:44:24.636053 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.635985    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:44:24.636053 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.636000    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:44:24.636591 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.636578    2580 cpu_manager.go:226] "Starting" policy="none"
Oct 30 12:44:24.636591 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.636590    2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 12:44:24.636714 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.636583    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:44:24.636714 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.636611    2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 12:44:24.637432 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.637412    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:44:24.637517 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.637441    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:44:24.637517 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.637454    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:44:24.637604 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:24.637567    2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-38.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 12:44:24.637655 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.637603    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-38.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 12:44:24.638689 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:24.637563    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-38.ec2.internal.18e34f4c0890e628  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-38.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-38.ec2.internal,},FirstTimestamp:2026-10-30 12:44:24.603412008 +0000 UTC m=+0.408774667,LastTimestamp:2026-10-30 12:44:24.603412008 +0000 UTC m=+0.408774667,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-38.ec2.internal,}"
Oct 30 12:44:24.638896 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.638885    2580 policy_none.go:50] "Start"
Oct 30 12:44:24.638951 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.638905    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 12:44:24.638951 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.638916    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 12:44:24.640895 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.640883    2580 policy_none.go:44] "Start"
Oct 30 12:44:24.652516 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:24.652482    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-38.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 12:44:24.652666 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:24.652525    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 12:44:24.652726 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:24.652657    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-38.ec2.internal.18e34f4c0a8199bc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-38.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-38.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-38.ec2.internal,},FirstTimestamp:2026-10-30 12:44:24.635963836 +0000 UTC m=+0.441326503,LastTimestamp:2026-10-30 12:44:24.635963836 +0000 UTC m=+0.441326503,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-38.ec2.internal,}"
Oct 30 12:44:24.655637 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.655608    2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 12:44:24.655744 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.655721    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-m4cmd"
Oct 30 12:44:24.666575 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.666554    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-m4cmd"
Oct 30 12:44:24.682045 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.682003    2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 12:44:24.686700 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.686673    2580 manager.go:341] "Starting Device Plugin manager"
Oct 30 12:44:24.686791 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:24.686723    2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 12:44:24.686791 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.686739    2580 server.go:87] "Starting device plugin registration server"
Oct 30 12:44:24.686903 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:24.686872    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 12:44:24.687005 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:24.686917    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-38.ec2.internal.18e34f4c0a820805  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-38.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-38.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-38.ec2.internal,},FirstTimestamp:2026-10-30 12:44:24.635992069 +0000 UTC m=+0.441354729,LastTimestamp:2026-10-30 12:44:24.635992069 +0000 UTC m=+0.441354729,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-38.ec2.internal,}"
Oct 30 12:44:24.687140 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.687126    2580 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 12:44:24.687180 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.687144    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 12:44:24.687302 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.687278    2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 12:44:24.687389 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.687377    2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 12:44:24.687389 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.687388    2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 12:44:24.687916 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:24.687892    2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 12:44:24.687982 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:24.687949    2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-38.ec2.internal\" not found"
Oct 30 12:44:24.751963 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.751914    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 12:44:24.754107 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.753174    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 12:44:24.754107 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.753203    2580 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 12:44:24.754107 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.753243    2580 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 12:44:24.754107 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:24.753285    2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 12:44:24.768189 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.768159    2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:44:24.788340 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.788236    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:44:24.789267 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.789242    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:44:24.789402 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.789283    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:44:24.789402 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.789297    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:44:24.789402 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.789333    2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-38.ec2.internal"
Oct 30 12:44:24.807416 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.807385    2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-38.ec2.internal"
Oct 30 12:44:24.807416 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:24.807416    2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-38.ec2.internal\": node \"ip-10-0-143-38.ec2.internal\" not found"
Oct 30 12:44:24.849267 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:24.849211    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-38.ec2.internal\" not found"
Oct 30 12:44:24.854296 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.854272    2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-38.ec2.internal"]
Oct 30 12:44:24.854372 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.854341    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:44:24.855351 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.855336    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:44:24.855421 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.855365    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:44:24.855421 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.855377    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:44:24.857672 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.857660    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:44:24.857813 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.857797    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal"
Oct 30 12:44:24.857865 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.857834    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:44:24.858498 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.858480    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:44:24.858596 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.858502    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:44:24.858596 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.858514    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:44:24.858596 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.858520    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:44:24.858596 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.858541    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:44:24.858596 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.858551    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:44:24.860696 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.860678    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-38.ec2.internal"
Oct 30 12:44:24.860765 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.860713    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:44:24.861552 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.861536    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:44:24.861617 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.861564    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:44:24.861617 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.861573    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:44:24.896375 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:24.896354    2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-38.ec2.internal\" not found" node="ip-10-0-143-38.ec2.internal"
Oct 30 12:44:24.899451 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:24.899430    2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-38.ec2.internal\" not found" node="ip-10-0-143-38.ec2.internal"
Oct 30 12:44:24.916441 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.916410    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6d2044c13bd181c94b6dd5acd1654d5c-config\") pod \"kube-apiserver-proxy-ip-10-0-143-38.ec2.internal\" (UID: \"6d2044c13bd181c94b6dd5acd1654d5c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-38.ec2.internal"
Oct 30 12:44:24.916533 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.916445    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/36805beca2c15cc87891009bcced9111-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal\" (UID: \"36805beca2c15cc87891009bcced9111\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal"
Oct 30 12:44:24.916533 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:24.916472    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36805beca2c15cc87891009bcced9111-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal\" (UID: \"36805beca2c15cc87891009bcced9111\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal"
Oct 30 12:44:24.949611 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:24.949576    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-38.ec2.internal\" not found"
Oct 30 12:44:25.017165 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:25.017126    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6d2044c13bd181c94b6dd5acd1654d5c-config\") pod \"kube-apiserver-proxy-ip-10-0-143-38.ec2.internal\" (UID: \"6d2044c13bd181c94b6dd5acd1654d5c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-38.ec2.internal"
Oct 30 12:44:25.017165 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:25.017165    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/36805beca2c15cc87891009bcced9111-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal\" (UID: \"36805beca2c15cc87891009bcced9111\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal"
Oct 30 12:44:25.017358 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:25.017183    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36805beca2c15cc87891009bcced9111-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal\" (UID: \"36805beca2c15cc87891009bcced9111\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal"
Oct 30 12:44:25.017358 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:25.017254    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36805beca2c15cc87891009bcced9111-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal\" (UID: \"36805beca2c15cc87891009bcced9111\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal"
Oct 30 12:44:25.017358 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:25.017258    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/36805beca2c15cc87891009bcced9111-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal\" (UID: \"36805beca2c15cc87891009bcced9111\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal"
Oct 30 12:44:25.017358 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:25.017259    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6d2044c13bd181c94b6dd5acd1654d5c-config\") pod \"kube-apiserver-proxy-ip-10-0-143-38.ec2.internal\" (UID: \"6d2044c13bd181c94b6dd5acd1654d5c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-38.ec2.internal"
Oct 30 12:44:25.049784 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:25.049692    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-38.ec2.internal\" not found"
Oct 30 12:44:25.149871 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:25.149816    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-38.ec2.internal\" not found"
Oct 30 12:44:25.200036 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:25.200006    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal"
Oct 30 12:44:25.203175 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:25.203155    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-38.ec2.internal"
Oct 30 12:44:25.250764 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:25.250715    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-38.ec2.internal\" not found"
Oct 30 12:44:25.353479 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:25.353400    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-38.ec2.internal\" not found"
Oct 30 12:44:25.453783 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:25.453732    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-38.ec2.internal\" not found"
Oct 30 12:44:25.543893 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:25.543856    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 12:44:25.544424 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:25.544022    2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 12:44:25.554419 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:25.554388    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-38.ec2.internal\" not found"
Oct 30 12:44:25.655634 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:25.655444    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-38.ec2.internal\" not found"
Oct 30 12:44:25.672385 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:25.672336    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 12:39:25 +0000 UTC" deadline="2028-05-01 10:31:04.915494655 +0000 UTC"
Oct 30 12:44:25.672385 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:25.672391    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13173h46m39.243121547s"
Oct 30 12:44:25.716526 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:25.716487    2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:44:25.755709 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:25.755671    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-38.ec2.internal\" not found"
Oct 30 12:44:25.770135 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:25.770102    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:44:25.771442 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:25.771420    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:44:25.771593 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:25.771453    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:44:25.771593 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:25.771465    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:44:25.777524 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:25.777488    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d2044c13bd181c94b6dd5acd1654d5c.slice/crio-1712802f314501d149a03ffad2781f32b71ddf66a385dd3026d1ffe2585c8157 WatchSource:0}: Error finding container 1712802f314501d149a03ffad2781f32b71ddf66a385dd3026d1ffe2585c8157: Status 404 returned error can't find the container with id 1712802f314501d149a03ffad2781f32b71ddf66a385dd3026d1ffe2585c8157
Oct 30 12:44:25.781384 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:25.781359    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36805beca2c15cc87891009bcced9111.slice/crio-ada0dc5751c1b2c179d3403d8ebca87eb8e0f244a186544542c715f38395c108 WatchSource:0}: Error finding container ada0dc5751c1b2c179d3403d8ebca87eb8e0f244a186544542c715f38395c108: Status 404 returned error can't find the container with id ada0dc5751c1b2c179d3403d8ebca87eb8e0f244a186544542c715f38395c108
Oct 30 12:44:25.783636 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:25.783611    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 12:44:25.856659 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:25.856609    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-38.ec2.internal\" not found"
Oct 30 12:44:25.959201 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:25.959120    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-38.ec2.internal\" not found"
Oct 30 12:44:26.059968 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:26.059926    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-38.ec2.internal\" not found"
Oct 30 12:44:26.077694 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.077670    2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:44:26.113959 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.113928    2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal"
Oct 30 12:44:26.138360 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.138324    2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:44:26.139364 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.139348    2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-38.ec2.internal"
Oct 30 12:44:26.161600 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.161569    2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:44:26.204760 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.204728    2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:44:26.431061 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.430975    2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:44:26.584237 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.584177    2580 apiserver.go:52] "Watching apiserver"
Oct 30 12:44:26.613899 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.613861    2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 12:44:26.614290 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.614269    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 12:44:26.614436 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.614356    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-v9khv","openshift-ovn-kubernetes/ovnkube-node-9zfnt","kube-system/kube-apiserver-proxy-ip-10-0-143-38.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal","openshift-multus/multus-n8f6q","openshift-multus/network-metrics-daemon-mgctc","kube-system/global-pull-secret-syncer-xd7lj","openshift-dns/node-resolver-v68vl","openshift-network-diagnostics/network-check-target-jt6tp","openshift-cluster-node-tuning-operator/tuned-79zhh","openshift-multus/multus-additional-cni-plugins-bdjsz","openshift-network-operator/iptables-alerter-f82zx","kube-system/konnectivity-agent-9sprl"]
Oct 30 12:44:26.617186 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.617161    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgctc"
Oct 30 12:44:26.617335 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:26.617280    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgctc" podUID="d57e941d-1795-48f2-9c7a-7d04c84d0078"
Oct 30 12:44:26.619931 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.619904    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.622985 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.622328    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9sprl"
Oct 30 12:44:26.625576 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.624909    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:44:26.627660 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.627636    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:44:26.627780 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:26.627726    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xd7lj" podUID="b9736332-b3de-4fda-8372-880e2196a136"
Oct 30 12:44:26.628342 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.628316    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d57e941d-1795-48f2-9c7a-7d04c84d0078-metrics-certs\") pod \"network-metrics-daemon-mgctc\" (UID: \"d57e941d-1795-48f2-9c7a-7d04c84d0078\") " pod="openshift-multus/network-metrics-daemon-mgctc"
Oct 30 12:44:26.628444 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.628358    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l42fz\" (UniqueName: \"kubernetes.io/projected/d57e941d-1795-48f2-9c7a-7d04c84d0078-kube-api-access-l42fz\") pod \"network-metrics-daemon-mgctc\" (UID: \"d57e941d-1795-48f2-9c7a-7d04c84d0078\") " pod="openshift-multus/network-metrics-daemon-mgctc"
Oct 30 12:44:26.630894 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.630874    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.633427 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.633354    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v68vl"
Oct 30 12:44:26.636668 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.636644    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 12:44:26.637101 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.637082    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6f4z2\""
Oct 30 12:44:26.641646 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.641624    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f82zx"
Oct 30 12:44:26.643531 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.642824    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 12:44:26.643531 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.642929    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:44:26.643531 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.642978    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 12:44:26.643531 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.642984    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 12:44:26.643531 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.642980    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 12:44:26.643531 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.643037    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 12:44:26.643531 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.643112    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-l246t\""
Oct 30 12:44:26.643531 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.643227    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mwttw\""
Oct 30 12:44:26.644050 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.644028    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.646072 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.646054    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 12:44:26.646331 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.646307    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 12:44:26.646422 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.646352    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 12:44:26.646422 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.646371    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-b79n8\""
Oct 30 12:44:26.646535 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.646438    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 12:44:26.646752 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.646729    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt6tp"
Oct 30 12:44:26.646851 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:26.646814    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt6tp" podUID="5f33217b-afa5-4fff-a284-5fe1be413d5c"
Oct 30 12:44:26.649948 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.649301    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v9khv"
Oct 30 12:44:26.651111 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.651073    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 12:44:26.651553 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.651530    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-trr9l\""
Oct 30 12:44:26.651656 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.651594    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 12:44:26.652094 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.652048    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:44:26.665496 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.665461    2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 12:44:26.665607 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.665546    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6n2jd"
Oct 30 12:44:26.675954 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.675928    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6n2jd"
Oct 30 12:44:26.715581 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.715546    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 12:44:26.728965 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.728929    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-sys\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.729143 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.728978    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l42fz\" (UniqueName: \"kubernetes.io/projected/d57e941d-1795-48f2-9c7a-7d04c84d0078-kube-api-access-l42fz\") pod \"network-metrics-daemon-mgctc\" (UID: \"d57e941d-1795-48f2-9c7a-7d04c84d0078\") " pod="openshift-multus/network-metrics-daemon-mgctc"
Oct 30 12:44:26.729143 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729007    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-modprobe-d\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.729143 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729027    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-sysconfig\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.729143 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729044    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-run\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.729143 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729082    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-multus-socket-dir-parent\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.729143 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729110    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xffpq\" (UniqueName: \"kubernetes.io/projected/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-kube-api-access-xffpq\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.729143 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729134    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/abe27783-da2a-46b9-81f6-cf551787b1bc-konnectivity-ca\") pod \"konnectivity-agent-9sprl\" (UID: \"abe27783-da2a-46b9-81f6-cf551787b1bc\") " pod="kube-system/konnectivity-agent-9sprl"
Oct 30 12:44:26.729536 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729156    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:44:26.729536 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729173    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-cnibin\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.729536 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729188    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-var-lib-openvswitch\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.729536 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729209    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/48212ac1-a400-4369-87dd-07b4d5317399-host-slash\") pod \"iptables-alerter-f82zx\" (UID: \"48212ac1-a400-4369-87dd-07b4d5317399\") " pod="openshift-network-operator/iptables-alerter-f82zx"
Oct 30 12:44:26.729536 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729241    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-cni-binary-copy\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:44:26.729536 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-systemd\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.729536 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729325    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-host\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.729536 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729374    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-systemd-units\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.729536 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729400    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-env-overrides\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.729536 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729432    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfhxh\" (UniqueName: \"kubernetes.io/projected/2eef9c4d-7079-47ad-9fad-c89c16ac69b4-kube-api-access-kfhxh\") pod \"node-ca-v9khv\" (UID: \"2eef9c4d-7079-47ad-9fad-c89c16ac69b4\") " pod="openshift-image-registry/node-ca-v9khv"
Oct 30 12:44:26.729536 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729465    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-kubernetes\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.729536 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729484    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-etc-kubernetes\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.729536 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729519    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-run-netns\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.729536 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729538    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkk44\" (UniqueName: \"kubernetes.io/projected/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-kube-api-access-nkk44\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.730230 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729566    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:44:26.730230 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729606    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-device-dir\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:44:26.730230 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729626    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-sys-fs\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:44:26.730230 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729665    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-system-cni-dir\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.730230 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729694    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-var-lib-kubelet\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.730230 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729722    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/940fa76d-d452-41fc-99ad-0f16f51c842e-tmp\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.730230 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729760    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z5dz\" (UniqueName: \"kubernetes.io/projected/940fa76d-d452-41fc-99ad-0f16f51c842e-kube-api-access-5z5dz\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.730230 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729793    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-socket-dir\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:44:26.730230 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729823    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-cni-binary-copy\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.730230 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729853    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d57e941d-1795-48f2-9c7a-7d04c84d0078-metrics-certs\") pod \"network-metrics-daemon-mgctc\" (UID: \"d57e941d-1795-48f2-9c7a-7d04c84d0078\") " pod="openshift-multus/network-metrics-daemon-mgctc"
Oct 30 12:44:26.730230 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729870    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-multus-daemon-config\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.730230 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729888    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncgsv\" (UniqueName: \"kubernetes.io/projected/7b607752-d51b-4781-a57f-99e95883d236-kube-api-access-ncgsv\") pod \"node-resolver-v68vl\" (UID: \"7b607752-d51b-4781-a57f-99e95883d236\") " pod="openshift-dns/node-resolver-v68vl"
Oct 30 12:44:26.730230 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729910    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-cnibin\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:44:26.730230 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729933    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-run-systemd\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.730230 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729975    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-etc-openvswitch\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.730230 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.729999    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-log-socket\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.731962 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.730022    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-multus-cni-dir\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.731962 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.730053    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-sysctl-conf\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.731962 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.730084    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-run-netns\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.731962 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.730104    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-hostroot\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.731962 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.730147    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-multus-conf-dir\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.731962 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.730164    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-cni-bin\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.731962 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.730181    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-var-lib-cni-multus\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.731962 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.730200    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-run-ovn-kubernetes\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.731962 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.730296    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.731962 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.730380    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-ovn-node-metrics-cert\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.731962 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.730412    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-ovnkube-script-lib\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.731962 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.730436    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2eef9c4d-7079-47ad-9fad-c89c16ac69b4-serviceca\") pod \"node-ca-v9khv\" (UID: \"2eef9c4d-7079-47ad-9fad-c89c16ac69b4\") " pod="openshift-image-registry/node-ca-v9khv"
Oct 30 12:44:26.731962 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.730473    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-system-cni-dir\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:44:26.731962 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.730501    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:44:26.731962 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.730537    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-var-lib-kubelet\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.731962 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.730574    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-kubelet\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.732565 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.730607    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/48212ac1-a400-4369-87dd-07b4d5317399-iptables-alerter-script\") pod \"iptables-alerter-f82zx\" (UID: \"48212ac1-a400-4369-87dd-07b4d5317399\") " pod="openshift-network-operator/iptables-alerter-f82zx"
Oct 30 12:44:26.732565 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.730639    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldcfg\" (UniqueName: \"kubernetes.io/projected/48212ac1-a400-4369-87dd-07b4d5317399-kube-api-access-ldcfg\") pod \"iptables-alerter-f82zx\" (UID: \"48212ac1-a400-4369-87dd-07b4d5317399\") " pod="openshift-network-operator/iptables-alerter-f82zx"
Oct 30 12:44:26.732565 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.730685    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-tuned\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.732565 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.730735    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlrgs\" (UniqueName: \"kubernetes.io/projected/7bb8182c-0949-40e4-adeb-b8254e44922e-kube-api-access-jlrgs\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:44:26.732565 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.730777    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-var-lib-cni-bin\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.732565 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:26.730805    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:44:26.732565 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.730808    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7b607752-d51b-4781-a57f-99e95883d236-hosts-file\") pod \"node-resolver-v68vl\" (UID: \"7b607752-d51b-4781-a57f-99e95883d236\") " pod="openshift-dns/node-resolver-v68vl"
Oct 30 12:44:26.732565 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:26.730901    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d57e941d-1795-48f2-9c7a-7d04c84d0078-metrics-certs podName:d57e941d-1795-48f2-9c7a-7d04c84d0078 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:27.230868984 +0000 UTC m=+3.036231652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d57e941d-1795-48f2-9c7a-7d04c84d0078-metrics-certs") pod "network-metrics-daemon-mgctc" (UID: "d57e941d-1795-48f2-9c7a-7d04c84d0078") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:44:26.732565 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.730941    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-cni-netd\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.732565 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.730996    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/abe27783-da2a-46b9-81f6-cf551787b1bc-agent-certs\") pod \"konnectivity-agent-9sprl\" (UID: \"abe27783-da2a-46b9-81f6-cf551787b1bc\") " pod="kube-system/konnectivity-agent-9sprl"
Oct 30 12:44:26.732565 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.731035    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b9736332-b3de-4fda-8372-880e2196a136-kubelet-config\") pod \"global-pull-secret-syncer-xd7lj\" (UID: \"b9736332-b3de-4fda-8372-880e2196a136\") " pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:44:26.732565 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.731053    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2eef9c4d-7079-47ad-9fad-c89c16ac69b4-host\") pod \"node-ca-v9khv\" (UID: \"2eef9c4d-7079-47ad-9fad-c89c16ac69b4\") " pod="openshift-image-registry/node-ca-v9khv"
Oct 30 12:44:26.732565 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.731097    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-lib-modules\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.732565 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.731139    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-run-k8s-cni-cncf-io\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.732565 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.731171    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-run-ovn\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.732565 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.731210    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-node-log\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.733209 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.731237    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-sysctl-d\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.733209 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.731272    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-registration-dir\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:44:26.733209 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.731306    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-etc-selinux\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:44:26.733209 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.731342    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b607752-d51b-4781-a57f-99e95883d236-tmp-dir\") pod \"node-resolver-v68vl\" (UID: \"7b607752-d51b-4781-a57f-99e95883d236\") " pod="openshift-dns/node-resolver-v68vl"
Oct 30 12:44:26.733209 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.731380    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-ovnkube-config\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.733209 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.731408    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2jqh\" (UniqueName: \"kubernetes.io/projected/5f33217b-afa5-4fff-a284-5fe1be413d5c-kube-api-access-w2jqh\") pod \"network-check-target-jt6tp\" (UID: \"5f33217b-afa5-4fff-a284-5fe1be413d5c\") " pod="openshift-network-diagnostics/network-check-target-jt6tp"
Oct 30 12:44:26.733209 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.731443    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8ns9\" (UniqueName: \"kubernetes.io/projected/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-kube-api-access-p8ns9\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:44:26.733209 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.731476    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-run-multus-certs\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.733209 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.731505    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-run-openvswitch\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.733209 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.731559    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b9736332-b3de-4fda-8372-880e2196a136-dbus\") pod \"global-pull-secret-syncer-xd7lj\" (UID: \"b9736332-b3de-4fda-8372-880e2196a136\") " pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:44:26.733209 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.731602    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9736332-b3de-4fda-8372-880e2196a136-original-pull-secret\") pod \"global-pull-secret-syncer-xd7lj\" (UID: \"b9736332-b3de-4fda-8372-880e2196a136\") " pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:44:26.733209 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.731648    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:44:26.733209 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.731677    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-slash\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.744265 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.744238    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 12:44:26.744602 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.744578    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-svz4t\""
Oct 30 12:44:26.744718 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.744606    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6g9fh\""
Oct 30 12:44:26.744718 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.744617    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 12:44:26.744896 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.744873    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 12:44:26.744998 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.744914    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 12:44:26.744998 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.744944    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 12:44:26.744998 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.744972    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 12:44:26.745620 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.745596    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 12:44:26.745715 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.745679    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 12:44:26.745715 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.745711    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:44:26.745819 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.745765    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 12:44:26.745919 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.745901    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 12:44:26.745974 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.745933    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k9vxw\""
Oct 30 12:44:26.746169 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.746152    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 12:44:26.746284 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.746269    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 12:44:26.746366 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.746349    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pglr2\""
Oct 30 12:44:26.746422 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.746403    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 12:44:26.754399 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.754370    2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 12:44:26.758391 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.758307    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal" event={"ID":"36805beca2c15cc87891009bcced9111","Type":"ContainerStarted","Data":"ada0dc5751c1b2c179d3403d8ebca87eb8e0f244a186544542c715f38395c108"}
Oct 30 12:44:26.759318 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.759280    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l42fz\" (UniqueName: \"kubernetes.io/projected/d57e941d-1795-48f2-9c7a-7d04c84d0078-kube-api-access-l42fz\") pod \"network-metrics-daemon-mgctc\" (UID: \"d57e941d-1795-48f2-9c7a-7d04c84d0078\") " pod="openshift-multus/network-metrics-daemon-mgctc"
Oct 30 12:44:26.759441 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.759380    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-38.ec2.internal" event={"ID":"6d2044c13bd181c94b6dd5acd1654d5c","Type":"ContainerStarted","Data":"1712802f314501d149a03ffad2781f32b71ddf66a385dd3026d1ffe2585c8157"}
Oct 30 12:44:26.832998 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.832966    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-run-systemd\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.832998 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833005    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-etc-openvswitch\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.833309 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833035    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-log-socket\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.833309 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833058    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-multus-cni-dir\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.833309 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833087    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-sysctl-conf\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.833309 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833113    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-run-netns\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.833309 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833138    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-hostroot\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.833309 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833162    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-multus-conf-dir\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.833309 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833179    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-cni-bin\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.833309 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833197    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-var-lib-cni-multus\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.833309 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833217    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-run-ovn-kubernetes\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.833309 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833243    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.833309 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833284    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-ovn-node-metrics-cert\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.833309 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833310    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-ovnkube-script-lib\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.833926 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833330    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2eef9c4d-7079-47ad-9fad-c89c16ac69b4-serviceca\") pod \"node-ca-v9khv\" (UID: \"2eef9c4d-7079-47ad-9fad-c89c16ac69b4\") " pod="openshift-image-registry/node-ca-v9khv"
Oct 30 12:44:26.833926 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833355    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-system-cni-dir\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:44:26.833926 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833371    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:44:26.833926 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833392    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-var-lib-kubelet\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.833926 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833408    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-kubelet\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.833926 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833436    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/48212ac1-a400-4369-87dd-07b4d5317399-iptables-alerter-script\") pod \"iptables-alerter-f82zx\" (UID: \"48212ac1-a400-4369-87dd-07b4d5317399\") " pod="openshift-network-operator/iptables-alerter-f82zx"
Oct 30 12:44:26.833926 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833459    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldcfg\" (UniqueName: \"kubernetes.io/projected/48212ac1-a400-4369-87dd-07b4d5317399-kube-api-access-ldcfg\") pod \"iptables-alerter-f82zx\" (UID: \"48212ac1-a400-4369-87dd-07b4d5317399\") " pod="openshift-network-operator/iptables-alerter-f82zx"
Oct 30 12:44:26.833926 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833476    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-tuned\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.833926 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833493    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlrgs\" (UniqueName: \"kubernetes.io/projected/7bb8182c-0949-40e4-adeb-b8254e44922e-kube-api-access-jlrgs\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:44:26.833926 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833509    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-var-lib-cni-bin\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.833926 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833527    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7b607752-d51b-4781-a57f-99e95883d236-hosts-file\") pod \"node-resolver-v68vl\" (UID: \"7b607752-d51b-4781-a57f-99e95883d236\") " pod="openshift-dns/node-resolver-v68vl"
Oct 30 12:44:26.833926 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833548    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-cni-netd\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.833926 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833569    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/abe27783-da2a-46b9-81f6-cf551787b1bc-agent-certs\") pod \"konnectivity-agent-9sprl\" (UID: \"abe27783-da2a-46b9-81f6-cf551787b1bc\") " pod="kube-system/konnectivity-agent-9sprl"
Oct 30 12:44:26.833926 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833594    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b9736332-b3de-4fda-8372-880e2196a136-kubelet-config\") pod \"global-pull-secret-syncer-xd7lj\" (UID: \"b9736332-b3de-4fda-8372-880e2196a136\") " pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:44:26.833926 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833612    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2eef9c4d-7079-47ad-9fad-c89c16ac69b4-host\") pod \"node-ca-v9khv\" (UID: \"2eef9c4d-7079-47ad-9fad-c89c16ac69b4\") " pod="openshift-image-registry/node-ca-v9khv"
Oct 30 12:44:26.833926 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833637    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-lib-modules\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.833926 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833663    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-run-k8s-cni-cncf-io\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.834763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833680    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-run-ovn\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.834763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833699    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-node-log\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.834763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833726    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-sysctl-d\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.834763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833749    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-registration-dir\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:44:26.834763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833773    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-etc-selinux\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:44:26.834763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833790    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b607752-d51b-4781-a57f-99e95883d236-tmp-dir\") pod \"node-resolver-v68vl\" (UID: \"7b607752-d51b-4781-a57f-99e95883d236\") " pod="openshift-dns/node-resolver-v68vl"
Oct 30 12:44:26.834763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833814    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-ovnkube-config\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.834763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833837    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2jqh\" (UniqueName: \"kubernetes.io/projected/5f33217b-afa5-4fff-a284-5fe1be413d5c-kube-api-access-w2jqh\") pod \"network-check-target-jt6tp\" (UID: \"5f33217b-afa5-4fff-a284-5fe1be413d5c\") " pod="openshift-network-diagnostics/network-check-target-jt6tp"
Oct 30 12:44:26.834763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833858    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8ns9\" (UniqueName: \"kubernetes.io/projected/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-kube-api-access-p8ns9\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:44:26.834763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833876    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-run-multus-certs\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.834763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833901    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-run-openvswitch\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.834763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833927    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b9736332-b3de-4fda-8372-880e2196a136-dbus\") pod \"global-pull-secret-syncer-xd7lj\" (UID: \"b9736332-b3de-4fda-8372-880e2196a136\") " pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:44:26.834763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833954    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9736332-b3de-4fda-8372-880e2196a136-original-pull-secret\") pod \"global-pull-secret-syncer-xd7lj\" (UID: \"b9736332-b3de-4fda-8372-880e2196a136\") " pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:44:26.834763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.833982    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:44:26.834763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834016    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-slash\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.834763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834039    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-sys\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.834763 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834072    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-modprobe-d\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.835444 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834098    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-sysconfig\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.835444 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834132    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-run\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.835444 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834155    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-multus-socket-dir-parent\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.835444 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834183    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xffpq\" (UniqueName: \"kubernetes.io/projected/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-kube-api-access-xffpq\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.835444 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834210    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/abe27783-da2a-46b9-81f6-cf551787b1bc-konnectivity-ca\") pod \"konnectivity-agent-9sprl\" (UID: \"abe27783-da2a-46b9-81f6-cf551787b1bc\") " pod="kube-system/konnectivity-agent-9sprl"
Oct 30 12:44:26.835444 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834238    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:44:26.835444 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834256    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-cnibin\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.835444 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834274    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-var-lib-openvswitch\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.835444 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834324    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/48212ac1-a400-4369-87dd-07b4d5317399-host-slash\") pod \"iptables-alerter-f82zx\" (UID: \"48212ac1-a400-4369-87dd-07b4d5317399\") " pod="openshift-network-operator/iptables-alerter-f82zx"
Oct 30 12:44:26.835444 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834350    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-cni-binary-copy\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:44:26.835444 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-systemd\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.835444 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834396    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-host\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.835444 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834417    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b9736332-b3de-4fda-8372-880e2196a136-kubelet-config\") pod \"global-pull-secret-syncer-xd7lj\" (UID: \"b9736332-b3de-4fda-8372-880e2196a136\") " pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:44:26.835444 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834432    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-run-openvswitch\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.835444 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834421    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-systemd-units\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.835444 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834459    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-systemd-units\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.835444 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834480    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-env-overrides\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.835444 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834492    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2eef9c4d-7079-47ad-9fad-c89c16ac69b4-host\") pod \"node-ca-v9khv\" (UID: \"2eef9c4d-7079-47ad-9fad-c89c16ac69b4\") " pod="openshift-image-registry/node-ca-v9khv"
Oct 30 12:44:26.836340 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834507    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfhxh\" (UniqueName: \"kubernetes.io/projected/2eef9c4d-7079-47ad-9fad-c89c16ac69b4-kube-api-access-kfhxh\") pod \"node-ca-v9khv\" (UID: \"2eef9c4d-7079-47ad-9fad-c89c16ac69b4\") " pod="openshift-image-registry/node-ca-v9khv"
Oct 30 12:44:26.836340 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834535    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-kubernetes\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.836340 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834559    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-etc-kubernetes\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.836340 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834582    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-run-netns\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.836340 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834612    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkk44\" (UniqueName: \"kubernetes.io/projected/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-kube-api-access-nkk44\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.836340 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834639    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-run-k8s-cni-cncf-io\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.836340 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834641    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:44:26.836340 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834614    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b9736332-b3de-4fda-8372-880e2196a136-dbus\") pod \"global-pull-secret-syncer-xd7lj\" (UID: \"b9736332-b3de-4fda-8372-880e2196a136\") " pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:44:26.836340 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834675    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-device-dir\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:44:26.836340 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834688    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-run-ovn\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.836340 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834701    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-sys-fs\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:44:26.836340 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:26.834712    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:44:26.836340 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834727    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-system-cni-dir\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.836340 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834733    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-node-log\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.836340 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834753    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-var-lib-kubelet\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.836340 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:26.834777    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9736332-b3de-4fda-8372-880e2196a136-original-pull-secret podName:b9736332-b3de-4fda-8372-880e2196a136 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:27.334754555 +0000 UTC m=+3.140117229 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9736332-b3de-4fda-8372-880e2196a136-original-pull-secret") pod "global-pull-secret-syncer-xd7lj" (UID: "b9736332-b3de-4fda-8372-880e2196a136") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:44:26.836340 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834805    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-var-lib-kubelet\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.837191 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834806    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/940fa76d-d452-41fc-99ad-0f16f51c842e-tmp\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.837191 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834839    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-sysctl-d\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.837191 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834845    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z5dz\" (UniqueName: \"kubernetes.io/projected/940fa76d-d452-41fc-99ad-0f16f51c842e-kube-api-access-5z5dz\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.837191 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834900    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-registration-dir\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:44:26.837191 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834991    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-etc-selinux\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:44:26.837191 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.835033    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-run-systemd\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.837191 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.835071    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-etc-openvswitch\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.837191 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.835107    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-log-socket\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.837191 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.835166    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-multus-cni-dir\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.837191 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.835174    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b607752-d51b-4781-a57f-99e95883d236-tmp-dir\") pod \"node-resolver-v68vl\" (UID: \"7b607752-d51b-4781-a57f-99e95883d236\") " pod="openshift-dns/node-resolver-v68vl"
Oct 30 12:44:26.837191 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.835288    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-sysctl-conf\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.837191 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.835349    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-run-netns\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.837191 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.835396    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-hostroot\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.837191 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.835438    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-multus-conf-dir\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.837191 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.835486    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-cni-bin\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.837191 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.835529    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-var-lib-cni-multus\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.837191 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.835571    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-run-multus-certs\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.837191 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.835581    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-run-ovn-kubernetes\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.838061 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.835624    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.838061 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.835632    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/48212ac1-a400-4369-87dd-07b4d5317399-host-slash\") pod \"iptables-alerter-f82zx\" (UID: \"48212ac1-a400-4369-87dd-07b4d5317399\") " pod="openshift-network-operator/iptables-alerter-f82zx"
Oct 30 12:44:26.838061 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.835680    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-var-lib-openvswitch\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.838061 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.835731    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-systemd\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.838061 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.835803    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/abe27783-da2a-46b9-81f6-cf551787b1bc-konnectivity-ca\") pod \"konnectivity-agent-9sprl\" (UID: \"abe27783-da2a-46b9-81f6-cf551787b1bc\") " pod="kube-system/konnectivity-agent-9sprl"
Oct 30 12:44:26.838061 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.836318    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-cni-binary-copy\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:44:26.838061 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.836413    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-ovnkube-config\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.838061 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.836936    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:44:26.838061 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.836983    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-ovnkube-script-lib\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.838061 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.836993    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-slash\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.838061 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.837051    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-system-cni-dir\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:44:26.838061 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.837056    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-sys\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.838061 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.837194    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:44:26.838061 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.837240    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-var-lib-kubelet\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.838061 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.837273    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-kubelet\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.838061 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.837493    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2eef9c4d-7079-47ad-9fad-c89c16ac69b4-serviceca\") pod \"node-ca-v9khv\" (UID: \"2eef9c4d-7079-47ad-9fad-c89c16ac69b4\") " pod="openshift-image-registry/node-ca-v9khv"
Oct 30 12:44:26.838061 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.837624    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-modprobe-d\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.838908 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.837690    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-sysconfig\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.838908 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.837742    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-run\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.838908 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.837754    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/48212ac1-a400-4369-87dd-07b4d5317399-iptables-alerter-script\") pod \"iptables-alerter-f82zx\" (UID: \"48212ac1-a400-4369-87dd-07b4d5317399\") " pod="openshift-network-operator/iptables-alerter-f82zx"
Oct 30 12:44:26.838908 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.837780    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-multus-socket-dir-parent\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.838908 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.838010    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-host\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.838908 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.838055    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-sys-fs\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:44:26.838908 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.838109    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-kubernetes\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.838908 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.838159    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-etc-kubernetes\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.838908 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.838202    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-run-netns\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.838908 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.838279    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-socket-dir\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:44:26.838908 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.838353    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-cni-binary-copy\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.838908 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.838408    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-multus-daemon-config\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.838908 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.838440    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncgsv\" (UniqueName: \"kubernetes.io/projected/7b607752-d51b-4781-a57f-99e95883d236-kube-api-access-ncgsv\") pod \"node-resolver-v68vl\" (UID: \"7b607752-d51b-4781-a57f-99e95883d236\") " pod="openshift-dns/node-resolver-v68vl"
Oct 30 12:44:26.838908 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.838465    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7b607752-d51b-4781-a57f-99e95883d236-hosts-file\") pod \"node-resolver-v68vl\" (UID: \"7b607752-d51b-4781-a57f-99e95883d236\") " pod="openshift-dns/node-resolver-v68vl"
Oct 30 12:44:26.838908 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.838473    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-cnibin\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:44:26.838908 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.838488    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-ovn-node-metrics-cert\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.838908 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.838522    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-cnibin\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:44:26.838908 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.834614    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-lib-modules\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.839861 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.838600    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-system-cni-dir\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.839861 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.838635    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-cnibin\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.839861 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.838677    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-var-lib-cni-bin\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.839861 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.838688    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:44:26.839861 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.838752    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-socket-dir\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:44:26.839861 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.838984    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:44:26.839861 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.839015    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-cni-netd\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.839861 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.839065    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-device-dir\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:44:26.840269 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.839910    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-multus-daemon-config\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.840269 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.840101    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-env-overrides\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.840450 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.840382    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-cni-binary-copy\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.840949 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.840920    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-tuned\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.841168 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.841145    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/abe27783-da2a-46b9-81f6-cf551787b1bc-agent-certs\") pod \"konnectivity-agent-9sprl\" (UID: \"abe27783-da2a-46b9-81f6-cf551787b1bc\") " pod="kube-system/konnectivity-agent-9sprl"
Oct 30 12:44:26.842890 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.842865    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/940fa76d-d452-41fc-99ad-0f16f51c842e-tmp\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.860338 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.860284    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8ns9\" (UniqueName: \"kubernetes.io/projected/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-kube-api-access-p8ns9\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:44:26.860778 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.860753    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfhxh\" (UniqueName: \"kubernetes.io/projected/2eef9c4d-7079-47ad-9fad-c89c16ac69b4-kube-api-access-kfhxh\") pod \"node-ca-v9khv\" (UID: \"2eef9c4d-7079-47ad-9fad-c89c16ac69b4\") " pod="openshift-image-registry/node-ca-v9khv"
Oct 30 12:44:26.861771 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.861747    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xffpq\" (UniqueName: \"kubernetes.io/projected/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-kube-api-access-xffpq\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.861870 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.861799    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldcfg\" (UniqueName: \"kubernetes.io/projected/48212ac1-a400-4369-87dd-07b4d5317399-kube-api-access-ldcfg\") pod \"iptables-alerter-f82zx\" (UID: \"48212ac1-a400-4369-87dd-07b4d5317399\") " pod="openshift-network-operator/iptables-alerter-f82zx"
Oct 30 12:44:26.862517 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:26.862495    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:44:26.862624 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:26.862528    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:44:26.862624 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:26.862544    2580 projected.go:196] Error preparing data for projected volume kube-api-access-w2jqh for pod openshift-network-diagnostics/network-check-target-jt6tp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:44:26.862740 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:26.862625    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f33217b-afa5-4fff-a284-5fe1be413d5c-kube-api-access-w2jqh podName:5f33217b-afa5-4fff-a284-5fe1be413d5c nodeName:}" failed. No retries permitted until 2026-10-30 12:44:27.362602775 +0000 UTC m=+3.167965430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-w2jqh" (UniqueName: "kubernetes.io/projected/5f33217b-afa5-4fff-a284-5fe1be413d5c-kube-api-access-w2jqh") pod "network-check-target-jt6tp" (UID: "5f33217b-afa5-4fff-a284-5fe1be413d5c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:44:26.865596 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.865570    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkk44\" (UniqueName: \"kubernetes.io/projected/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-kube-api-access-nkk44\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:26.866638 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.866613    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z5dz\" (UniqueName: \"kubernetes.io/projected/940fa76d-d452-41fc-99ad-0f16f51c842e-kube-api-access-5z5dz\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.867535 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.867366    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlrgs\" (UniqueName: \"kubernetes.io/projected/7bb8182c-0949-40e4-adeb-b8254e44922e-kube-api-access-jlrgs\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:44:26.867535 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.867473    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncgsv\" (UniqueName: \"kubernetes.io/projected/7b607752-d51b-4781-a57f-99e95883d236-kube-api-access-ncgsv\") pod \"node-resolver-v68vl\" (UID: \"7b607752-d51b-4781-a57f-99e95883d236\") " pod="openshift-dns/node-resolver-v68vl"
Oct 30 12:44:26.940881 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.940793    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:44:26.945187 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.945151    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9sprl"
Oct 30 12:44:26.954695 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:26.954664    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940fa76d_d452_41fc_99ad_0f16f51c842e.slice/crio-d0011c776d3e787aca8ce0d2ccd0840efc0669e666d866aa632f7c37de6ba5d0 WatchSource:0}: Error finding container d0011c776d3e787aca8ce0d2ccd0840efc0669e666d866aa632f7c37de6ba5d0: Status 404 returned error can't find the container with id d0011c776d3e787aca8ce0d2ccd0840efc0669e666d866aa632f7c37de6ba5d0
Oct 30 12:44:26.968776 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.968738    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:44:26.971446 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.971417    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v9khv"
Oct 30 12:44:26.978286 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.978071    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n8f6q"
Oct 30 12:44:26.983066 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:26.983029    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb8182c_0949_40e4_adeb_b8254e44922e.slice/crio-e0008197a374974d904d3d54eccbdb31bb9ee232b6ec21ada276f6249614ca2a WatchSource:0}: Error finding container e0008197a374974d904d3d54eccbdb31bb9ee232b6ec21ada276f6249614ca2a: Status 404 returned error can't find the container with id e0008197a374974d904d3d54eccbdb31bb9ee232b6ec21ada276f6249614ca2a
Oct 30 12:44:26.984138 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:26.984106    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eef9c4d_7079_47ad_9fad_c89c16ac69b4.slice/crio-2db7f273c7518b0745c5e8bae9f6e47ed93e507e118eda0fab3419ec56a49269 WatchSource:0}: Error finding container 2db7f273c7518b0745c5e8bae9f6e47ed93e507e118eda0fab3419ec56a49269: Status 404 returned error can't find the container with id 2db7f273c7518b0745c5e8bae9f6e47ed93e507e118eda0fab3419ec56a49269
Oct 30 12:44:26.986974 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.986946    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v68vl"
Oct 30 12:44:26.989555 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:26.989528    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod836d7dec_cc0f_49ae_b1d5_d07a8dcefaf2.slice/crio-2bc309e11242418011070b5c2efea6b00aa043f633edc74ae4d4b9631abca96d WatchSource:0}: Error finding container 2bc309e11242418011070b5c2efea6b00aa043f633edc74ae4d4b9631abca96d: Status 404 returned error can't find the container with id 2bc309e11242418011070b5c2efea6b00aa043f633edc74ae4d4b9631abca96d
Oct 30 12:44:26.994554 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:26.994531    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f82zx"
Oct 30 12:44:26.999614 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:26.999582    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b607752_d51b_4781_a57f_99e95883d236.slice/crio-ccba6cd87c2d004d3980a01255dde1f600c6883887a957ee1ebb161c1c73df5e WatchSource:0}: Error finding container ccba6cd87c2d004d3980a01255dde1f600c6883887a957ee1ebb161c1c73df5e: Status 404 returned error can't find the container with id ccba6cd87c2d004d3980a01255dde1f600c6883887a957ee1ebb161c1c73df5e
Oct 30 12:44:27.007806 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:27.007768    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48212ac1_a400_4369_87dd_07b4d5317399.slice/crio-a98741e4e980f2331f8ad5c157a7062e44ab9d43ad76901e9f5e1473c7878a75 WatchSource:0}: Error finding container a98741e4e980f2331f8ad5c157a7062e44ab9d43ad76901e9f5e1473c7878a75: Status 404 returned error can't find the container with id a98741e4e980f2331f8ad5c157a7062e44ab9d43ad76901e9f5e1473c7878a75
Oct 30 12:44:27.027038 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:27.026998    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:27.036590 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:27.036561    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:44:27.039014 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:27.038984    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b0e3727_f71f_4edd_bfc3_0b1ef8c4f623.slice/crio-80aa64600269b92137dd94b6ffe57210ba58dc9cf999790ad098f9f7502399d9 WatchSource:0}: Error finding container 80aa64600269b92137dd94b6ffe57210ba58dc9cf999790ad098f9f7502399d9: Status 404 returned error can't find the container with id 80aa64600269b92137dd94b6ffe57210ba58dc9cf999790ad098f9f7502399d9
Oct 30 12:44:27.241142 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:27.241095    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d57e941d-1795-48f2-9c7a-7d04c84d0078-metrics-certs\") pod \"network-metrics-daemon-mgctc\" (UID: \"d57e941d-1795-48f2-9c7a-7d04c84d0078\") " pod="openshift-multus/network-metrics-daemon-mgctc"
Oct 30 12:44:27.241326 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:27.241302    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:44:27.241393 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:27.241369    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d57e941d-1795-48f2-9c7a-7d04c84d0078-metrics-certs podName:d57e941d-1795-48f2-9c7a-7d04c84d0078 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:28.241353983 +0000 UTC m=+4.046716634 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d57e941d-1795-48f2-9c7a-7d04c84d0078-metrics-certs") pod "network-metrics-daemon-mgctc" (UID: "d57e941d-1795-48f2-9c7a-7d04c84d0078") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:44:27.342402 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:27.342367    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9736332-b3de-4fda-8372-880e2196a136-original-pull-secret\") pod \"global-pull-secret-syncer-xd7lj\" (UID: \"b9736332-b3de-4fda-8372-880e2196a136\") " pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:44:27.342576 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:27.342510    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:44:27.342638 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:27.342581    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9736332-b3de-4fda-8372-880e2196a136-original-pull-secret podName:b9736332-b3de-4fda-8372-880e2196a136 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:28.342562856 +0000 UTC m=+4.147925504 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9736332-b3de-4fda-8372-880e2196a136-original-pull-secret") pod "global-pull-secret-syncer-xd7lj" (UID: "b9736332-b3de-4fda-8372-880e2196a136") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:44:27.444485 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:27.443625    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2jqh\" (UniqueName: \"kubernetes.io/projected/5f33217b-afa5-4fff-a284-5fe1be413d5c-kube-api-access-w2jqh\") pod \"network-check-target-jt6tp\" (UID: \"5f33217b-afa5-4fff-a284-5fe1be413d5c\") " pod="openshift-network-diagnostics/network-check-target-jt6tp"
Oct 30 12:44:27.444485 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:27.443835    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:44:27.444485 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:27.443864    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:44:27.444485 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:27.443879    2580 projected.go:196] Error preparing data for projected volume kube-api-access-w2jqh for pod openshift-network-diagnostics/network-check-target-jt6tp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:44:27.444485 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:27.443990    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f33217b-afa5-4fff-a284-5fe1be413d5c-kube-api-access-w2jqh podName:5f33217b-afa5-4fff-a284-5fe1be413d5c nodeName:}" failed. No retries permitted until 2026-10-30 12:44:28.443967843 +0000 UTC m=+4.249330500 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-w2jqh" (UniqueName: "kubernetes.io/projected/5f33217b-afa5-4fff-a284-5fe1be413d5c-kube-api-access-w2jqh") pod "network-check-target-jt6tp" (UID: "5f33217b-afa5-4fff-a284-5fe1be413d5c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:44:27.679914 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:27.679728    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:39:27 +0000 UTC" deadline="2028-08-07 14:52:57.758974606 +0000 UTC"
Oct 30 12:44:27.679914 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:27.679798    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15530h8m30.079181204s"
Oct 30 12:44:27.754329 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:27.754287    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt6tp"
Oct 30 12:44:27.754534 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:27.754506    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt6tp" podUID="5f33217b-afa5-4fff-a284-5fe1be413d5c"
Oct 30 12:44:27.768158 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:27.767348    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-38.ec2.internal" event={"ID":"6d2044c13bd181c94b6dd5acd1654d5c","Type":"ContainerStarted","Data":"4fb53e3788edcdcb66639c8d45b625a6dcebb33ddc596ba9704b6474ee39abfc"}
Oct 30 12:44:27.777664 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:27.777571    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bdjsz" event={"ID":"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee","Type":"ContainerStarted","Data":"73c9689bdccb305b8354b2a73384d8f7fe052a3fbbc05fd6f2a3346267cf689b"}
Oct 30 12:44:27.784352 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:27.784169    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt" event={"ID":"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623","Type":"ContainerStarted","Data":"80aa64600269b92137dd94b6ffe57210ba58dc9cf999790ad098f9f7502399d9"}
Oct 30 12:44:27.786752 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:27.786680    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v68vl" event={"ID":"7b607752-d51b-4781-a57f-99e95883d236","Type":"ContainerStarted","Data":"ccba6cd87c2d004d3980a01255dde1f600c6883887a957ee1ebb161c1c73df5e"}
Oct 30 12:44:27.788566 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:27.788538    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n8f6q" event={"ID":"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2","Type":"ContainerStarted","Data":"2bc309e11242418011070b5c2efea6b00aa043f633edc74ae4d4b9631abca96d"}
Oct 30 12:44:27.791046 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:27.791017    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9sprl" event={"ID":"abe27783-da2a-46b9-81f6-cf551787b1bc","Type":"ContainerStarted","Data":"696affb719b735bd04e03d28e25d2958829aeaff062c9fe97e096a2b8197acbd"}
Oct 30 12:44:27.793991 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:27.793965    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f82zx" event={"ID":"48212ac1-a400-4369-87dd-07b4d5317399","Type":"ContainerStarted","Data":"a98741e4e980f2331f8ad5c157a7062e44ab9d43ad76901e9f5e1473c7878a75"}
Oct 30 12:44:27.795677 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:27.795616    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v9khv" event={"ID":"2eef9c4d-7079-47ad-9fad-c89c16ac69b4","Type":"ContainerStarted","Data":"2db7f273c7518b0745c5e8bae9f6e47ed93e507e118eda0fab3419ec56a49269"}
Oct 30 12:44:27.799138 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:27.799055    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx" event={"ID":"7bb8182c-0949-40e4-adeb-b8254e44922e","Type":"ContainerStarted","Data":"e0008197a374974d904d3d54eccbdb31bb9ee232b6ec21ada276f6249614ca2a"}
Oct 30 12:44:27.805014 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:27.804961    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-79zhh" event={"ID":"940fa76d-d452-41fc-99ad-0f16f51c842e","Type":"ContainerStarted","Data":"d0011c776d3e787aca8ce0d2ccd0840efc0669e666d866aa632f7c37de6ba5d0"}
Oct 30 12:44:28.251187 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:28.250591    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d57e941d-1795-48f2-9c7a-7d04c84d0078-metrics-certs\") pod \"network-metrics-daemon-mgctc\" (UID: \"d57e941d-1795-48f2-9c7a-7d04c84d0078\") " pod="openshift-multus/network-metrics-daemon-mgctc"
Oct 30 12:44:28.251187 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:28.250739    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:44:28.251187 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:28.250806    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d57e941d-1795-48f2-9c7a-7d04c84d0078-metrics-certs podName:d57e941d-1795-48f2-9c7a-7d04c84d0078 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:30.250785726 +0000 UTC m=+6.056148380 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d57e941d-1795-48f2-9c7a-7d04c84d0078-metrics-certs") pod "network-metrics-daemon-mgctc" (UID: "d57e941d-1795-48f2-9c7a-7d04c84d0078") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:44:28.351578 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:28.350910    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9736332-b3de-4fda-8372-880e2196a136-original-pull-secret\") pod \"global-pull-secret-syncer-xd7lj\" (UID: \"b9736332-b3de-4fda-8372-880e2196a136\") " pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:44:28.351578 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:28.351102    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:44:28.351578 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:28.351166    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9736332-b3de-4fda-8372-880e2196a136-original-pull-secret podName:b9736332-b3de-4fda-8372-880e2196a136 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:30.351147036 +0000 UTC m=+6.156509702 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9736332-b3de-4fda-8372-880e2196a136-original-pull-secret") pod "global-pull-secret-syncer-xd7lj" (UID: "b9736332-b3de-4fda-8372-880e2196a136") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:44:28.452058 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:28.451355    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2jqh\" (UniqueName: \"kubernetes.io/projected/5f33217b-afa5-4fff-a284-5fe1be413d5c-kube-api-access-w2jqh\") pod \"network-check-target-jt6tp\" (UID: \"5f33217b-afa5-4fff-a284-5fe1be413d5c\") " pod="openshift-network-diagnostics/network-check-target-jt6tp"
Oct 30 12:44:28.452058 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:28.451548    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:44:28.452058 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:28.451569    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:44:28.452058 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:28.451582    2580 projected.go:196] Error preparing data for projected volume kube-api-access-w2jqh for pod openshift-network-diagnostics/network-check-target-jt6tp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:44:28.452058 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:28.451642    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f33217b-afa5-4fff-a284-5fe1be413d5c-kube-api-access-w2jqh podName:5f33217b-afa5-4fff-a284-5fe1be413d5c nodeName:}" failed. No retries permitted until 2026-10-30 12:44:30.451623245 +0000 UTC m=+6.256985897 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-w2jqh" (UniqueName: "kubernetes.io/projected/5f33217b-afa5-4fff-a284-5fe1be413d5c-kube-api-access-w2jqh") pod "network-check-target-jt6tp" (UID: "5f33217b-afa5-4fff-a284-5fe1be413d5c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:44:28.680441 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:28.680326    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:39:27 +0000 UTC" deadline="2028-07-31 11:58:45.614385454 +0000 UTC"
Oct 30 12:44:28.680441 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:28.680366    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15359h14m16.934024555s"
Oct 30 12:44:28.757123 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:28.757087    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgctc"
Oct 30 12:44:28.757316 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:28.757250    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgctc" podUID="d57e941d-1795-48f2-9c7a-7d04c84d0078"
Oct 30 12:44:28.757845 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:28.757818    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:44:28.757965 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:28.757947    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xd7lj" podUID="b9736332-b3de-4fda-8372-880e2196a136"
Oct 30 12:44:28.823656 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:28.823618    2580 generic.go:356] "Generic (PLEG): container finished" podID="36805beca2c15cc87891009bcced9111" containerID="003de8c9f3729006608a735e75fe63ed3672163a4cc7c43545e62b12571bc4ae" exitCode=0
Oct 30 12:44:28.824708 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:28.824677    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal" event={"ID":"36805beca2c15cc87891009bcced9111","Type":"ContainerDied","Data":"003de8c9f3729006608a735e75fe63ed3672163a4cc7c43545e62b12571bc4ae"}
Oct 30 12:44:28.866109 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:28.865838    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-38.ec2.internal" podStartSLOduration=2.865816219 podStartE2EDuration="2.865816219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:27.809596888 +0000 UTC m=+3.614959565" watchObservedRunningTime="2026-10-30 12:44:28.865816219 +0000 UTC m=+4.671178894"
Oct 30 12:44:29.754499 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:29.754409    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt6tp"
Oct 30 12:44:29.754997 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:29.754568    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt6tp" podUID="5f33217b-afa5-4fff-a284-5fe1be413d5c"
Oct 30 12:44:29.833561 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:29.833520    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal" event={"ID":"36805beca2c15cc87891009bcced9111","Type":"ContainerStarted","Data":"283bea144822dced6fd6272652a0256a02f6690eeef3340e4043badd276bcfba"}
Oct 30 12:44:30.276568 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:30.276224    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d57e941d-1795-48f2-9c7a-7d04c84d0078-metrics-certs\") pod \"network-metrics-daemon-mgctc\" (UID: \"d57e941d-1795-48f2-9c7a-7d04c84d0078\") " pod="openshift-multus/network-metrics-daemon-mgctc"
Oct 30 12:44:30.276568 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:30.276404    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:44:30.276568 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:30.276566    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d57e941d-1795-48f2-9c7a-7d04c84d0078-metrics-certs podName:d57e941d-1795-48f2-9c7a-7d04c84d0078 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:34.276466335 +0000 UTC m=+10.081828983 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d57e941d-1795-48f2-9c7a-7d04c84d0078-metrics-certs") pod "network-metrics-daemon-mgctc" (UID: "d57e941d-1795-48f2-9c7a-7d04c84d0078") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:44:30.377283 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:30.377172    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9736332-b3de-4fda-8372-880e2196a136-original-pull-secret\") pod \"global-pull-secret-syncer-xd7lj\" (UID: \"b9736332-b3de-4fda-8372-880e2196a136\") " pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:44:30.378139 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:30.377613    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:44:30.378139 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:30.377710    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9736332-b3de-4fda-8372-880e2196a136-original-pull-secret podName:b9736332-b3de-4fda-8372-880e2196a136 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:34.377684095 +0000 UTC m=+10.183046751 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9736332-b3de-4fda-8372-880e2196a136-original-pull-secret") pod "global-pull-secret-syncer-xd7lj" (UID: "b9736332-b3de-4fda-8372-880e2196a136") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:44:30.479391 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:30.478450    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2jqh\" (UniqueName: \"kubernetes.io/projected/5f33217b-afa5-4fff-a284-5fe1be413d5c-kube-api-access-w2jqh\") pod \"network-check-target-jt6tp\" (UID: \"5f33217b-afa5-4fff-a284-5fe1be413d5c\") " pod="openshift-network-diagnostics/network-check-target-jt6tp"
Oct 30 12:44:30.479391 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:30.478653    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:44:30.479391 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:30.478680    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:44:30.479391 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:30.478697    2580 projected.go:196] Error preparing data for projected volume kube-api-access-w2jqh for pod openshift-network-diagnostics/network-check-target-jt6tp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:44:30.479391 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:30.478760    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f33217b-afa5-4fff-a284-5fe1be413d5c-kube-api-access-w2jqh podName:5f33217b-afa5-4fff-a284-5fe1be413d5c nodeName:}" failed. No retries permitted until 2026-10-30 12:44:34.478739722 +0000 UTC m=+10.284102377 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-w2jqh" (UniqueName: "kubernetes.io/projected/5f33217b-afa5-4fff-a284-5fe1be413d5c-kube-api-access-w2jqh") pod "network-check-target-jt6tp" (UID: "5f33217b-afa5-4fff-a284-5fe1be413d5c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:44:30.756908 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:30.756829    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgctc"
Oct 30 12:44:30.757430 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:30.756998    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgctc" podUID="d57e941d-1795-48f2-9c7a-7d04c84d0078"
Oct 30 12:44:30.757648 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:30.757555    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:44:30.757734 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:30.757692    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xd7lj" podUID="b9736332-b3de-4fda-8372-880e2196a136"
Oct 30 12:44:30.840706 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:30.840655    2580 generic.go:356] "Generic (PLEG): container finished" podID="d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee" containerID="b0cba6a6d2adf75e1212e74825f2c39040ded3545f0d335cce3fe447d00bf5c6" exitCode=0
Oct 30 12:44:30.841921 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:30.841890    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bdjsz" event={"ID":"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee","Type":"ContainerDied","Data":"b0cba6a6d2adf75e1212e74825f2c39040ded3545f0d335cce3fe447d00bf5c6"}
Oct 30 12:44:30.917244 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:30.917160    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal" podStartSLOduration=4.917135928 podStartE2EDuration="4.917135928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:29.876486872 +0000 UTC m=+5.681849545" watchObservedRunningTime="2026-10-30 12:44:30.917135928 +0000 UTC m=+6.722498601"
Oct 30 12:44:31.754476 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:31.754440    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt6tp"
Oct 30 12:44:31.754696 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:31.754587    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt6tp" podUID="5f33217b-afa5-4fff-a284-5fe1be413d5c"
Oct 30 12:44:31.852323 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:31.851741    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9sprl" event={"ID":"abe27783-da2a-46b9-81f6-cf551787b1bc","Type":"ContainerStarted","Data":"5f1ce613200103e0a9eb07b5c07fc32a07a1661291c5939b1a45e43a03663d04"}
Oct 30 12:44:31.895958 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:31.894920    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9sprl" podStartSLOduration=2.769597913 podStartE2EDuration="6.894900592s" totalImagesPullingTime="4.125302679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:44:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:31.894574139 +0000 UTC m=+7.699936810" watchObservedRunningTime="2026-10-30 12:44:31.894900592 +0000 UTC m=+7.700263266"
Oct 30 12:44:32.756428 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:32.756395    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgctc"
Oct 30 12:44:32.756605 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:32.756549    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgctc" podUID="d57e941d-1795-48f2-9c7a-7d04c84d0078"
Oct 30 12:44:32.756978 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:32.756953    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:44:32.757072 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:32.757055    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xd7lj" podUID="b9736332-b3de-4fda-8372-880e2196a136"
Oct 30 12:44:32.859581 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:32.859507    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v9khv" event={"ID":"2eef9c4d-7079-47ad-9fad-c89c16ac69b4","Type":"ContainerStarted","Data":"f6e9b283a9b3e0c9829df74557e243aa3769640fbf897c8f7bf638e283f7d3b4"}
Oct 30 12:44:33.567448 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:33.567135    2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9sprl"
Oct 30 12:44:33.568093 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:33.567890    2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9sprl"
Oct 30 12:44:33.610345 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:33.610256    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-v9khv" podStartSLOduration=3.439698203 podStartE2EDuration="8.61023452s" totalImagesPullingTime="5.170536317s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:44:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:32.907430488 +0000 UTC m=+8.712793160" watchObservedRunningTime="2026-10-30 12:44:33.61023452 +0000 UTC m=+9.415597196"
Oct 30 12:44:33.755054 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:33.754471    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt6tp"
Oct 30 12:44:33.755054 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:33.754621    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt6tp" podUID="5f33217b-afa5-4fff-a284-5fe1be413d5c"
Oct 30 12:44:33.864913 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:33.864739    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx" event={"ID":"7bb8182c-0949-40e4-adeb-b8254e44922e","Type":"ContainerStarted","Data":"f8964e298ec430f0725e9d28d7782bb913a5fd15a6d72dd11cf2a7579406fc04"}
Oct 30 12:44:33.864913 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:33.864823    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9sprl"
Oct 30 12:44:33.868377 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:33.868114    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9sprl"
Oct 30 12:44:34.313583 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:34.313493    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d57e941d-1795-48f2-9c7a-7d04c84d0078-metrics-certs\") pod \"network-metrics-daemon-mgctc\" (UID: \"d57e941d-1795-48f2-9c7a-7d04c84d0078\") " pod="openshift-multus/network-metrics-daemon-mgctc"
Oct 30 12:44:34.313752 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:34.313612    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:44:34.313752 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:34.313677    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d57e941d-1795-48f2-9c7a-7d04c84d0078-metrics-certs podName:d57e941d-1795-48f2-9c7a-7d04c84d0078 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:42.313659016 +0000 UTC m=+18.119021668 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d57e941d-1795-48f2-9c7a-7d04c84d0078-metrics-certs") pod "network-metrics-daemon-mgctc" (UID: "d57e941d-1795-48f2-9c7a-7d04c84d0078") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:44:34.413900 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:34.413856    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9736332-b3de-4fda-8372-880e2196a136-original-pull-secret\") pod \"global-pull-secret-syncer-xd7lj\" (UID: \"b9736332-b3de-4fda-8372-880e2196a136\") " pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:44:34.414071 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:34.414058    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:44:34.414168 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:34.414127    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9736332-b3de-4fda-8372-880e2196a136-original-pull-secret podName:b9736332-b3de-4fda-8372-880e2196a136 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:42.414108213 +0000 UTC m=+18.219470881 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9736332-b3de-4fda-8372-880e2196a136-original-pull-secret") pod "global-pull-secret-syncer-xd7lj" (UID: "b9736332-b3de-4fda-8372-880e2196a136") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:44:34.516531 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:34.515052    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2jqh\" (UniqueName: \"kubernetes.io/projected/5f33217b-afa5-4fff-a284-5fe1be413d5c-kube-api-access-w2jqh\") pod \"network-check-target-jt6tp\" (UID: \"5f33217b-afa5-4fff-a284-5fe1be413d5c\") " pod="openshift-network-diagnostics/network-check-target-jt6tp"
Oct 30 12:44:34.516531 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:34.515191    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:44:34.516531 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:34.515207    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:44:34.516531 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:34.515233    2580 projected.go:196] Error preparing data for projected volume kube-api-access-w2jqh for pod openshift-network-diagnostics/network-check-target-jt6tp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:44:34.516531 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:34.515278    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f33217b-afa5-4fff-a284-5fe1be413d5c-kube-api-access-w2jqh podName:5f33217b-afa5-4fff-a284-5fe1be413d5c nodeName:}" failed. No retries permitted until 2026-10-30 12:44:42.515266539 +0000 UTC m=+18.320629185 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-w2jqh" (UniqueName: "kubernetes.io/projected/5f33217b-afa5-4fff-a284-5fe1be413d5c-kube-api-access-w2jqh") pod "network-check-target-jt6tp" (UID: "5f33217b-afa5-4fff-a284-5fe1be413d5c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:44:34.525415 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:34.525375    2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 12:44:34.692558 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:34.692417    2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T12:44:34.525390473Z","UUID":"d66c93b4-8e4a-488f-9327-f6030a616d11","Handler":null,"Name":"","Endpoint":""}
Oct 30 12:44:34.694236 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:34.694198    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 12:44:34.694370 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:34.694255    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 12:44:34.756261 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:34.756232    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:44:34.756395 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:34.756348    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xd7lj" podUID="b9736332-b3de-4fda-8372-880e2196a136"
Oct 30 12:44:34.756671 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:34.756654    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgctc"
Oct 30 12:44:34.756749 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:34.756732    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgctc" podUID="d57e941d-1795-48f2-9c7a-7d04c84d0078"
Oct 30 12:44:34.872315 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:34.872238    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx" event={"ID":"7bb8182c-0949-40e4-adeb-b8254e44922e","Type":"ContainerStarted","Data":"7328cf6a9472b28191a3b7d830e9329e83447c8ccc9b3a1d4953dd520601df80"}
Oct 30 12:44:35.753761 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:35.753726    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt6tp"
Oct 30 12:44:35.753886 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:35.753856    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt6tp" podUID="5f33217b-afa5-4fff-a284-5fe1be413d5c"
Oct 30 12:44:35.876563 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:35.876488    2580 generic.go:356] "Generic (PLEG): container finished" podID="d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee" containerID="4a7415d816c2731872e0e631e1e1c610a96624124a3f67671163761821276404" exitCode=0
Oct 30 12:44:35.876915 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:35.876578    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bdjsz" event={"ID":"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee","Type":"ContainerDied","Data":"4a7415d816c2731872e0e631e1e1c610a96624124a3f67671163761821276404"}
Oct 30 12:44:35.880209 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:35.880183    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v68vl" event={"ID":"7b607752-d51b-4781-a57f-99e95883d236","Type":"ContainerStarted","Data":"3fd4626c101f00d85fee4e6de0512ff9e76b186da20379465312ced92c3f8b92"}
Oct 30 12:44:35.881896 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:35.881861    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n8f6q" event={"ID":"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2","Type":"ContainerStarted","Data":"4cfa84456981a64b926f9dba1131729db974c54a91aba0e402d8f3d759ae6105"}
Oct 30 12:44:35.884454 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:35.884426    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx" event={"ID":"7bb8182c-0949-40e4-adeb-b8254e44922e","Type":"ContainerStarted","Data":"244974ea62ed9645869db7fafa596b6971ac147a316a1d4803721f05e06a86c2"}
Oct 30 12:44:36.020419 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:36.020360    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx" podStartSLOduration=2.7884575639999998 podStartE2EDuration="11.020342295s" totalImagesPullingTime="8.231884731s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:44:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:35.976008593 +0000 UTC m=+11.781371260" watchObservedRunningTime="2026-10-30 12:44:36.020342295 +0000 UTC m=+11.825704965"
Oct 30 12:44:36.020623 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:36.020580    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-n8f6q" podStartSLOduration=3.119809615 podStartE2EDuration="11.02056805s" totalImagesPullingTime="7.900758435s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:44:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:36.019981929 +0000 UTC m=+11.825344610" watchObservedRunningTime="2026-10-30 12:44:36.02056805 +0000 UTC m=+11.825930721"
Oct 30 12:44:36.754461 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:36.754424    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgctc"
Oct 30 12:44:36.754609 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:36.754581    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgctc" podUID="d57e941d-1795-48f2-9c7a-7d04c84d0078"
Oct 30 12:44:36.755005 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:36.754984    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:44:36.757089 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:36.755115    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xd7lj" podUID="b9736332-b3de-4fda-8372-880e2196a136"
Oct 30 12:44:36.900291 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:36.898824    2580 generic.go:356] "Generic (PLEG): container finished" podID="d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee" containerID="8e8016a8675a6ee0b98efa262607299bd2b276940e405755f81c0fc4e8bf875c" exitCode=0
Oct 30 12:44:36.900291 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:36.898879    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bdjsz" event={"ID":"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee","Type":"ContainerDied","Data":"8e8016a8675a6ee0b98efa262607299bd2b276940e405755f81c0fc4e8bf875c"}
Oct 30 12:44:36.957770 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:36.957706    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-v68vl" podStartSLOduration=3.592783194 podStartE2EDuration="11.957691099s" totalImagesPullingTime="8.364907905s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:44:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:36.057279431 +0000 UTC m=+11.862642159" watchObservedRunningTime="2026-10-30 12:44:36.957691099 +0000 UTC m=+12.763053766"
Oct 30 12:44:37.754635 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:37.754399    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt6tp"
Oct 30 12:44:37.754788 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:37.754735    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt6tp" podUID="5f33217b-afa5-4fff-a284-5fe1be413d5c"
Oct 30 12:44:37.902657 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:37.902625    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f82zx" event={"ID":"48212ac1-a400-4369-87dd-07b4d5317399","Type":"ContainerStarted","Data":"519d544416808286766c5a3d39e49ce41a07c8ee4278a4f925255e9f5fac24e4"}
Oct 30 12:44:37.904480 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:37.904449    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-79zhh" event={"ID":"940fa76d-d452-41fc-99ad-0f16f51c842e","Type":"ContainerStarted","Data":"ae40d3260c538d9f95c56b940183c02a127336418373ee2ecd8409d4c6fdd072"}
Oct 30 12:44:37.981405 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:37.981351    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-79zhh" podStartSLOduration=2.423024429 podStartE2EDuration="12.981334365s" totalImagesPullingTime="10.558309936s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:44:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:37.981281815 +0000 UTC m=+13.786644488" watchObservedRunningTime="2026-10-30 12:44:37.981334365 +0000 UTC m=+13.786697033"
Oct 30 12:44:37.981551 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:37.981520    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-f82zx" podStartSLOduration=4.477030677 podStartE2EDuration="12.981514068s" totalImagesPullingTime="8.504483391s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:44:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:37.940344529 +0000 UTC m=+13.745707196" watchObservedRunningTime="2026-10-30 12:44:37.981514068 +0000 UTC m=+13.786876735"
Oct 30 12:44:38.753883 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:38.753846    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgctc"
Oct 30 12:44:38.754077 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:38.753898    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:44:38.754077 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:38.753999    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgctc" podUID="d57e941d-1795-48f2-9c7a-7d04c84d0078"
Oct 30 12:44:38.754196 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:38.754072    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xd7lj" podUID="b9736332-b3de-4fda-8372-880e2196a136"
Oct 30 12:44:38.909054 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:38.909016    2580 generic.go:356] "Generic (PLEG): container finished" podID="d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee" containerID="c9340509c033232136b344617f71d9b83197a6c8465dc6dd7fc94e68365556d5" exitCode=0
Oct 30 12:44:38.909573 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:38.909128    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bdjsz" event={"ID":"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee","Type":"ContainerDied","Data":"c9340509c033232136b344617f71d9b83197a6c8465dc6dd7fc94e68365556d5"}
Oct 30 12:44:39.754067 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:39.754028    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt6tp"
Oct 30 12:44:39.754267 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:39.754163    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt6tp" podUID="5f33217b-afa5-4fff-a284-5fe1be413d5c"
Oct 30 12:44:40.753845 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:40.753808    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:44:40.754502 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:40.753858    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgctc"
Oct 30 12:44:40.754502 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:40.753964    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xd7lj" podUID="b9736332-b3de-4fda-8372-880e2196a136"
Oct 30 12:44:40.754502 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:40.754115    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgctc" podUID="d57e941d-1795-48f2-9c7a-7d04c84d0078"
Oct 30 12:44:40.917342 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:40.917126    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bdjsz" event={"ID":"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee","Type":"ContainerStarted","Data":"fb7f83e67a2edff97cc95a525267f920e60a53ecf68976b6004e03763d151d9e"}
Oct 30 12:44:41.754285 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:41.754253    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt6tp"
Oct 30 12:44:41.754702 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:41.754348    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt6tp" podUID="5f33217b-afa5-4fff-a284-5fe1be413d5c"
Oct 30 12:44:41.921689 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:41.921662    2580 generic.go:356] "Generic (PLEG): container finished" podID="d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee" containerID="fb7f83e67a2edff97cc95a525267f920e60a53ecf68976b6004e03763d151d9e" exitCode=0
Oct 30 12:44:41.921803 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:41.921707    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bdjsz" event={"ID":"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee","Type":"ContainerDied","Data":"fb7f83e67a2edff97cc95a525267f920e60a53ecf68976b6004e03763d151d9e"}
Oct 30 12:44:42.373590 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:42.373499    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d57e941d-1795-48f2-9c7a-7d04c84d0078-metrics-certs\") pod \"network-metrics-daemon-mgctc\" (UID: \"d57e941d-1795-48f2-9c7a-7d04c84d0078\") " pod="openshift-multus/network-metrics-daemon-mgctc"
Oct 30 12:44:42.373742 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:42.373655    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:44:42.373742 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:42.373723    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d57e941d-1795-48f2-9c7a-7d04c84d0078-metrics-certs podName:d57e941d-1795-48f2-9c7a-7d04c84d0078 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:58.373700705 +0000 UTC m=+34.179063373 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d57e941d-1795-48f2-9c7a-7d04c84d0078-metrics-certs") pod "network-metrics-daemon-mgctc" (UID: "d57e941d-1795-48f2-9c7a-7d04c84d0078") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:44:42.474674 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:42.474644    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9736332-b3de-4fda-8372-880e2196a136-original-pull-secret\") pod \"global-pull-secret-syncer-xd7lj\" (UID: \"b9736332-b3de-4fda-8372-880e2196a136\") " pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:44:42.474795 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:42.474781    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:44:42.474849 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:42.474839    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9736332-b3de-4fda-8372-880e2196a136-original-pull-secret podName:b9736332-b3de-4fda-8372-880e2196a136 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:58.474822841 +0000 UTC m=+34.280185495 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9736332-b3de-4fda-8372-880e2196a136-original-pull-secret") pod "global-pull-secret-syncer-xd7lj" (UID: "b9736332-b3de-4fda-8372-880e2196a136") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:44:42.575176 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:42.575144    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2jqh\" (UniqueName: \"kubernetes.io/projected/5f33217b-afa5-4fff-a284-5fe1be413d5c-kube-api-access-w2jqh\") pod \"network-check-target-jt6tp\" (UID: \"5f33217b-afa5-4fff-a284-5fe1be413d5c\") " pod="openshift-network-diagnostics/network-check-target-jt6tp"
Oct 30 12:44:42.575367 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:42.575312    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:44:42.575367 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:42.575330    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:44:42.575367 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:42.575339    2580 projected.go:196] Error preparing data for projected volume kube-api-access-w2jqh for pod openshift-network-diagnostics/network-check-target-jt6tp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:44:42.575458 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:42.575387    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f33217b-afa5-4fff-a284-5fe1be413d5c-kube-api-access-w2jqh podName:5f33217b-afa5-4fff-a284-5fe1be413d5c nodeName:}" failed. No retries permitted until 2026-10-30 12:44:58.575374581 +0000 UTC m=+34.380737227 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-w2jqh" (UniqueName: "kubernetes.io/projected/5f33217b-afa5-4fff-a284-5fe1be413d5c-kube-api-access-w2jqh") pod "network-check-target-jt6tp" (UID: "5f33217b-afa5-4fff-a284-5fe1be413d5c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:44:42.754539 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:42.754447    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgctc"
Oct 30 12:44:42.754996 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:42.754560    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgctc" podUID="d57e941d-1795-48f2-9c7a-7d04c84d0078"
Oct 30 12:44:42.754996 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:42.754622    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:44:42.754996 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:42.754748    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xd7lj" podUID="b9736332-b3de-4fda-8372-880e2196a136"
Oct 30 12:44:42.926834 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:42.926802    2580 generic.go:356] "Generic (PLEG): container finished" podID="d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee" containerID="23aa074fb005b07b4d4eadbbe2f595c35882d05b74ed6ed251d4709db86a7c5e" exitCode=0
Oct 30 12:44:42.926998 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:42.926842    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bdjsz" event={"ID":"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee","Type":"ContainerDied","Data":"23aa074fb005b07b4d4eadbbe2f595c35882d05b74ed6ed251d4709db86a7c5e"}
Oct 30 12:44:42.930117 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:42.930096    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt" event={"ID":"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623","Type":"ContainerStarted","Data":"26635498206a6e709c8749938e908da686de4d961f066ed667690b61eb2ed81c"}
Oct 30 12:44:42.930178 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:42.930161    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt" event={"ID":"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623","Type":"ContainerStarted","Data":"9712a4eef21cb0081df02ece57b60bae15ea9f2f2de5cf8e40d45c27a21ef5d3"}
Oct 30 12:44:42.930238 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:42.930190    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt" event={"ID":"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623","Type":"ContainerStarted","Data":"0bd242940afc158e1f5df77cab1ed5234a462db98f97dd399bf7bd2839cacad5"}
Oct 30 12:44:42.930273 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:42.930239    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt" event={"ID":"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623","Type":"ContainerStarted","Data":"55ed1d3348071642a2fb46e8f4325f2a531e77742dda4b935fa4b03514447355"}
Oct 30 12:44:42.930321 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:42.930310    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt" event={"ID":"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623","Type":"ContainerStarted","Data":"dd1a814a8214982702dedb6b616843823976fc82527696ab5a0676ac6c9be1fd"}
Oct 30 12:44:42.930351 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:42.930343    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt" event={"ID":"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623","Type":"ContainerStarted","Data":"2fdb723efa2daf7772f138be98db8445f6c7aea0853b4dfeffd9717b0e39349f"}
Oct 30 12:44:43.753871 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:43.753832    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt6tp"
Oct 30 12:44:43.754036 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:43.753937    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt6tp" podUID="5f33217b-afa5-4fff-a284-5fe1be413d5c"
Oct 30 12:44:43.935130 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:43.935096    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bdjsz" event={"ID":"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee","Type":"ContainerStarted","Data":"e35a2fb3f4f151ebfa4674d5ee62e3357507dc1e583256ea44b8836a57b7517c"}
Oct 30 12:44:44.007450 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:44.007350    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bdjsz" podStartSLOduration=8.368879982 podStartE2EDuration="19.007333369s" totalImagesPullingTime="10.638453387s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:44:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:43.995044408 +0000 UTC m=+19.800407076" watchObservedRunningTime="2026-10-30 12:44:44.007333369 +0000 UTC m=+19.812696037"
Oct 30 12:44:44.756781 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:44.756700    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:44:44.756946 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:44.756827    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xd7lj" podUID="b9736332-b3de-4fda-8372-880e2196a136"
Oct 30 12:44:44.756946 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:44.756860    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgctc"
Oct 30 12:44:44.756946 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:44.756934    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgctc" podUID="d57e941d-1795-48f2-9c7a-7d04c84d0078"
Oct 30 12:44:44.940258 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:44.940212    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt" event={"ID":"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623","Type":"ContainerStarted","Data":"4d4ec8231b0e5db2c306401b753a917603e988e47bb987f2f176c9533e3edaee"}
Oct 30 12:44:45.754211 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:45.754184    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt6tp"
Oct 30 12:44:45.754396 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:45.754297    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt6tp" podUID="5f33217b-afa5-4fff-a284-5fe1be413d5c"
Oct 30 12:44:46.754247 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:46.754151    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:44:46.754857 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:46.754286    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xd7lj" podUID="b9736332-b3de-4fda-8372-880e2196a136"
Oct 30 12:44:46.754857 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:46.754360    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgctc"
Oct 30 12:44:46.754857 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:46.754442    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgctc" podUID="d57e941d-1795-48f2-9c7a-7d04c84d0078"
Oct 30 12:44:46.946950 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:46.946916    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt" event={"ID":"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623","Type":"ContainerStarted","Data":"4ceb63eeb94c796ede3ee1d214079e3dfa30d1bce1bdb0de35726d28d7c1992f"}
Oct 30 12:44:46.947154 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:46.947135    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:46.962711 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:46.962690    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:47.038916 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:47.038823    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt" podStartSLOduration=7.173263961 podStartE2EDuration="22.038809417s" totalImagesPullingTime="14.865545456s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:44:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:47.038343797 +0000 UTC m=+22.843706456" watchObservedRunningTime="2026-10-30 12:44:47.038809417 +0000 UTC m=+22.844172085"
Oct 30 12:44:47.754166 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:47.753713    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt6tp"
Oct 30 12:44:47.754166 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:47.753819    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt6tp" podUID="5f33217b-afa5-4fff-a284-5fe1be413d5c"
Oct 30 12:44:47.949715 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:47.949676    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:47.949871 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:47.949750    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:47.964750 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:47.964725    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:44:48.492112 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:48.492047    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jt6tp"]
Oct 30 12:44:48.492271 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:48.492141    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt6tp"
Oct 30 12:44:48.492271 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:48.492214    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt6tp" podUID="5f33217b-afa5-4fff-a284-5fe1be413d5c"
Oct 30 12:44:48.494909 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:48.494885    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xd7lj"]
Oct 30 12:44:48.495014 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:48.494975    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:44:48.495062 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:48.495046    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xd7lj" podUID="b9736332-b3de-4fda-8372-880e2196a136"
Oct 30 12:44:48.495470 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:48.495452    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mgctc"]
Oct 30 12:44:48.495608 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:48.495590    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgctc"
Oct 30 12:44:48.495714 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:48.495698    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgctc" podUID="d57e941d-1795-48f2-9c7a-7d04c84d0078"
Oct 30 12:44:49.754348 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:49.754319    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgctc"
Oct 30 12:44:49.754775 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:49.754420    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgctc" podUID="d57e941d-1795-48f2-9c7a-7d04c84d0078"
Oct 30 12:44:50.569692 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:50.569656    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:50.729780 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:50.729705    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:50.753981 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:50.753955    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:44:50.754148 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:50.754119    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xd7lj" podUID="b9736332-b3de-4fda-8372-880e2196a136"
Oct 30 12:44:50.754243 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:50.754159    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt6tp"
Oct 30 12:44:50.754300 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:50.754244    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt6tp" podUID="5f33217b-afa5-4fff-a284-5fe1be413d5c"
Oct 30 12:44:51.753687 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:51.753657    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgctc"
Oct 30 12:44:51.754152 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:51.753768    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgctc" podUID="d57e941d-1795-48f2-9c7a-7d04c84d0078"
Oct 30 12:44:52.754235 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:52.754202    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:44:52.754235 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:52.754214    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt6tp"
Oct 30 12:44:52.754640 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:52.754323    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xd7lj" podUID="b9736332-b3de-4fda-8372-880e2196a136"
Oct 30 12:44:52.754640 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:52.754436    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jt6tp" podUID="5f33217b-afa5-4fff-a284-5fe1be413d5c"
Oct 30 12:44:53.753814 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:53.753786    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgctc"
Oct 30 12:44:53.754001 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:53.753906    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgctc" podUID="d57e941d-1795-48f2-9c7a-7d04c84d0078"
Oct 30 12:44:54.013468 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.013387    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:54.121459 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.121425    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:54.529869 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.529846    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeReady"
Oct 30 12:44:54.530008 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.529942    2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 12:44:54.586757 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.586700    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bgxhx"]
Oct 30 12:44:54.589050 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.589024    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bgxhx"
Oct 30 12:44:54.591345 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.591306    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xmn5t"]
Oct 30 12:44:54.595134 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.595113    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xmn5t"
Oct 30 12:44:54.604901 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.604782    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 12:44:54.605029 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.604900    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 12:44:54.605029 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.604987    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 12:44:54.605029 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.605008    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xlfj2\""
Oct 30 12:44:54.616732 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.616712    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bgxhx"]
Oct 30 12:44:54.624659 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.624638    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xmn5t"]
Oct 30 12:44:54.704625 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.704602    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 12:44:54.705344 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.705329    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 12:44:54.705415 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.705384    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mxwzk\""
Oct 30 12:44:54.754790 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.754763    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt6tp"
Oct 30 12:44:54.755031 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.755004    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:44:54.766837 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.766774    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c8d17a8-9a82-4a4f-a791-bf72f59fd756-cert\") pod \"ingress-canary-bgxhx\" (UID: \"7c8d17a8-9a82-4a4f-a791-bf72f59fd756\") " pod="openshift-ingress-canary/ingress-canary-bgxhx"
Oct 30 12:44:54.766837 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.766807    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a36ceaac-5967-4321-802e-42f4c4b37032-metrics-tls\") pod \"dns-default-xmn5t\" (UID: \"a36ceaac-5967-4321-802e-42f4c4b37032\") " pod="openshift-dns/dns-default-xmn5t"
Oct 30 12:44:54.766981 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.766903    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4v98\" (UniqueName: \"kubernetes.io/projected/7c8d17a8-9a82-4a4f-a791-bf72f59fd756-kube-api-access-d4v98\") pod \"ingress-canary-bgxhx\" (UID: \"7c8d17a8-9a82-4a4f-a791-bf72f59fd756\") " pod="openshift-ingress-canary/ingress-canary-bgxhx"
Oct 30 12:44:54.766981 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.766951    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a36ceaac-5967-4321-802e-42f4c4b37032-tmp-dir\") pod \"dns-default-xmn5t\" (UID: \"a36ceaac-5967-4321-802e-42f4c4b37032\") " pod="openshift-dns/dns-default-xmn5t"
Oct 30 12:44:54.766981 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.766968    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87d2p\" (UniqueName: \"kubernetes.io/projected/a36ceaac-5967-4321-802e-42f4c4b37032-kube-api-access-87d2p\") pod \"dns-default-xmn5t\" (UID: \"a36ceaac-5967-4321-802e-42f4c4b37032\") " pod="openshift-dns/dns-default-xmn5t"
Oct 30 12:44:54.767161 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.767143    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a36ceaac-5967-4321-802e-42f4c4b37032-config-volume\") pod \"dns-default-xmn5t\" (UID: \"a36ceaac-5967-4321-802e-42f4c4b37032\") " pod="openshift-dns/dns-default-xmn5t"
Oct 30 12:44:54.784819 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.784796    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:54.805621 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.805601    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 12:44:54.805621 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.805601    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 12:44:54.805832 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.805773    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-w75sr\""
Oct 30 12:44:54.805832 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.805788    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5fhhc\""
Oct 30 12:44:54.805901 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.805835    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 12:44:54.868077 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.868039    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a36ceaac-5967-4321-802e-42f4c4b37032-config-volume\") pod \"dns-default-xmn5t\" (UID: \"a36ceaac-5967-4321-802e-42f4c4b37032\") " pod="openshift-dns/dns-default-xmn5t"
Oct 30 12:44:54.868077 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.868070    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c8d17a8-9a82-4a4f-a791-bf72f59fd756-cert\") pod \"ingress-canary-bgxhx\" (UID: \"7c8d17a8-9a82-4a4f-a791-bf72f59fd756\") " pod="openshift-ingress-canary/ingress-canary-bgxhx"
Oct 30 12:44:54.868186 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.868088    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a36ceaac-5967-4321-802e-42f4c4b37032-metrics-tls\") pod \"dns-default-xmn5t\" (UID: \"a36ceaac-5967-4321-802e-42f4c4b37032\") " pod="openshift-dns/dns-default-xmn5t"
Oct 30 12:44:54.868186 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.868114    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4v98\" (UniqueName: \"kubernetes.io/projected/7c8d17a8-9a82-4a4f-a791-bf72f59fd756-kube-api-access-d4v98\") pod \"ingress-canary-bgxhx\" (UID: \"7c8d17a8-9a82-4a4f-a791-bf72f59fd756\") " pod="openshift-ingress-canary/ingress-canary-bgxhx"
Oct 30 12:44:54.868311 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.868185    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a36ceaac-5967-4321-802e-42f4c4b37032-tmp-dir\") pod \"dns-default-xmn5t\" (UID: \"a36ceaac-5967-4321-802e-42f4c4b37032\") " pod="openshift-dns/dns-default-xmn5t"
Oct 30 12:44:54.868311 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.868238    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87d2p\" (UniqueName: \"kubernetes.io/projected/a36ceaac-5967-4321-802e-42f4c4b37032-kube-api-access-87d2p\") pod \"dns-default-xmn5t\" (UID: \"a36ceaac-5967-4321-802e-42f4c4b37032\") " pod="openshift-dns/dns-default-xmn5t"
Oct 30 12:44:54.868435 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:54.868340    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:44:54.868435 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:54.868410    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a36ceaac-5967-4321-802e-42f4c4b37032-metrics-tls podName:a36ceaac-5967-4321-802e-42f4c4b37032 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:55.368391109 +0000 UTC m=+31.173753771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a36ceaac-5967-4321-802e-42f4c4b37032-metrics-tls") pod "dns-default-xmn5t" (UID: "a36ceaac-5967-4321-802e-42f4c4b37032") : secret "dns-default-metrics-tls" not found
Oct 30 12:44:54.868668 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.868647    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a36ceaac-5967-4321-802e-42f4c4b37032-tmp-dir\") pod \"dns-default-xmn5t\" (UID: \"a36ceaac-5967-4321-802e-42f4c4b37032\") " pod="openshift-dns/dns-default-xmn5t"
Oct 30 12:44:54.868874 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.868859    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a36ceaac-5967-4321-802e-42f4c4b37032-config-volume\") pod \"dns-default-xmn5t\" (UID: \"a36ceaac-5967-4321-802e-42f4c4b37032\") " pod="openshift-dns/dns-default-xmn5t"
Oct 30 12:44:54.871940 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.871918    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c8d17a8-9a82-4a4f-a791-bf72f59fd756-cert\") pod \"ingress-canary-bgxhx\" (UID: \"7c8d17a8-9a82-4a4f-a791-bf72f59fd756\") " pod="openshift-ingress-canary/ingress-canary-bgxhx"
Oct 30 12:44:54.886891 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.886864    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:54.891138 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.891117    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4v98\" (UniqueName: \"kubernetes.io/projected/7c8d17a8-9a82-4a4f-a791-bf72f59fd756-kube-api-access-d4v98\") pod \"ingress-canary-bgxhx\" (UID: \"7c8d17a8-9a82-4a4f-a791-bf72f59fd756\") " pod="openshift-ingress-canary/ingress-canary-bgxhx"
Oct 30 12:44:54.891210 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.891156    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87d2p\" (UniqueName: \"kubernetes.io/projected/a36ceaac-5967-4321-802e-42f4c4b37032-kube-api-access-87d2p\") pod \"dns-default-xmn5t\" (UID: \"a36ceaac-5967-4321-802e-42f4c4b37032\") " pod="openshift-dns/dns-default-xmn5t"
Oct 30 12:44:54.904068 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:54.904048    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bgxhx"
Oct 30 12:44:55.206801 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:55.206712    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bgxhx"]
Oct 30 12:44:55.212187 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:55.212162    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c8d17a8_9a82_4a4f_a791_bf72f59fd756.slice/crio-76e24688a1d436fedf2f2af4d99baf4a345cd30c5ae2918d5d1cf21da12a4af0 WatchSource:0}: Error finding container 76e24688a1d436fedf2f2af4d99baf4a345cd30c5ae2918d5d1cf21da12a4af0: Status 404 returned error can't find the container with id 76e24688a1d436fedf2f2af4d99baf4a345cd30c5ae2918d5d1cf21da12a4af0
Oct 30 12:44:55.371398 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:55.371358    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a36ceaac-5967-4321-802e-42f4c4b37032-metrics-tls\") pod \"dns-default-xmn5t\" (UID: \"a36ceaac-5967-4321-802e-42f4c4b37032\") " pod="openshift-dns/dns-default-xmn5t"
Oct 30 12:44:55.371584 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:55.371480    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:44:55.371584 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:55.371526    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a36ceaac-5967-4321-802e-42f4c4b37032-metrics-tls podName:a36ceaac-5967-4321-802e-42f4c4b37032 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:56.371514621 +0000 UTC m=+32.176877267 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a36ceaac-5967-4321-802e-42f4c4b37032-metrics-tls") pod "dns-default-xmn5t" (UID: "a36ceaac-5967-4321-802e-42f4c4b37032") : secret "dns-default-metrics-tls" not found
Oct 30 12:44:55.753557 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:55.753525    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgctc"
Oct 30 12:44:55.767901 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:55.767872    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 12:44:55.768019 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:55.767970    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zngfb\""
Oct 30 12:44:55.967804 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:55.967770    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bgxhx" event={"ID":"7c8d17a8-9a82-4a4f-a791-bf72f59fd756","Type":"ContainerStarted","Data":"76e24688a1d436fedf2f2af4d99baf4a345cd30c5ae2918d5d1cf21da12a4af0"}
Oct 30 12:44:56.380873 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:56.380834    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a36ceaac-5967-4321-802e-42f4c4b37032-metrics-tls\") pod \"dns-default-xmn5t\" (UID: \"a36ceaac-5967-4321-802e-42f4c4b37032\") " pod="openshift-dns/dns-default-xmn5t"
Oct 30 12:44:56.381348 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:56.380997    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:44:56.381348 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:56.381056    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a36ceaac-5967-4321-802e-42f4c4b37032-metrics-tls podName:a36ceaac-5967-4321-802e-42f4c4b37032 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:58.381039674 +0000 UTC m=+34.186402337 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a36ceaac-5967-4321-802e-42f4c4b37032-metrics-tls") pod "dns-default-xmn5t" (UID: "a36ceaac-5967-4321-802e-42f4c4b37032") : secret "dns-default-metrics-tls" not found
Oct 30 12:44:56.978199 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:56.978164    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:57.092439 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.092356    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:57.264902 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.264825    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-q6dlw"]
Oct 30 12:44:57.294242 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.294199    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q6dlw"]
Oct 30 12:44:57.294403 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.294338    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:44:57.329839 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.329812    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 12:44:57.329952 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.329876    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 12:44:57.330098 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.330084    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 12:44:57.331437 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.330843    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-sb2kl-jnhml"]
Oct 30 12:44:57.332530 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.332494    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:57.346053 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.346035    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-sb2kl-jnhml"
Oct 30 12:44:57.356621 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.356592    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-sb2kl-jnhml"]
Oct 30 12:44:57.384399 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.384367    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 12:44:57.384714 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.384601    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-wp9fl\""
Oct 30 12:44:57.384714 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.384698    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g92fm\""
Oct 30 12:44:57.389029 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.389001    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gxqq\" (UniqueName: \"kubernetes.io/projected/b35d2751-096e-4a78-b32c-2d15f831954c-kube-api-access-5gxqq\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:44:57.389149 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.389073    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b35d2751-096e-4a78-b32c-2d15f831954c-data-volume\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:44:57.389244 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.389203    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b35d2751-096e-4a78-b32c-2d15f831954c-crio-socket\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:44:57.389303 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.389261    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b35d2751-096e-4a78-b32c-2d15f831954c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:44:57.389400 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.389383    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b35d2751-096e-4a78-b32c-2d15f831954c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:44:57.410806 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.410786    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 12:44:57.424192 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.424174    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 12:44:57.461193 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.461159    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:57.490670 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.490637    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b35d2751-096e-4a78-b32c-2d15f831954c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:44:57.490796 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.490707    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gxqq\" (UniqueName: \"kubernetes.io/projected/b35d2751-096e-4a78-b32c-2d15f831954c-kube-api-access-5gxqq\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:44:57.490796 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.490735    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/184ffc49-7c79-41c2-b5b8-66e219e4f23e-service-ca-bundle\") pod \"periodic-gathering-sb2kl-jnhml\" (UID: \"184ffc49-7c79-41c2-b5b8-66e219e4f23e\") " pod="openshift-insights/periodic-gathering-sb2kl-jnhml"
Oct 30 12:44:57.490796 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:57.490755    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 12:44:57.490942 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.490758    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/184ffc49-7c79-41c2-b5b8-66e219e4f23e-archives-path\") pod \"periodic-gathering-sb2kl-jnhml\" (UID: \"184ffc49-7c79-41c2-b5b8-66e219e4f23e\") " pod="openshift-insights/periodic-gathering-sb2kl-jnhml"
Oct 30 12:44:57.490942 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:57.490829    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b35d2751-096e-4a78-b32c-2d15f831954c-kube-rbac-proxy-cm podName:b35d2751-096e-4a78-b32c-2d15f831954c nodeName:}" failed. No retries permitted until 2026-10-30 12:44:57.990813813 +0000 UTC m=+33.796176476 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b35d2751-096e-4a78-b32c-2d15f831954c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q6dlw" (UID: "b35d2751-096e-4a78-b32c-2d15f831954c") : configmap "kube-rbac-proxy" not found
Oct 30 12:44:57.490942 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.490865    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98j7q\" (UniqueName: \"kubernetes.io/projected/184ffc49-7c79-41c2-b5b8-66e219e4f23e-kube-api-access-98j7q\") pod \"periodic-gathering-sb2kl-jnhml\" (UID: \"184ffc49-7c79-41c2-b5b8-66e219e4f23e\") " pod="openshift-insights/periodic-gathering-sb2kl-jnhml"
Oct 30 12:44:57.490942 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.490885    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/184ffc49-7c79-41c2-b5b8-66e219e4f23e-trusted-ca-bundle\") pod \"periodic-gathering-sb2kl-jnhml\" (UID: \"184ffc49-7c79-41c2-b5b8-66e219e4f23e\") " pod="openshift-insights/periodic-gathering-sb2kl-jnhml"
Oct 30 12:44:57.491068 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.490975    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b35d2751-096e-4a78-b32c-2d15f831954c-data-volume\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:44:57.491068 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.491015    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b35d2751-096e-4a78-b32c-2d15f831954c-crio-socket\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:44:57.491068 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.491036    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b35d2751-096e-4a78-b32c-2d15f831954c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:44:57.491180 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:57.491155    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 12:44:57.491325 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.491205    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b35d2751-096e-4a78-b32c-2d15f831954c-crio-socket\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:44:57.491325 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:57.491244    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b35d2751-096e-4a78-b32c-2d15f831954c-insights-runtime-extractor-tls podName:b35d2751-096e-4a78-b32c-2d15f831954c nodeName:}" failed. No retries permitted until 2026-10-30 12:44:57.991210234 +0000 UTC m=+33.796572879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b35d2751-096e-4a78-b32c-2d15f831954c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q6dlw" (UID: "b35d2751-096e-4a78-b32c-2d15f831954c") : secret "insights-runtime-extractor-tls" not found
Oct 30 12:44:57.491437 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.491370    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b35d2751-096e-4a78-b32c-2d15f831954c-data-volume\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:44:57.519966 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.519899    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gxqq\" (UniqueName: \"kubernetes.io/projected/b35d2751-096e-4a78-b32c-2d15f831954c-kube-api-access-5gxqq\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:44:57.591779 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.591746    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/184ffc49-7c79-41c2-b5b8-66e219e4f23e-service-ca-bundle\") pod \"periodic-gathering-sb2kl-jnhml\" (UID: \"184ffc49-7c79-41c2-b5b8-66e219e4f23e\") " pod="openshift-insights/periodic-gathering-sb2kl-jnhml"
Oct 30 12:44:57.591779 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.591777    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/184ffc49-7c79-41c2-b5b8-66e219e4f23e-archives-path\") pod \"periodic-gathering-sb2kl-jnhml\" (UID: \"184ffc49-7c79-41c2-b5b8-66e219e4f23e\") " pod="openshift-insights/periodic-gathering-sb2kl-jnhml"
Oct 30 12:44:57.591976 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.591793    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98j7q\" (UniqueName: \"kubernetes.io/projected/184ffc49-7c79-41c2-b5b8-66e219e4f23e-kube-api-access-98j7q\") pod \"periodic-gathering-sb2kl-jnhml\" (UID: \"184ffc49-7c79-41c2-b5b8-66e219e4f23e\") " pod="openshift-insights/periodic-gathering-sb2kl-jnhml"
Oct 30 12:44:57.591976 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.591810    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/184ffc49-7c79-41c2-b5b8-66e219e4f23e-trusted-ca-bundle\") pod \"periodic-gathering-sb2kl-jnhml\" (UID: \"184ffc49-7c79-41c2-b5b8-66e219e4f23e\") " pod="openshift-insights/periodic-gathering-sb2kl-jnhml"
Oct 30 12:44:57.592191 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.592169    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/184ffc49-7c79-41c2-b5b8-66e219e4f23e-archives-path\") pod \"periodic-gathering-sb2kl-jnhml\" (UID: \"184ffc49-7c79-41c2-b5b8-66e219e4f23e\") " pod="openshift-insights/periodic-gathering-sb2kl-jnhml"
Oct 30 12:44:57.592407 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.592386    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/184ffc49-7c79-41c2-b5b8-66e219e4f23e-service-ca-bundle\") pod \"periodic-gathering-sb2kl-jnhml\" (UID: \"184ffc49-7c79-41c2-b5b8-66e219e4f23e\") " pod="openshift-insights/periodic-gathering-sb2kl-jnhml"
Oct 30 12:44:57.592974 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.592958    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/184ffc49-7c79-41c2-b5b8-66e219e4f23e-trusted-ca-bundle\") pod \"periodic-gathering-sb2kl-jnhml\" (UID: \"184ffc49-7c79-41c2-b5b8-66e219e4f23e\") " pod="openshift-insights/periodic-gathering-sb2kl-jnhml"
Oct 30 12:44:57.599246 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.599210    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-r8xn7"]
Oct 30 12:44:57.609035 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.609015    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-r8xn7"
Oct 30 12:44:57.615548 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.615519    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:57.621545 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.621527    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98j7q\" (UniqueName: \"kubernetes.io/projected/184ffc49-7c79-41c2-b5b8-66e219e4f23e-kube-api-access-98j7q\") pod \"periodic-gathering-sb2kl-jnhml\" (UID: \"184ffc49-7c79-41c2-b5b8-66e219e4f23e\") " pod="openshift-insights/periodic-gathering-sb2kl-jnhml"
Oct 30 12:44:57.631188 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.631169    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-r8xn7"]
Oct 30 12:44:57.631283 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.631196    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-k9mqg\""
Oct 30 12:44:57.678289 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.678268    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-sb2kl-jnhml"
Oct 30 12:44:57.690992 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.690966    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 12:44:57.690992 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.690966    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 12:44:57.761488 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.761460    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:57.792972 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.792950    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t58hp\" (UniqueName: \"kubernetes.io/projected/7fb7a7df-9a89-4cb7-9bef-bfcf542ed28e-kube-api-access-t58hp\") pod \"migrator-66df5d496b-r8xn7\" (UID: \"7fb7a7df-9a89-4cb7-9bef-bfcf542ed28e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-r8xn7"
Oct 30 12:44:57.871933 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.871900    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:57.893892 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.893868    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t58hp\" (UniqueName: \"kubernetes.io/projected/7fb7a7df-9a89-4cb7-9bef-bfcf542ed28e-kube-api-access-t58hp\") pod \"migrator-66df5d496b-r8xn7\" (UID: \"7fb7a7df-9a89-4cb7-9bef-bfcf542ed28e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-r8xn7"
Oct 30 12:44:57.919710 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.919683    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t58hp\" (UniqueName: \"kubernetes.io/projected/7fb7a7df-9a89-4cb7-9bef-bfcf542ed28e-kube-api-access-t58hp\") pod \"migrator-66df5d496b-r8xn7\" (UID: \"7fb7a7df-9a89-4cb7-9bef-bfcf542ed28e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-r8xn7"
Oct 30 12:44:57.930601 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.930576    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-r8xn7"
Oct 30 12:44:57.978944 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.978911    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bgxhx" event={"ID":"7c8d17a8-9a82-4a4f-a791-bf72f59fd756","Type":"ContainerStarted","Data":"36e2d76c11a0cbe635981a8e648d3fa8f7992f5cbd1bc62a230a9cbe692f1d7b"}
Oct 30 12:44:57.994373 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.994344    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b35d2751-096e-4a78-b32c-2d15f831954c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:44:57.994488 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:57.994458    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b35d2751-096e-4a78-b32c-2d15f831954c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:44:57.994488 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:57.994465    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 12:44:57.994587 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:57.994524    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b35d2751-096e-4a78-b32c-2d15f831954c-kube-rbac-proxy-cm podName:b35d2751-096e-4a78-b32c-2d15f831954c nodeName:}" failed. No retries permitted until 2026-10-30 12:44:58.994506905 +0000 UTC m=+34.799869562 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b35d2751-096e-4a78-b32c-2d15f831954c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q6dlw" (UID: "b35d2751-096e-4a78-b32c-2d15f831954c") : configmap "kube-rbac-proxy" not found
Oct 30 12:44:57.994587 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:57.994563    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 12:44:57.994679 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:57.994601    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b35d2751-096e-4a78-b32c-2d15f831954c-insights-runtime-extractor-tls podName:b35d2751-096e-4a78-b32c-2d15f831954c nodeName:}" failed. No retries permitted until 2026-10-30 12:44:58.994589832 +0000 UTC m=+34.799952480 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b35d2751-096e-4a78-b32c-2d15f831954c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q6dlw" (UID: "b35d2751-096e-4a78-b32c-2d15f831954c") : secret "insights-runtime-extractor-tls" not found
Oct 30 12:44:58.023738 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.023659    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bgxhx" podStartSLOduration=1.459211392 podStartE2EDuration="4.0236472s" totalImagesPullingTime="2.564435808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:44:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:58.023143352 +0000 UTC m=+33.828506019" watchObservedRunningTime="2026-10-30 12:44:58.0236472 +0000 UTC m=+33.829009868"
Oct 30 12:44:58.037430 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.037389    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-ft2cr"]
Oct 30 12:44:58.068776 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:58.068742    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod184ffc49_7c79_41c2_b5b8_66e219e4f23e.slice/crio-41f57abb74c1a669c430f0dbb87718baa6489308d0cc793d9ccb1db942687041 WatchSource:0}: Error finding container 41f57abb74c1a669c430f0dbb87718baa6489308d0cc793d9ccb1db942687041: Status 404 returned error can't find the container with id 41f57abb74c1a669c430f0dbb87718baa6489308d0cc793d9ccb1db942687041
Oct 30 12:44:58.077511 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.077487    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-ft2cr"]
Oct 30 12:44:58.077597 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.077527    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-sb2kl-jnhml"]
Oct 30 12:44:58.077647 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.077627    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-ft2cr"
Oct 30 12:44:58.094737 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.094688    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 12:44:58.094737 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.094722    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 12:44:58.094900 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.094739    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 12:44:58.094900 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.094756    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 12:44:58.094900 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.094788    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9fv4n\""
Oct 30 12:44:58.195686 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.195648    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b08436c9-9897-4b2a-b59f-40c7f26ac3c8-signing-cabundle\") pod \"service-ca-5fffc97bdf-ft2cr\" (UID: \"b08436c9-9897-4b2a-b59f-40c7f26ac3c8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ft2cr"
Oct 30 12:44:58.195828 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.195689    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b08436c9-9897-4b2a-b59f-40c7f26ac3c8-signing-key\") pod \"service-ca-5fffc97bdf-ft2cr\" (UID: \"b08436c9-9897-4b2a-b59f-40c7f26ac3c8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ft2cr"
Oct 30 12:44:58.195828 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.195710    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2bmk\" (UniqueName: \"kubernetes.io/projected/b08436c9-9897-4b2a-b59f-40c7f26ac3c8-kube-api-access-h2bmk\") pod \"service-ca-5fffc97bdf-ft2cr\" (UID: \"b08436c9-9897-4b2a-b59f-40c7f26ac3c8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ft2cr"
Oct 30 12:44:58.232452 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.232420    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-r8xn7"]
Oct 30 12:44:58.238705 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:58.238677    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fb7a7df_9a89_4cb7_9bef_bfcf542ed28e.slice/crio-503de692d00b91b8d502a7076ac6017b75167b1d394a025a0440d568beb9b66c WatchSource:0}: Error finding container 503de692d00b91b8d502a7076ac6017b75167b1d394a025a0440d568beb9b66c: Status 404 returned error can't find the container with id 503de692d00b91b8d502a7076ac6017b75167b1d394a025a0440d568beb9b66c
Oct 30 12:44:58.297027 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.296969    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b08436c9-9897-4b2a-b59f-40c7f26ac3c8-signing-cabundle\") pod \"service-ca-5fffc97bdf-ft2cr\" (UID: \"b08436c9-9897-4b2a-b59f-40c7f26ac3c8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ft2cr"
Oct 30 12:44:58.297027 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.297006    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b08436c9-9897-4b2a-b59f-40c7f26ac3c8-signing-key\") pod \"service-ca-5fffc97bdf-ft2cr\" (UID: \"b08436c9-9897-4b2a-b59f-40c7f26ac3c8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ft2cr"
Oct 30 12:44:58.297172 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.297027    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2bmk\" (UniqueName: \"kubernetes.io/projected/b08436c9-9897-4b2a-b59f-40c7f26ac3c8-kube-api-access-h2bmk\") pod \"service-ca-5fffc97bdf-ft2cr\" (UID: \"b08436c9-9897-4b2a-b59f-40c7f26ac3c8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ft2cr"
Oct 30 12:44:58.297690 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.297669    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b08436c9-9897-4b2a-b59f-40c7f26ac3c8-signing-cabundle\") pod \"service-ca-5fffc97bdf-ft2cr\" (UID: \"b08436c9-9897-4b2a-b59f-40c7f26ac3c8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ft2cr"
Oct 30 12:44:58.299352 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.299334    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b08436c9-9897-4b2a-b59f-40c7f26ac3c8-signing-key\") pod \"service-ca-5fffc97bdf-ft2cr\" (UID: \"b08436c9-9897-4b2a-b59f-40c7f26ac3c8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ft2cr"
Oct 30 12:44:58.307482 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.307461    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:58.319618 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.319601    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2bmk\" (UniqueName: \"kubernetes.io/projected/b08436c9-9897-4b2a-b59f-40c7f26ac3c8-kube-api-access-h2bmk\") pod \"service-ca-5fffc97bdf-ft2cr\" (UID: \"b08436c9-9897-4b2a-b59f-40c7f26ac3c8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ft2cr"
Oct 30 12:44:58.397802 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.397776    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a36ceaac-5967-4321-802e-42f4c4b37032-metrics-tls\") pod \"dns-default-xmn5t\" (UID: \"a36ceaac-5967-4321-802e-42f4c4b37032\") " pod="openshift-dns/dns-default-xmn5t"
Oct 30 12:44:58.398138 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.397885    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d57e941d-1795-48f2-9c7a-7d04c84d0078-metrics-certs\") pod \"network-metrics-daemon-mgctc\" (UID: \"d57e941d-1795-48f2-9c7a-7d04c84d0078\") " pod="openshift-multus/network-metrics-daemon-mgctc"
Oct 30 12:44:58.398138 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:58.397942    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:44:58.398138 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:58.397970    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 12:44:58.398138 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:58.398000    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a36ceaac-5967-4321-802e-42f4c4b37032-metrics-tls podName:a36ceaac-5967-4321-802e-42f4c4b37032 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:02.397983402 +0000 UTC m=+38.203346069 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a36ceaac-5967-4321-802e-42f4c4b37032-metrics-tls") pod "dns-default-xmn5t" (UID: "a36ceaac-5967-4321-802e-42f4c4b37032") : secret "dns-default-metrics-tls" not found
Oct 30 12:44:58.398138 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:58.398016    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d57e941d-1795-48f2-9c7a-7d04c84d0078-metrics-certs podName:d57e941d-1795-48f2-9c7a-7d04c84d0078 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:30.398009598 +0000 UTC m=+66.203372244 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d57e941d-1795-48f2-9c7a-7d04c84d0078-metrics-certs") pod "network-metrics-daemon-mgctc" (UID: "d57e941d-1795-48f2-9c7a-7d04c84d0078") : secret "metrics-daemon-secret" not found
Oct 30 12:44:58.400792 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.400770    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-ft2cr"
Oct 30 12:44:58.477980 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.477438    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:58.498689 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.498663    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9736332-b3de-4fda-8372-880e2196a136-original-pull-secret\") pod \"global-pull-secret-syncer-xd7lj\" (UID: \"b9736332-b3de-4fda-8372-880e2196a136\") " pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:44:58.500650 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.500631    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9736332-b3de-4fda-8372-880e2196a136-original-pull-secret\") pod \"global-pull-secret-syncer-xd7lj\" (UID: \"b9736332-b3de-4fda-8372-880e2196a136\") " pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:44:58.595569 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.595463    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:58.599322 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.599291    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2jqh\" (UniqueName: \"kubernetes.io/projected/5f33217b-afa5-4fff-a284-5fe1be413d5c-kube-api-access-w2jqh\") pod \"network-check-target-jt6tp\" (UID: \"5f33217b-afa5-4fff-a284-5fe1be413d5c\") " pod="openshift-network-diagnostics/network-check-target-jt6tp"
Oct 30 12:44:58.602113 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.602087    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2jqh\" (UniqueName: \"kubernetes.io/projected/5f33217b-afa5-4fff-a284-5fe1be413d5c-kube-api-access-w2jqh\") pod \"network-check-target-jt6tp\" (UID: \"5f33217b-afa5-4fff-a284-5fe1be413d5c\") " pod="openshift-network-diagnostics/network-check-target-jt6tp"
Oct 30 12:44:58.667257 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.667185    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jt6tp"
Oct 30 12:44:58.672341 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.672312    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:44:58.734755 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.734726    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-ft2cr"]
Oct 30 12:44:58.984356 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.984256    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-r8xn7" event={"ID":"7fb7a7df-9a89-4cb7-9bef-bfcf542ed28e","Type":"ContainerStarted","Data":"503de692d00b91b8d502a7076ac6017b75167b1d394a025a0440d568beb9b66c"}
Oct 30 12:44:58.986049 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.985976    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-sb2kl-jnhml" event={"ID":"184ffc49-7c79-41c2-b5b8-66e219e4f23e","Type":"ContainerStarted","Data":"41f57abb74c1a669c430f0dbb87718baa6489308d0cc793d9ccb1db942687041"}
Oct 30 12:44:58.987718 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.987675    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-ft2cr" event={"ID":"b08436c9-9897-4b2a-b59f-40c7f26ac3c8","Type":"ContainerStarted","Data":"4102dfcdc76e5ce0a289515ac852a4a0d1b2a458279518b5c53e0e846c82bd7c"}
Oct 30 12:44:58.995782 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:58.995751    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:44:59.002795 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:59.002768    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b35d2751-096e-4a78-b32c-2d15f831954c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:44:59.002911 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:59.002830    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b35d2751-096e-4a78-b32c-2d15f831954c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:44:59.002959 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:59.002915    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 12:44:59.002959 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:59.002925    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 12:44:59.003050 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:59.002986    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b35d2751-096e-4a78-b32c-2d15f831954c-kube-rbac-proxy-cm podName:b35d2751-096e-4a78-b32c-2d15f831954c nodeName:}" failed. No retries permitted until 2026-10-30 12:45:01.002966903 +0000 UTC m=+36.808329572 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b35d2751-096e-4a78-b32c-2d15f831954c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q6dlw" (UID: "b35d2751-096e-4a78-b32c-2d15f831954c") : configmap "kube-rbac-proxy" not found
Oct 30 12:44:59.003050 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:44:59.003006    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b35d2751-096e-4a78-b32c-2d15f831954c-insights-runtime-extractor-tls podName:b35d2751-096e-4a78-b32c-2d15f831954c nodeName:}" failed. No retries permitted until 2026-10-30 12:45:01.002996285 +0000 UTC m=+36.808358939 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b35d2751-096e-4a78-b32c-2d15f831954c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q6dlw" (UID: "b35d2751-096e-4a78-b32c-2d15f831954c") : secret "insights-runtime-extractor-tls" not found
Oct 30 12:44:59.005668 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:59.005644    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jt6tp"]
Oct 30 12:44:59.012420 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:59.012384    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f33217b_afa5_4fff_a284_5fe1be413d5c.slice/crio-852721cfc7b104180559ad14aac4ef585c465890e93f883a8991738a9b3e8bbf WatchSource:0}: Error finding container 852721cfc7b104180559ad14aac4ef585c465890e93f883a8991738a9b3e8bbf: Status 404 returned error can't find the container with id 852721cfc7b104180559ad14aac4ef585c465890e93f883a8991738a9b3e8bbf
Oct 30 12:44:59.031362 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:59.031286    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xd7lj"]
Oct 30 12:44:59.038743 ip-10-0-143-38 kubenswrapper[2580]: W1030 12:44:59.038714    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9736332_b3de_4fda_8372_880e2196a136.slice/crio-52d3557ffaa9dcb0c60e23002732a5fee588b9bc148de34a7d48651b523141f4 WatchSource:0}: Error finding container 52d3557ffaa9dcb0c60e23002732a5fee588b9bc148de34a7d48651b523141f4: Status 404 returned error can't find the container with id 52d3557ffaa9dcb0c60e23002732a5fee588b9bc148de34a7d48651b523141f4
Oct 30 12:44:59.995188 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:59.995155    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xd7lj" event={"ID":"b9736332-b3de-4fda-8372-880e2196a136","Type":"ContainerStarted","Data":"52d3557ffaa9dcb0c60e23002732a5fee588b9bc148de34a7d48651b523141f4"}
Oct 30 12:44:59.997099 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:44:59.997070    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jt6tp" event={"ID":"5f33217b-afa5-4fff-a284-5fe1be413d5c","Type":"ContainerStarted","Data":"852721cfc7b104180559ad14aac4ef585c465890e93f883a8991738a9b3e8bbf"}
Oct 30 12:45:01.002156 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:45:01.002118    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-r8xn7" event={"ID":"7fb7a7df-9a89-4cb7-9bef-bfcf542ed28e","Type":"ContainerStarted","Data":"82d61d504879d8d35c0dd1e119ba6eb35446bf7c327007643ec03fd70113b9a7"}
Oct 30 12:45:01.023295 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:45:01.023254    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b35d2751-096e-4a78-b32c-2d15f831954c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:45:01.023461 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:45:01.023327    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b35d2751-096e-4a78-b32c-2d15f831954c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:45:01.023461 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:45:01.023382    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 12:45:01.023461 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:45:01.023459    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b35d2751-096e-4a78-b32c-2d15f831954c-insights-runtime-extractor-tls podName:b35d2751-096e-4a78-b32c-2d15f831954c nodeName:}" failed. No retries permitted until 2026-10-30 12:45:05.023438645 +0000 UTC m=+40.828801306 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b35d2751-096e-4a78-b32c-2d15f831954c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q6dlw" (UID: "b35d2751-096e-4a78-b32c-2d15f831954c") : secret "insights-runtime-extractor-tls" not found
Oct 30 12:45:01.023621 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:45:01.023462    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 12:45:01.023621 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:45:01.023521    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b35d2751-096e-4a78-b32c-2d15f831954c-kube-rbac-proxy-cm podName:b35d2751-096e-4a78-b32c-2d15f831954c nodeName:}" failed. No retries permitted until 2026-10-30 12:45:05.023505065 +0000 UTC m=+40.828867713 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b35d2751-096e-4a78-b32c-2d15f831954c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q6dlw" (UID: "b35d2751-096e-4a78-b32c-2d15f831954c") : configmap "kube-rbac-proxy" not found
Oct 30 12:45:01.043526 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:45:01.043470    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-r8xn7" podStartSLOduration=2.397190433 podStartE2EDuration="4.043453453s" totalImagesPullingTime="1.64626302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:44:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:01.042733918 +0000 UTC m=+36.848096588" watchObservedRunningTime="2026-10-30 12:45:01.043453453 +0000 UTC m=+36.848816119"
Oct 30 12:45:01.808358 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:45:01.808323    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:02.006659 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:45:02.006579    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-sb2kl-jnhml" event={"ID":"184ffc49-7c79-41c2-b5b8-66e219e4f23e","Type":"ContainerStarted","Data":"4d78dbd1484fd3c59b6a1bd4ea09b644028395f9fc8a6872c3a61f763abe0f5d"}
Oct 30 12:45:02.046569 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:45:02.046515    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-sb2kl-jnhml" podStartSLOduration=2.03501026 podStartE2EDuration="5.046500788s" totalImagesPullingTime="3.011490528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:44:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:02.045938782 +0000 UTC m=+37.851301454" watchObservedRunningTime="2026-10-30 12:45:02.046500788 +0000 UTC m=+37.851863458"
Oct 30 12:45:02.438755 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:45:02.438686    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a36ceaac-5967-4321-802e-42f4c4b37032-metrics-tls\") pod \"dns-default-xmn5t\" (UID: \"a36ceaac-5967-4321-802e-42f4c4b37032\") " pod="openshift-dns/dns-default-xmn5t"
Oct 30 12:45:02.438927 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:45:02.438900    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:45:02.438973 ip-10-0-143-38 kubenswrapper[2580]: E1030 12:45:02.438958    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a36ceaac-5967-4321-802e-42f4c4b37032-metrics-tls podName:a36ceaac-5967-4321-802e-42f4c4b37032 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:10.438940282 +0000 UTC m=+46.244302931 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a36ceaac-5967-4321-802e-42f4c4b37032-metrics-tls") pod "dns-default-xmn5t" (UID: "a36ceaac-5967-4321-802e-42f4c4b37032") : secret "dns-default-metrics-tls" not found
Oct 30 12:45:03.012917 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:45:03.012872    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-ft2cr" event={"ID":"b08436c9-9897-4b2a-b59f-40c7f26ac3c8","Type":"ContainerStarted","Data":"28ef1ba46d19a1377d7fb16bd2e5a4a35b0597942c666a58d0155859a4287eae"}
Oct 30 12:45:03.014633 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:45:03.014604    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jt6tp" event={"ID":"5f33217b-afa5-4fff-a284-5fe1be413d5c","Type":"ContainerStarted","Data":"434ca2e5c276921553128f9e2f07ae837c3cf0d82a41aba5ee22e6ec4237989b"}
Oct 30 12:45:03.052613 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:45:03.052557    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-ft2cr" podStartSLOduration=1.453096658 podStartE2EDuration="5.052544019s" totalImagesPullingTime="3.599447361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:44:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:03.052036273 +0000 UTC m=+38.857398943" watchObservedRunningTime="2026-10-30 12:45:03.052544019 +0000 UTC m=+38.857906687"
Oct 30 12:45:03.092766 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:45:03.092709    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jt6tp" podStartSLOduration=34.237555868 podStartE2EDuration="38.092698965s" totalImagesPullingTime="3.855143097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:44:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:03.092215817 +0000 UTC m=+38.897578492" watchObservedRunningTime="2026-10-30 12:45:03.092698965 +0000 UTC m=+38.898061666"
Oct 30 12:45:03.220153 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:45:03.220069    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:03.412394 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:45:03.412354    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:03.522292 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:45:03.521666    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:03.971663 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:45:03.971559    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:04.017635 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:45:04.017604    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jt6tp"
Oct 30 12:45:04.170729 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:45:04.170705    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v68vl_7b607752-d51b-4781-a57f-99e95883d236/dns-node-resolver/0.log"
Oct 30 12:45:04.793407 ip-10-0-143-38 kubenswrapper[2580]: I1030 12:45:04.793371    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:04.876891 ip-10-0-143-38 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 12:45:04.891699 ip-10-0-143-38 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 12:45:04.891866 ip-10-0-143-38 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 12:45:04.891916 ip-10-0-143-38 systemd[1]: kubelet.service: Consumed 2.518s CPU time.
Oct 30 12:45:04.903017 ip-10-0-143-38 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 12:45:04.984098 ip-10-0-143-38 kubenswrapper[4544]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:45:04.984098 ip-10-0-143-38 kubenswrapper[4544]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 12:45:04.984098 ip-10-0-143-38 kubenswrapper[4544]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:45:04.984098 ip-10-0-143-38 kubenswrapper[4544]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:45:04.986351 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986247    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:45:04.986351 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986350    4544 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:45:04.986351 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986354    4544 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:45:04.986351 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986357    4544 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:45:04.986486 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986360    4544 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:45:04.986486 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986364    4544 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:45:04.986486 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986367    4544 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:45:04.986486 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986370    4544 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:45:04.986486 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986373    4544 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:45:04.986486 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986376    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:45:04.986486 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986379    4544 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:45:04.986486 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986382    4544 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:45:04.986486 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986384    4544 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:45:04.986486 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986387    4544 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:45:04.986486 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986390    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:45:04.986486 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986392    4544 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:45:04.986486 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986395    4544 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:45:04.986486 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986399    4544 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:45:04.986486 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986402    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:45:04.986486 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986405    4544 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:45:04.986486 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986408    4544 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:45:04.986486 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986410    4544 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:45:04.986486 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986413    4544 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:45:04.986486 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986416    4544 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:45:04.987085 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986418    4544 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:45:04.987085 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986421    4544 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:45:04.987085 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986424    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:45:04.987085 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986426    4544 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:45:04.987085 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986429    4544 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:45:04.987085 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986432    4544 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:45:04.987085 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986435    4544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:45:04.987085 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986437    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:45:04.987085 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986440    4544 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:45:04.987085 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986443    4544 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:45:04.987085 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986446    4544 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:45:04.987085 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986448    4544 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:45:04.987085 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986451    4544 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:45:04.987085 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986454    4544 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:45:04.987085 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986457    4544 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:45:04.987085 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986460    4544 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:45:04.987085 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986463    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:45:04.987085 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986465    4544 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:45:04.987085 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986468    4544 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:45:04.987615 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986471    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:45:04.987615 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986473    4544 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:45:04.987615 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986476    4544 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:45:04.987615 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986478    4544 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:45:04.987615 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986481    4544 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:45:04.987615 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986484    4544 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:45:04.987615 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986486    4544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:45:04.987615 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986491    4544 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:45:04.987615 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986495    4544 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:45:04.987615 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986500    4544 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:45:04.987615 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986504    4544 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:45:04.987615 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986509    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:45:04.987615 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986514    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:45:04.987615 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986519    4544 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:45:04.987615 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986534    4544 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:45:04.987615 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986539    4544 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:45:04.987615 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986543    4544 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:45:04.987615 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986546    4544 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:45:04.987615 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986548    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:45:04.988128 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986551    4544 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:45:04.988128 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986555    4544 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:45:04.988128 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986558    4544 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:45:04.988128 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986561    4544 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:45:04.988128 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986563    4544 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:45:04.988128 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986567    4544 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:45:04.988128 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986569    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:45:04.988128 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986572    4544 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:45:04.988128 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986576    4544 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:45:04.988128 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986579    4544 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:45:04.988128 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986582    4544 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:45:04.988128 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986584    4544 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:45:04.988128 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986587    4544 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:45:04.988128 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986590    4544 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:45:04.988128 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986593    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:45:04.988128 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986596    4544 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:45:04.988128 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986598    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:45:04.988128 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986601    4544 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:45:04.988128 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986604    4544 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:45:04.988625 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986607    4544 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:45:04.988625 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986612    4544 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:45:04.988625 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986616    4544 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:45:04.988625 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986618    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:45:04.988625 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986621    4544 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:45:04.988625 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986624    4544 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:45:04.988625 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986628    4544 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:45:04.988625 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986631    4544 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:45:04.988625 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986633    4544 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:45:04.988625 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986636    4544 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:45:04.988625 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986638    4544 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:45:04.988625 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986641    4544 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:45:04.988625 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986643    4544 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:45:04.988625 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986646    4544 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:45:04.988625 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986649    4544 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:45:04.988625 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986652    4544 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:45:04.988625 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986655    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:45:04.988625 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986666    4544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:45:04.988625 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986669    4544 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:45:04.989112 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986672    4544 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:45:04.989112 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986674    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:45:04.989112 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986677    4544 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:45:04.989112 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986680    4544 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:45:04.989112 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986683    4544 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:45:04.989112 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.986685    4544 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:45:04.989112 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987583    4544 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:45:04.989112 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987592    4544 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:45:04.989112 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987595    4544 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:45:04.989112 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987598    4544 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:45:04.989112 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987601    4544 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:45:04.989112 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987604    4544 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:45:04.989112 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987606    4544 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:45:04.989112 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987609    4544 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:45:04.989112 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987612    4544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:45:04.989112 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987618    4544 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:45:04.989112 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987621    4544 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:45:04.989112 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987624    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:45:04.989112 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987628    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:45:04.989605 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987630    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:45:04.989605 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987633    4544 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:45:04.989605 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987636    4544 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:45:04.989605 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987639    4544 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:45:04.989605 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987642    4544 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:45:04.989605 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987651    4544 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:45:04.989605 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987654    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:45:04.989605 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987657    4544 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:45:04.989605 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987660    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:45:04.989605 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987663    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:45:04.989605 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987665    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:45:04.989605 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987668    4544 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:45:04.989605 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987677    4544 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:45:04.989605 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987680    4544 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:45:04.989605 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987682    4544 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:45:04.989605 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987685    4544 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:45:04.989605 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987688    4544 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:45:04.989605 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987690    4544 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:45:04.989605 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987693    4544 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:45:04.989605 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987696    4544 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:45:04.990121 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987699    4544 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:45:04.990121 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987701    4544 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:45:04.990121 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987704    4544 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:45:04.990121 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987706    4544 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:45:04.990121 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987709    4544 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:45:04.990121 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987712    4544 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:45:04.990121 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987714    4544 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:45:04.990121 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987717    4544 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:45:04.990121 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987720    4544 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:45:04.990121 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987723    4544 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:45:04.990121 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987725    4544 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:45:04.990121 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987729    4544 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:45:04.990121 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987731    4544 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:45:04.990121 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987734    4544 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:45:04.990121 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987736    4544 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:45:04.990121 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987739    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:45:04.990121 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987742    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:45:04.990121 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987744    4544 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:45:04.990121 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987747    4544 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:45:04.990121 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987750    4544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:45:04.990646 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987754    4544 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:45:04.990646 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987758    4544 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:45:04.990646 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987763    4544 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:45:04.990646 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987766    4544 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:45:04.990646 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987771    4544 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:45:04.990646 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987783    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:45:04.990646 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987786    4544 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:45:04.990646 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987790    4544 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:45:04.990646 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987793    4544 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:45:04.990646 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987796    4544 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:45:04.990646 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987798    4544 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:45:04.990646 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987803    4544 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:45:04.990646 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987807    4544 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:45:04.990646 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987811    4544 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:45:04.990646 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987813    4544 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:45:04.990646 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987816    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:45:04.990646 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987819    4544 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:45:04.990646 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987823    4544 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:45:04.991105 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987827    4544 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:45:04.991105 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987831    4544 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:45:04.991105 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987833    4544 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:45:04.991105 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987837    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:45:04.991105 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987839    4544 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:45:04.991105 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987842    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:45:04.991105 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987845    4544 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:45:04.991105 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987848    4544 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:45:04.991105 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987850    4544 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:45:04.991105 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987853    4544 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:45:04.991105 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987856    4544 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:45:04.991105 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987858    4544 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:45:04.991105 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987862    4544 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:45:04.991105 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987865    4544 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:45:04.991105 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987868    4544 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:45:04.991105 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987870    4544 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:45:04.991105 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987873    4544 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:45:04.991105 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987876    4544 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:45:04.991105 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987878    4544 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:45:04.991105 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987881    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:45:04.991654 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987891    4544 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:45:04.991654 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987894    4544 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:45:04.991654 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987897    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:45:04.991654 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987899    4544 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:45:04.991654 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987902    4544 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:45:04.991654 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987904    4544 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:45:04.991654 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987907    4544 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:45:04.991654 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987909    4544 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:45:04.991654 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987912    4544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:45:04.991654 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987914    4544 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:45:04.991654 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987917    4544 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:45:04.991654 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987919    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:45:04.991654 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987922    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:45:04.991654 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987925    4544 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:45:04.991654 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.987927    4544 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:45:04.991654 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988545    4544 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:45:04.991654 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988552    4544 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:45:04.991654 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988556    4544 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:45:04.991654 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988559    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:45:04.991654 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988562    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:45:04.992171 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988565    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:45:04.992171 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988567    4544 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:45:04.992171 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988569    4544 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:45:04.992171 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988572    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:45:04.992171 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988575    4544 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:45:04.992171 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988578    4544 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:45:04.992171 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988582    4544 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:45:04.992171 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988586    4544 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:45:04.992171 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988589    4544 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:45:04.992171 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988592    4544 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:45:04.992171 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988595    4544 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:45:04.992171 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988615    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:45:04.992171 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988626    4544 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:45:04.992171 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988629    4544 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:45:04.992171 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988632    4544 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:45:04.992171 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988635    4544 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:45:04.992171 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988638    4544 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:45:04.992171 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988641    4544 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:45:04.992171 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988643    4544 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:45:04.992697 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988646    4544 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:45:04.992697 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988649    4544 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:45:04.992697 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988651    4544 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:45:04.992697 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988655    4544 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:45:04.992697 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988657    4544 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:45:04.992697 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988660    4544 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:45:04.992697 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988663    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:45:04.992697 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988666    4544 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:45:04.992697 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988668    4544 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:45:04.992697 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988671    4544 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:45:04.992697 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988673    4544 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:45:04.992697 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988676    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:45:04.992697 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988678    4544 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:45:04.992697 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988681    4544 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:45:04.992697 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988684    4544 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:45:04.992697 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988686    4544 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:45:04.992697 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988688    4544 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:45:04.992697 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988691    4544 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:45:04.992697 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988694    4544 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:45:04.992697 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988696    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:45:04.993215 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988701    4544 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:45:04.993215 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988703    4544 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:45:04.993215 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988706    4544 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:45:04.993215 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988714    4544 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:45:04.993215 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988717    4544 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:45:04.993215 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988720    4544 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:45:04.993215 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988729    4544 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:45:04.993215 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988731    4544 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:45:04.993215 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988734    4544 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:45:04.993215 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988737    4544 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:45:04.993215 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988739    4544 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:45:04.993215 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988742    4544 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:45:04.993215 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988744    4544 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:45:04.993215 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988747    4544 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:45:04.993215 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988749    4544 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:45:04.993215 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988752    4544 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:45:04.993215 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988755    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:45:04.993215 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988758    4544 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:45:04.993215 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988761    4544 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:45:04.993215 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988763    4544 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:45:04.993845 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988766    4544 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:45:04.993845 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988768    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:45:04.993845 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988771    4544 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:45:04.993845 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988774    4544 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:45:04.993845 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988776    4544 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:45:04.993845 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988780    4544 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:45:04.993845 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988784    4544 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:45:04.993845 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988787    4544 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:45:04.993845 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988790    4544 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:45:04.993845 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988793    4544 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:45:04.993845 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988796    4544 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:45:04.993845 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988798    4544 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:45:04.993845 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988803    4544 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:45:04.993845 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988805    4544 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:45:04.993845 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988808    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:45:04.993845 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988811    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:45:04.993845 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988813    4544 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:45:04.993845 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988816    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:45:04.993845 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988818    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:45:04.994661 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988828    4544 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:45:04.994661 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988831    4544 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:45:04.994661 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988834    4544 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:45:04.994661 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988836    4544 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:45:04.994661 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988839    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:45:04.994661 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988841    4544 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:45:04.994661 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988848    4544 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:45:04.994661 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988851    4544 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:45:04.994661 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988854    4544 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:45:04.994661 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988857    4544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:45:04.994661 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988859    4544 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:45:04.994661 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988862    4544 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:45:04.994661 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988865    4544 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:45:04.994661 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988867    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:45:04.994661 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988870    4544 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:45:04.994661 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988872    4544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:45:04.994661 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988875    4544 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:45:04.994661 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988877    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:45:04.994661 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988880    4544 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:45:04.994661 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988882    4544 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:45:04.995297 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988885    4544 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:45:04.995297 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988887    4544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:45:04.995297 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.988890    4544 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:45:04.995297 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.988990    4544 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 12:45:04.995297 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.988998    4544 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 12:45:04.995297 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989011    4544 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 12:45:04.995297 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989019    4544 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 12:45:04.995297 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989024    4544 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 12:45:04.995297 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989027    4544 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 12:45:04.995297 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989032    4544 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 12:45:04.995297 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989036    4544 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 12:45:04.995297 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989040    4544 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 12:45:04.995297 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989043    4544 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 12:45:04.995297 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989053    4544 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 12:45:04.995297 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989056    4544 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 12:45:04.995297 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989060    4544 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 12:45:04.995297 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989063    4544 flags.go:64] FLAG: --cgroup-root=""
Oct 30 12:45:04.995297 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989067    4544 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 12:45:04.995297 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989070    4544 flags.go:64] FLAG: --client-ca-file=""
Oct 30 12:45:04.995297 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989073    4544 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 12:45:04.995297 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989076    4544 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 12:45:04.995297 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989084    4544 flags.go:64] FLAG: --cluster-domain=""
Oct 30 12:45:04.995297 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989086    4544 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 12:45:04.995875 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989089    4544 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 12:45:04.995875 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989093    4544 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 12:45:04.995875 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989097    4544 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 12:45:04.995875 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989101    4544 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 12:45:04.995875 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989104    4544 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 12:45:04.995875 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989107    4544 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 12:45:04.995875 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989111    4544 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 12:45:04.995875 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989114    4544 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 12:45:04.995875 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989117    4544 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 12:45:04.995875 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989121    4544 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 12:45:04.995875 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989124    4544 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 12:45:04.995875 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989127    4544 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 12:45:04.995875 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989132    4544 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 12:45:04.995875 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989135    4544 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 12:45:04.995875 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989139    4544 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 12:45:04.995875 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989144    4544 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 12:45:04.995875 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989147    4544 flags.go:64] FLAG: --enable-server="true"
Oct 30 12:45:04.995875 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989151    4544 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 12:45:04.995875 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989158    4544 flags.go:64] FLAG: --event-burst="100"
Oct 30 12:45:04.995875 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989161    4544 flags.go:64] FLAG: --event-qps="50"
Oct 30 12:45:04.995875 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989164    4544 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 12:45:04.995875 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989167    4544 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 12:45:04.995875 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989171    4544 flags.go:64] FLAG: --eviction-hard=""
Oct 30 12:45:04.995875 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989181    4544 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 12:45:04.996521 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989185    4544 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 12:45:04.996521 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989188    4544 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 12:45:04.996521 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989191    4544 flags.go:64] FLAG: --eviction-soft=""
Oct 30 12:45:04.996521 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989195    4544 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 12:45:04.996521 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989198    4544 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 12:45:04.996521 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989201    4544 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 12:45:04.996521 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989204    4544 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 12:45:04.996521 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989207    4544 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 12:45:04.996521 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989210    4544 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 12:45:04.996521 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989213    4544 flags.go:64] FLAG: --feature-gates=""
Oct 30 12:45:04.996521 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989229    4544 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 12:45:04.996521 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989233    4544 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 12:45:04.996521 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989236    4544 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 12:45:04.996521 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989239    4544 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 12:45:04.996521 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989242    4544 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 12:45:04.996521 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989245    4544 flags.go:64] FLAG: --help="false"
Oct 30 12:45:04.996521 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989249    4544 flags.go:64] FLAG: --hostname-override="ip-10-0-143-38.ec2.internal"
Oct 30 12:45:04.996521 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989252    4544 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 12:45:04.996521 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989255    4544 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 12:45:04.996521 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989258    4544 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 12:45:04.996521 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989262    4544 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 12:45:04.996521 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989265    4544 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 12:45:04.996521 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989270    4544 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 12:45:04.996521 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989274    4544 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 12:45:04.997114 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989277    4544 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 12:45:04.997114 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989280    4544 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 12:45:04.997114 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989283    4544 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 12:45:04.997114 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989286    4544 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 12:45:04.997114 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989289    4544 flags.go:64] FLAG: --kube-reserved=""
Oct 30 12:45:04.997114 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989292    4544 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 12:45:04.997114 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989295    4544 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 12:45:04.997114 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989298    4544 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 12:45:04.997114 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989307    4544 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 12:45:04.997114 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989311    4544 flags.go:64] FLAG: --lock-file=""
Oct 30 12:45:04.997114 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989313    4544 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 12:45:04.997114 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989316    4544 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 12:45:04.997114 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989320    4544 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 12:45:04.997114 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989326    4544 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 12:45:04.997114 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989329    4544 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 12:45:04.997114 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989332    4544 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 12:45:04.997114 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989335    4544 flags.go:64] FLAG: --logging-format="text"
Oct 30 12:45:04.997114 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989338    4544 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 12:45:04.997114 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989341    4544 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 12:45:04.997114 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989344    4544 flags.go:64] FLAG: --manifest-url=""
Oct 30 12:45:04.997114 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989347    4544 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 12:45:04.997114 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989352    4544 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 12:45:04.997114 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989355    4544 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 12:45:04.997114 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989359    4544 flags.go:64] FLAG: --max-pods="110"
Oct 30 12:45:04.997114 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989362    4544 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 12:45:04.997799 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989365    4544 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 12:45:04.997799 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989368    4544 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 12:45:04.997799 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989372    4544 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 12:45:04.997799 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989375    4544 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 12:45:04.997799 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989378    4544 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 12:45:04.997799 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989382    4544 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 12:45:04.997799 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989391    4544 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 12:45:04.997799 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989394    4544 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 12:45:04.997799 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989397    4544 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 12:45:04.997799 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989400    4544 flags.go:64] FLAG: --pod-cidr=""
Oct 30 12:45:04.997799 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989403    4544 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 12:45:04.997799 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989406    4544 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 12:45:04.997799 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989409    4544 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 12:45:04.997799 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989412    4544 flags.go:64] FLAG: --port="10250"
Oct 30 12:45:04.997799 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989415    4544 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 12:45:04.997799 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989418    4544 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01f1782ea84ad690c"
Oct 30 12:45:04.997799 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989427    4544 flags.go:64] FLAG: --qos-reserved=""
Oct 30 12:45:04.997799 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989431    4544 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 12:45:04.997799 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989433    4544 flags.go:64] FLAG: --register-node="true"
Oct 30 12:45:04.997799 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989436    4544 flags.go:64] FLAG: --register-with-taints=""
Oct 30 12:45:04.997799 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989441    4544 flags.go:64] FLAG: --registry-burst="10"
Oct 30 12:45:04.997799 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989443    4544 flags.go:64] FLAG: --registry-qps="5"
Oct 30 12:45:04.997799 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989446    4544 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 12:45:04.997799 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989449    4544 flags.go:64] FLAG: --reserved-memory=""
Oct 30 12:45:04.997799 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989453    4544 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 12:45:04.997799 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989456    4544 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 12:45:04.998469 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989460    4544 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 12:45:04.998469 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989463    4544 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 12:45:04.998469 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989466    4544 flags.go:64] FLAG: --runonce="false"
Oct 30 12:45:04.998469 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989469    4544 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 12:45:04.998469 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989472    4544 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 12:45:04.998469 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989475    4544 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 12:45:04.998469 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989478    4544 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 12:45:04.998469 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989481    4544 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 12:45:04.998469 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989485    4544 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 12:45:04.998469 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989488    4544 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 12:45:04.998469 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989491    4544 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 12:45:04.998469 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989496    4544 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 12:45:04.998469 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989501    4544 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 12:45:04.998469 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989504    4544 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 12:45:04.998469 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989507    4544 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 12:45:04.998469 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989510    4544 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 12:45:04.998469 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989513    4544 flags.go:64] FLAG: --system-cgroups=""
Oct 30 12:45:04.998469 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989516    4544 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 12:45:04.998469 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989522    4544 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 12:45:04.998469 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989525    4544 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 12:45:04.998469 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989528    4544 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 12:45:04.998469 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989537    4544 flags.go:64] FLAG: --tls-min-version=""
Oct 30 12:45:04.998469 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989540    4544 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 12:45:04.998469 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989549    4544 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 12:45:04.998469 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989552    4544 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 12:45:04.999098 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989555    4544 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 12:45:04.999098 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989558    4544 flags.go:64] FLAG: --v="2"
Oct 30 12:45:04.999098 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989563    4544 flags.go:64] FLAG: --version="false"
Oct 30 12:45:04.999098 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989567    4544 flags.go:64] FLAG: --vmodule=""
Oct 30 12:45:04.999098 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989571    4544 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 12:45:04.999098 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.989574    4544 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 12:45:04.999098 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989784    4544 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:45:04.999098 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989789    4544 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:45:04.999098 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989792    4544 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:45:04.999098 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989795    4544 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:45:04.999098 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989797    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:45:04.999098 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989800    4544 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:45:04.999098 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989803    4544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:45:04.999098 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989805    4544 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:45:04.999098 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989808    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:45:04.999098 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989811    4544 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:45:04.999098 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989814    4544 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:45:04.999098 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989817    4544 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:45:04.999098 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989821    4544 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:45:04.999098 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989825    4544 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:45:04.999098 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989828    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:45:04.999638 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989830    4544 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:45:04.999638 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989834    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:45:04.999638 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989836    4544 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:45:04.999638 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989840    4544 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:45:04.999638 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989842    4544 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:45:04.999638 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989845    4544 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:45:04.999638 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989848    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:45:04.999638 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989850    4544 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:45:04.999638 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989853    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:45:04.999638 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989856    4544 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:45:04.999638 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989865    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:45:04.999638 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989868    4544 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:45:04.999638 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989871    4544 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:45:04.999638 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989874    4544 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:45:04.999638 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989877    4544 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:45:04.999638 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989879    4544 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:45:04.999638 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989882    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:45:04.999638 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989884    4544 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:45:04.999638 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989887    4544 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:45:05.000114 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989890    4544 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:45:05.000114 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989893    4544 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:45:05.000114 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989897    4544 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:45:05.000114 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989902    4544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:45:05.000114 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989906    4544 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:45:05.000114 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989921    4544 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:45:05.000114 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989928    4544 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:45:05.000114 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989931    4544 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:45:05.000114 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989934    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:45:05.000114 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989937    4544 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:45:05.000114 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989941    4544 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:45:05.000114 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989945    4544 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:45:05.000114 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989948    4544 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:45:05.000114 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989951    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:45:05.000114 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989954    4544 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:45:05.000114 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989957    4544 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:45:05.000114 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989960    4544 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:45:05.000114 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989962    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:45:05.000114 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989965    4544 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:45:05.000612 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989967    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:45:05.000612 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989970    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:45:05.000612 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989972    4544 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:45:05.000612 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989979    4544 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:45:05.000612 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989982    4544 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:45:05.000612 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989990    4544 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:45:05.000612 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989993    4544 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:45:05.000612 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989996    4544 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:45:05.000612 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.989999    4544 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:45:05.000612 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990002    4544 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:45:05.000612 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990004    4544 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:45:05.000612 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990007    4544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:45:05.000612 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990010    4544 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:45:05.000612 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990012    4544 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:45:05.000612 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990015    4544 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:45:05.000612 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990017    4544 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:45:05.000612 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990020    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:45:05.000612 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990023    4544 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:45:05.000612 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990025    4544 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:45:05.000612 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990028    4544 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:45:05.001111 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990031    4544 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:45:05.001111 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990033    4544 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:45:05.001111 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990036    4544 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:45:05.001111 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990041    4544 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:45:05.001111 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990047    4544 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:45:05.001111 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990050    4544 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:45:05.001111 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990052    4544 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:45:05.001111 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990055    4544 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:45:05.001111 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990058    4544 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:45:05.001111 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990061    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:45:05.001111 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990063    4544 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:45:05.001111 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990066    4544 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:45:05.001111 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990069    4544 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:45:05.001111 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990071    4544 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:45:05.001111 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990074    4544 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:45:05.001111 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990077    4544 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:45:05.001111 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990079    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:45:05.001111 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990082    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:45:05.001111 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990099    4544 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:45:05.001592 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990102    4544 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:45:05.001592 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990105    4544 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:45:05.001592 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990108    4544 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:45:05.001592 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990110    4544 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:45:05.001592 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990113    4544 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:45:05.001592 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990121    4544 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:45:05.001592 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990124    4544 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:45:05.001592 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990127    4544 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:45:05.001592 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990130    4544 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:45:05.001592 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990133    4544 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:45:05.001592 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990136    4544 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:45:05.001592 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990139    4544 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:45:05.001592 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990141    4544 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:45:05.001592 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990144    4544 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:45:05.001592 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.990172    4544 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:45:05.001592 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990242    4544 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:45:05.002002 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990247    4544 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:45:05.002002 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990252    4544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:45:05.002002 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990256    4544 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:45:05.002002 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990258    4544 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:45:05.002002 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990261    4544 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:45:05.002002 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990264    4544 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:45:05.002002 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990266    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:45:05.002002 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990269    4544 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:45:05.002002 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990272    4544 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:45:05.002002 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990274    4544 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:45:05.002002 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990277    4544 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:45:05.002002 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990280    4544 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:45:05.002002 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990282    4544 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:45:05.002002 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990286    4544 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:45:05.002002 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990290    4544 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:45:05.002002 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990293    4544 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:45:05.002002 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990302    4544 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:45:05.002002 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990305    4544 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:45:05.002002 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990308    4544 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:45:05.002002 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990311    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:45:05.002505 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990313    4544 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:45:05.002505 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990316    4544 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:45:05.002505 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990318    4544 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:45:05.002505 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990321    4544 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:45:05.002505 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990324    4544 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:45:05.002505 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990326    4544 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:45:05.002505 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990329    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:45:05.002505 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990332    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:45:05.002505 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990334    4544 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:45:05.002505 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990337    4544 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:45:05.002505 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990339    4544 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:45:05.002505 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990342    4544 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:45:05.002505 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990344    4544 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:45:05.002505 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990350    4544 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:45:05.002505 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990352    4544 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:45:05.002505 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990355    4544 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:45:05.002505 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990358    4544 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:45:05.002505 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990360    4544 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:45:05.002505 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990363    4544 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:45:05.002505 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990365    4544 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:45:05.002990 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990368    4544 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:45:05.002990 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990371    4544 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:45:05.002990 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990373    4544 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:45:05.002990 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990376    4544 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:45:05.002990 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990379    4544 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:45:05.002990 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990382    4544 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:45:05.002990 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990384    4544 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:45:05.002990 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990387    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:45:05.002990 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990389    4544 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:45:05.002990 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990398    4544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:45:05.002990 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990401    4544 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:45:05.002990 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990403    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:45:05.002990 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990406    4544 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:45:05.002990 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990409    4544 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:45:05.002990 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990411    4544 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:45:05.002990 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990414    4544 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:45:05.002990 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990416    4544 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:45:05.002990 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990419    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:45:05.002990 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990422    4544 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:45:05.003478 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990425    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:45:05.003478 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990427    4544 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:45:05.003478 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990430    4544 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:45:05.003478 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990433    4544 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:45:05.003478 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990435    4544 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:45:05.003478 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990438    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:45:05.003478 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990443    4544 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:45:05.003478 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990456    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:45:05.003478 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990460    4544 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:45:05.003478 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990463    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:45:05.003478 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990465    4544 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:45:05.003478 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990467    4544 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:45:05.003478 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990470    4544 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:45:05.003478 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990473    4544 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:45:05.003478 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990476    4544 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:45:05.003478 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990478    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:45:05.003478 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990481    4544 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:45:05.003478 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990483    4544 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:45:05.003478 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990486    4544 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:45:05.003947 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990489    4544 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:45:05.003947 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990491    4544 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:45:05.003947 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990494    4544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:45:05.003947 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990497    4544 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:45:05.003947 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990507    4544 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:45:05.003947 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990511    4544 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:45:05.003947 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990514    4544 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:45:05.003947 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990517    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:45:05.003947 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990519    4544 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:45:05.003947 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990522    4544 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:45:05.003947 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990525    4544 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:45:05.003947 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990527    4544 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:45:05.003947 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990531    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:45:05.003947 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990533    4544 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:45:05.003947 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990536    4544 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:45:05.003947 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990538    4544 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:45:05.003947 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990541    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:45:05.003947 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990543    4544 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:45:05.003947 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990546    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:45:05.004438 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990552    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:45:05.004438 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990554    4544 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:45:05.004438 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990557    4544 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:45:05.004438 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990560    4544 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:45:05.004438 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990562    4544 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:45:05.004438 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990565    4544 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:45:05.004438 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990567    4544 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:45:05.004438 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.990571    4544 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:45:05.004438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.995878    4544 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 12:45:05.004438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.995900    4544 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 12:45:05.004438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.995919    4544 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 12:45:05.004438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.995924    4544 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 12:45:05.004438 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.995991    4544 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:45:05.004438 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.995997    4544 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:45:05.004438 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996001    4544 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:45:05.004438 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996005    4544 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:45:05.004438 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996010    4544 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:45:05.004438 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996013    4544 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:45:05.004438 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996016    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:45:05.004438 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996019    4544 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:45:05.004946 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996021    4544 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:45:05.004946 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996024    4544 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:45:05.004946 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996027    4544 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:45:05.004946 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996030    4544 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:45:05.004946 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996033    4544 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:45:05.004946 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996036    4544 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:45:05.004946 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996039    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:45:05.004946 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996047    4544 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:45:05.004946 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996050    4544 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:45:05.004946 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996052    4544 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:45:05.004946 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996055    4544 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:45:05.004946 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996058    4544 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:45:05.004946 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996061    4544 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:45:05.004946 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996064    4544 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:45:05.004946 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996067    4544 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:45:05.004946 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996070    4544 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:45:05.004946 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996073    4544 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:45:05.004946 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996076    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:45:05.004946 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996079    4544 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:45:05.004946 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996081    4544 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:45:05.005453 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996084    4544 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:45:05.005453 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996087    4544 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:45:05.005453 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996089    4544 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:45:05.005453 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996092    4544 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:45:05.005453 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996095    4544 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:45:05.005453 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996098    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:45:05.005453 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996101    4544 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:45:05.005453 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996103    4544 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:45:05.005453 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996106    4544 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:45:05.005453 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996109    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:45:05.005453 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996112    4544 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:45:05.005453 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996115    4544 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:45:05.005453 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996117    4544 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:45:05.005453 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996120    4544 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:45:05.005453 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996123    4544 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:45:05.005453 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996125    4544 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:45:05.005453 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996128    4544 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:45:05.005453 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996131    4544 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:45:05.005453 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996139    4544 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:45:05.006205 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996142    4544 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:45:05.006205 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996145    4544 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:45:05.006205 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996148    4544 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:45:05.006205 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996150    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:45:05.006205 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996153    4544 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:45:05.006205 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996156    4544 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:45:05.006205 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996159    4544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:45:05.006205 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996162    4544 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:45:05.006205 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996165    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:45:05.006205 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996168    4544 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:45:05.006205 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996171    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:45:05.006205 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996173    4544 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:45:05.006205 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996176    4544 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:45:05.006205 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996178    4544 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:45:05.006205 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996181    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:45:05.006205 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996184    4544 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:45:05.006205 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996186    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:45:05.006205 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996189    4544 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:45:05.006205 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996191    4544 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:45:05.006205 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996194    4544 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:45:05.006726 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996197    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:45:05.006726 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996200    4544 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:45:05.006726 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996202    4544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:45:05.006726 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996205    4544 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:45:05.006726 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996208    4544 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:45:05.006726 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996211    4544 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:45:05.006726 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996213    4544 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:45:05.006726 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996216    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:45:05.006726 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996234    4544 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:45:05.006726 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996237    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:45:05.006726 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996240    4544 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:45:05.006726 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996242    4544 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:45:05.006726 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996252    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:45:05.006726 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996255    4544 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:45:05.006726 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996258    4544 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:45:05.006726 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996261    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:45:05.006726 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996264    4544 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:45:05.006726 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996267    4544 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:45:05.006726 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996270    4544 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:45:05.007187 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996272    4544 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:45:05.007187 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996275    4544 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:45:05.007187 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996278    4544 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:45:05.007187 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996280    4544 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:45:05.007187 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996284    4544 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:45:05.007187 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996288    4544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:45:05.007187 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996291    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:45:05.007187 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996294    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:45:05.007187 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996297    4544 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:45:05.007187 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996299    4544 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:45:05.007187 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996302    4544 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:45:05.007187 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996305    4544 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:45:05.007187 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996307    4544 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:45:05.007187 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996310    4544 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:45:05.007187 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996312    4544 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:45:05.007187 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996315    4544 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:45:05.007187 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996318    4544 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:45:05.007187 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996320    4544 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:45:05.007187 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996323    4544 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:45:05.007709 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996325    4544 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:45:05.007709 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.996356    4544 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:45:05.007709 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996524    4544 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:45:05.007709 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996529    4544 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:45:05.007709 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996533    4544 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:45:05.007709 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996536    4544 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:45:05.007709 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996540    4544 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:45:05.007709 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996544    4544 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:45:05.007709 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996548    4544 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:45:05.007709 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996550    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:45:05.007709 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996553    4544 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:45:05.007709 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996558    4544 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:45:05.007709 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996562    4544 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:45:05.007709 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996565    4544 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:45:05.007709 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996568    4544 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:45:05.007709 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996571    4544 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:45:05.008109 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996573    4544 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:45:05.008109 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996576    4544 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:45:05.008109 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996579    4544 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:45:05.008109 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996582    4544 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:45:05.008109 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996584    4544 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:45:05.008109 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996587    4544 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:45:05.008109 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996590    4544 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:45:05.008109 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996592    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:45:05.008109 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996595    4544 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:45:05.008109 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996598    4544 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:45:05.008109 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996600    4544 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:45:05.008109 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996603    4544 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:45:05.008109 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996606    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:45:05.008109 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996609    4544 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:45:05.008109 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996611    4544 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:45:05.008109 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996614    4544 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:45:05.008109 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996617    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:45:05.008109 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996620    4544 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:45:05.008109 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996622    4544 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:45:05.008109 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996625    4544 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:45:05.008617 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996628    4544 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:45:05.008617 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996631    4544 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:45:05.008617 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996634    4544 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:45:05.008617 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996636    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:45:05.008617 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996639    4544 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:45:05.008617 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996642    4544 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:45:05.008617 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996645    4544 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:45:05.008617 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996648    4544 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:45:05.008617 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996651    4544 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:45:05.008617 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996653    4544 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:45:05.008617 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996656    4544 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:45:05.008617 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996658    4544 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:45:05.008617 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996661    4544 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:45:05.008617 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996664    4544 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:45:05.008617 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996667    4544 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:45:05.008617 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996669    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:45:05.008617 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996672    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:45:05.008617 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996675    4544 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:45:05.008617 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996677    4544 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:45:05.008617 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996680    4544 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:45:05.009138 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996682    4544 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:45:05.009138 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996685    4544 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:45:05.009138 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996687    4544 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:45:05.009138 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996690    4544 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:45:05.009138 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996693    4544 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:45:05.009138 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996695    4544 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:45:05.009138 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996698    4544 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:45:05.009138 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996701    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:45:05.009138 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996703    4544 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:45:05.009138 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996706    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:45:05.009138 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996708    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:45:05.009138 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996711    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:45:05.009138 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996714    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:45:05.009138 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996717    4544 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:45:05.009138 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996719    4544 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:45:05.009138 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996722    4544 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:45:05.009138 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996724    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:45:05.009138 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996728    4544 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:45:05.009138 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996730    4544 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:45:05.009616 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996733    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:45:05.009616 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996735    4544 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:45:05.009616 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996738    4544 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:45:05.009616 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996740    4544 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:45:05.009616 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996743    4544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:45:05.009616 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996746    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:45:05.009616 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996748    4544 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:45:05.009616 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996751    4544 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:45:05.009616 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996753    4544 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:45:05.009616 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996756    4544 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:45:05.009616 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996759    4544 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:45:05.009616 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996761    4544 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:45:05.009616 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996764    4544 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:45:05.009616 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996767    4544 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:45:05.009616 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996771    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:45:05.009616 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996774    4544 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:45:05.009616 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996777    4544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:45:05.009616 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996779    4544 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:45:05.009616 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996782    4544 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:45:05.010093 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996785    4544 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:45:05.010093 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996787    4544 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:45:05.010093 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996790    4544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:45:05.010093 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996792    4544 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:45:05.010093 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996795    4544 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:45:05.010093 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996798    4544 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:45:05.010093 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996800    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:45:05.010093 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996803    4544 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:45:05.010093 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996806    4544 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:45:05.010093 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996808    4544 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:45:05.010093 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996811    4544 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:45:05.010093 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996813    4544 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:45:05.010093 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996816    4544 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:45:05.010093 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996819    4544 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:45:05.010093 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.996843    4544 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:45:05.010093 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996877    4544 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:45:05.010729 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996881    4544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:45:05.010729 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996884    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:45:05.010729 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996887    4544 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:45:05.010729 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996890    4544 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:45:05.010729 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996893    4544 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:45:05.010729 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996896    4544 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:45:05.010729 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996899    4544 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:45:05.010729 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996901    4544 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:45:05.010729 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996904    4544 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:45:05.010729 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996908    4544 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:45:05.010729 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996912    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:45:05.010729 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996915    4544 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:45:05.010729 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996918    4544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:45:05.010729 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996922    4544 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:45:05.010729 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996924    4544 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:45:05.010729 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996927    4544 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:45:05.010729 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996930    4544 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:45:05.010729 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996932    4544 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:45:05.010729 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996935    4544 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:45:05.011186 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996938    4544 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:45:05.011186 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996940    4544 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:45:05.011186 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996943    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:45:05.011186 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996946    4544 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:45:05.011186 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996948    4544 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:45:05.011186 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996951    4544 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:45:05.011186 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996957    4544 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:45:05.011186 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996959    4544 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:45:05.011186 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996962    4544 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:45:05.011186 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996965    4544 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:45:05.011186 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996968    4544 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:45:05.011186 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996971    4544 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:45:05.011186 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996973    4544 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:45:05.011186 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996976    4544 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:45:05.011186 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996978    4544 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:45:05.011186 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996981    4544 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:45:05.011186 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996983    4544 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:45:05.011186 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996986    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:45:05.011186 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996989    4544 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:45:05.011186 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996992    4544 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:45:05.011672 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.996997    4544 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:45:05.011672 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997000    4544 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:45:05.011672 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997004    4544 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:45:05.011672 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997007    4544 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:45:05.011672 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997009    4544 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:45:05.011672 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997013    4544 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:45:05.011672 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997016    4544 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:45:05.011672 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997019    4544 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:45:05.011672 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997021    4544 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:45:05.011672 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997024    4544 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:45:05.011672 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997027    4544 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:45:05.011672 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997029    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:45:05.011672 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997032    4544 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:45:05.011672 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997035    4544 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:45:05.011672 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997038    4544 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:45:05.011672 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997040    4544 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:45:05.011672 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997043    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:45:05.011672 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997045    4544 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:45:05.011672 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997050    4544 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:45:05.011672 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997052    4544 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:45:05.012142 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997055    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:45:05.012142 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997058    4544 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:45:05.012142 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997060    4544 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:45:05.012142 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997064    4544 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:45:05.012142 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997067    4544 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:45:05.012142 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997070    4544 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:45:05.012142 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997072    4544 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:45:05.012142 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997075    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:45:05.012142 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997078    4544 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:45:05.012142 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997081    4544 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:45:05.012142 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997083    4544 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:45:05.012142 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997086    4544 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:45:05.012142 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997088    4544 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:45:05.012142 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997091    4544 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:45:05.012142 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997093    4544 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:45:05.012142 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997096    4544 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:45:05.012142 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997098    4544 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:45:05.012142 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997103    4544 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:45:05.012142 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997106    4544 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:45:05.012142 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997109    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:45:05.012643 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997111    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:45:05.012643 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997114    4544 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:45:05.012643 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997117    4544 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:45:05.012643 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997120    4544 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:45:05.012643 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997123    4544 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:45:05.012643 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997125    4544 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:45:05.012643 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997128    4544 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:45:05.012643 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997131    4544 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:45:05.012643 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997134    4544 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:45:05.012643 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997137    4544 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:45:05.012643 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997141    4544 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:45:05.012643 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997143    4544 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:45:05.012643 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997146    4544 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:45:05.012643 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997148    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:45:05.012643 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997151    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:45:05.012643 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997155    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:45:05.012643 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997158    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:45:05.012643 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997161    4544 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:45:05.012643 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997163    4544 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:45:05.013106 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997166    4544 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:45:05.013106 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997169    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:45:05.013106 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997171    4544 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:45:05.013106 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997174    4544 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:45:05.013106 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997176    4544 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:45:05.013106 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997179    4544 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:45:05.013106 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:04.997182    4544 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:45:05.013106 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.997281    4544 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 12:45:05.013106 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.998322    4544 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 12:45:05.013106 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.998388    4544 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 12:45:05.013106 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.998581    4544 server.go:1014] "Starting client certificate rotation"
Oct 30 12:45:05.013106 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.998763    4544 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 12:45:05.013106 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.998818    4544 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 12:39:25 +0000 UTC" deadline="2028-06-22 18:45:18.439735694 +0000 UTC"
Oct 30 12:45:05.013106 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.998855    4544 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14430h0m13.440882348s"
Oct 30 12:45:05.013106 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:04.999073    4544 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 12:45:05.018003 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.017972    4544 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 12:45:05.019572 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.019553    4544 log.go:25] "Validated CRI v1 runtime API"
Oct 30 12:45:05.021163 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.021144    4544 log.go:25] "Validated CRI v1 image API"
Oct 30 12:45:05.021969 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.021953    4544 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 12:45:05.023983 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.023950    4544 fs.go:135] Filesystem UUIDs: map[207b088b-cfd2-47c4-a126-f0a601e2084d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e259b42c-307e-4cbc-84c2-9dbd799b6da3:/dev/nvme0n1p4]
Oct 30 12:45:05.024155 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.023985    4544 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1712802f314501d149a03ffad2781f32b71ddf66a385dd3026d1ffe2585c8157/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1712802f314501d149a03ffad2781f32b71ddf66a385dd3026d1ffe2585c8157/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2bc309e11242418011070b5c2efea6b00aa043f633edc74ae4d4b9631abca96d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2bc309e11242418011070b5c2efea6b00aa043f633edc74ae4d4b9631abca96d/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2db7f273c7518b0745c5e8bae9f6e47ed93e507e118eda0fab3419ec56a49269/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2db7f273c7518b0745c5e8bae9f6e47ed93e507e118eda0fab3419ec56a49269/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4102dfcdc76e5ce0a289515ac852a4a0d1b2a458279518b5c53e0e846c82bd7c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4102dfcdc76e5ce0a289515ac852a4a0d1b2a458279518b5c53e0e846c82bd7c/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/41f57abb74c1a669c430f0dbb87718baa6489308d0cc793d9ccb1db942687041/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/41f57abb74c1a669c430f0dbb87718baa6489308d0cc793d9ccb1db942687041/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/503de692d00b91b8d502a7076ac6017b75167b1d394a025a0440d568beb9b66c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/503de692d00b91b8d502a7076ac6017b75167b1d394a025a0440d568beb9b66c/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/52d3557ffaa9dcb0c60e23002732a5fee588b9bc148de34a7d48651b523141f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/52d3557ffaa9dcb0c60e23002732a5fee588b9bc148de34a7d48651b523141f4/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/696affb719b735bd04e03d28e25d2958829aeaff062c9fe97e096a2b8197acbd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/696affb719b735bd04e03d28e25d2958829aeaff062c9fe97e096a2b8197acbd/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/73c9689bdccb305b8354b2a73384d8f7fe052a3fbbc05fd6f2a3346267cf689b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/73c9689bdccb305b8354b2a73384d8f7fe052a3fbbc05fd6f2a3346267cf689b/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/76e24688a1d436fedf2f2af4d99baf4a345cd30c5ae2918d5d1cf21da12a4af0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/76e24688a1d436fedf2f2af4d99baf4a345cd30c5ae2918d5d1cf21da12a4af0/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80aa64600269b92137dd94b6ffe57210ba58dc9cf999790ad098f9f7502399d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80aa64600269b92137dd94b6ffe57210ba58dc9cf999790ad098f9f7502399d9/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/852721cfc7b104180559ad14aac4ef585c465890e93f883a8991738a9b3e8bbf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/852721cfc7b104180559ad14aac4ef585c465890e93f883a8991738a9b3e8bbf/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a98741e4e980f2331f8ad5c157a7062e44ab9d43ad76901e9f5e1473c7878a75/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a98741e4e980f2331f8ad5c157a7062e44ab9d43ad76901e9f5e1473c7878a75/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ada0dc5751c1b2c179d3403d8ebca87eb8e0f244a186544542c715f38395c108/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ada0dc5751c1b2c179d3403d8ebca87eb8e0f244a186544542c715f38395c108/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ccba6cd87c2d004d3980a01255dde1f600c6883887a957ee1ebb161c1c73df5e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ccba6cd87c2d004d3980a01255dde1f600c6883887a957ee1ebb161c1c73df5e/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e0008197a374974d904d3d54eccbdb31bb9ee232b6ec21ada276f6249614ca2a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e0008197a374974d904d3d54eccbdb31bb9ee232b6ec21ada276f6249614ca2a/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623/volumes/kubernetes.io~projected/kube-api-access-nkk44:{mountpoint:/var/lib/kubelet/pods/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623/volumes/kubernetes.io~projected/kube-api-access-nkk44 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/184ffc49-7c79-41c2-b5b8-66e219e4f23e/volumes/kubernetes.io~projected/kube-api-access-98j7q:{mountpoint:/var/lib/kubelet/pods/184ffc49-7c79-41c2-b5b8-66e219e4f23e/volumes/kubernetes.io~projected/kube-api-access-98j7q major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2eef9c4d-7079-47ad-9fad-c89c16ac69b4/volumes/kubernetes.io~projected/kube-api-access-kfhxh:{mountpoint:/var/lib/kubelet/pods/2eef9c4d-7079-47ad-9fad-c89c16ac69b4/volumes/kubernetes.io~projected/kube-api-access-kfhxh major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48212ac1-a400-4369-87dd-07b4d5317399/volumes/kubernetes.io~projected/kube-api-access-ldcfg:{mountpoint:/var/lib/kubelet/pods/48212ac1-a400-4369-87dd-07b4d5317399/volumes/kubernetes.io~projected/kube-api-access-ldcfg major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f33217b-afa5-4fff-a284-5fe1be413d5c/volumes/kubernetes.io~projected/kube-api-access-w2jqh:{mountpoint:/var/lib/kubelet/pods/5f33217b-afa5-4fff-a284-5fe1be413d5c/volumes/kubernetes.io~projected/kube-api-access-w2jqh major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b607752-d51b-4781-a57f-99e95883d236/volumes/kubernetes.io~projected/kube-api-access-ncgsv:{mountpoint:/var/lib/kubelet/pods/7b607752-d51b-4781-a57f-99e95883d236/volumes/kubernetes.io~projected/kube-api-access-ncgsv major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7bb8182c-0949-40e4-adeb-b8254e44922e/volumes/kubernetes.io~projected/kube-api-access-jlrgs:{mountpoint:/var/lib/kubelet/pods/7bb8182c-0949-40e4-adeb-b8254e44922e/volumes/kubernetes.io~projected/kube-api-access-jlrgs major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c8d17a8-9a82-4a4f-a791-bf72f59fd756/volumes/kubernetes.io~projected/kube-api-access-d4v98:{mountpoint:/var/lib/kubelet/pods/7c8d17a8-9a82-4a4f-a791-bf72f59fd756/volumes/kubernetes.io~projected/kube-api-access-d4v98 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c8d17a8-9a82-4a4f-a791-bf72f59fd756/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/7c8d17a8-9a82-4a4f-a791-bf72f59fd756/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7fb7a7df-9a89-4cb7-9bef-bfcf542ed28e/volumes/kubernetes.io~projected/kube-api-access-t58hp:{mountpoint:/var/lib/kubelet/pods/7fb7a7df-9a89-4cb7-9bef-bfcf542ed28e/volumes/kubernetes.io~projected/kube-api-access-t58hp major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2/volumes/kubernetes.io~projected/kube-api-access-xffpq:{mountpoint:/var/lib/kubelet/pods/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2/volumes/kubernetes.io~projected/kube-api-access-xffpq major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/940fa76d-d452-41fc-99ad-0f16f51c842e/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/940fa76d-d452-41fc-99ad-0f16f51c842e/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/940fa76d-d452-41fc-99ad-0f16f51c842e/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/940fa76d-d452-41fc-99ad-0f16f51c842e/volumes/kubernetes.io~empty-dir/tmp major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/940fa76d-d452-41fc-99ad-0f16f51c842e/volumes/kubernetes.io~projected/kube-api-access-5z5dz:{mountpoint:/var/lib/kubelet/pods/940fa76d-d452-41fc-99ad-0f16f51c842e/volumes/kubernetes.io~projected/kube-api-access-5z5dz major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a36ceaac-5967-4321-802e-42f4c4b37032/volumes/kubernetes.io~projected/kube-api-access-87d2p:{mountpoint:/var/lib/kubelet/pods/a36ceaac-5967-4321-802e-42f4c4b37032/volumes/kubernetes.io~projected/kube-api-access-87d2p major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abe27783-da2a-46b9-81f6-cf551787b1bc/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/abe27783-da2a-46b9-81f6-cf551787b1bc/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b08436c9-9897-4b2a-b59f-40c7f26ac3c8/volumes/kubernetes.io~projected/kube-api-access-h2bmk:{mountpoint:/var/lib/kubelet/pods/b08436c9-9897-4b2a-b59f-40c7f26ac3c8/volumes/kubernetes.io~projected/kube-api-access-h2bmk major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b08436c9-9897-4b2a-b59f-40c7f26ac3c8/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/b08436c9-9897-4b2a-b59f-40c7f26ac3c8/volumes/kubernetes.io~secret/signing-key major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b35d2751-096e-4a78-b32c-2d15f831954c/volumes/kubernetes.io~projected/kube-api-access-5gxqq:{mountpoint:/var/lib/kubelet/pods/b35d2751-096e-4a78-b32c-2d15f831954c/volumes/kubernetes.io~projected/kube-api-access-5gxqq major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9736332-b3de-4fda-8372-880e2196a136/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/b9736332-b3de-4fda-8372-880e2196a136/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d57e941d-1795-48f2-9c7a-7d04c84d0078/volumes/kubernetes.io~projected/kube-api-access-l42fz:{mountpoint:/var/lib/kubelet/pods/d57e941d-1795-48f2-9c7a-7d04c84d0078/volumes/kubernetes.io~projected/kube-api-access-l42fz major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee/volumes/kubernetes.io~projected/kube-api-access-p8ns9:{mountpoint:/var/lib/kubelet/pods/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee/volumes/kubernetes.io~projected/kube-api-access-p8ns9 major:0 minor:60 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/7adf266580c4564f96803326d738e891ce76594ae90c862b35f148129997e8aa/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/f9cdf9b9409a9f2c0ec0befd36ac61721f4fc0f7309a3e8a61a5ef4ee97c05f0/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/0c04ac5295a89c663de67b169c64733008f04717f3859cfc7528c7c55bd8baa1/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/51fc16c2622530202a438d11caabdef602a27f2fc79bb2e3f4f9a78974b36916/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/8c7424780944d6c36c75ff2337295a5be7f7b9fe64012c4da8a15e6ef83943a8/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/ca6025b63777aa4a7000c3f6099f959a00e32dfae62c90b1f2152320d5b488a2/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/a65c9855e2cda90beeb47ae1512250011efff3d63cdffd653eb317ad5502b078/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/8649461e4b60276dd879e2badae730a9a573c715d8276de546f201a9864e8c65/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/d9732008dfb32f57b91666c4f4b37b9e9d3f6dbe6c3ae4c1c0e3116722b22b70/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/c795df2e22c36a7cd2157587e32d044d17912017f6ce280df3a99d5b9daa90b2/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/b880078bd47d66e5f7a7e8577f3a8a5d3d9fc83cdc0fb3b4a32761a37bbe663d/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/9e795a7d9295d94744aa58d2250d3e9e2421df21fbec7952694778ab0311371e/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/bd527e60bc3873d6b84b809148322b4edf0cb90a12a80dca2d3b59675f60b2f4/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/8dad2de6f8c9d4fb994dfe349338d68dcc5067ae5bc2e9a9c0347d1c36b42b79/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-163:{mountpoint:/var/lib/containers/storage/overlay/c1b92df7b61be3d1e79672afec0990735d0551a6268701fd5007e607c1eb41da/merged major:0 minor:163 fsType:overlay blockSize:0} overlay_0-165:{mountpoint:/var/lib/containers/storage/overlay/603e024699e486bb3e7b896ecd7588e992564cababb46b8778dd377e143395ca/merged major:0 minor:165 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/3b389e7915f2db05c403fb0e7240f268e0c7a8820e0fb8b3776bb2f2c6ec7822/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/a2ba31f453ca2676e3876f933fdc9971067eff5519d019e64cdc7b538b62cd2e/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/f6b7acb2b7f7d51d5d7966e2ae04ca5295a5fb93ac84e0ee65d6c2781a912977/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/e81a27f74dc7846b6e94ce2abdc7c767af2c5cc972c688cc78bc863052beed7e/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/e5fe0c6fe42894002a99e1436c728493b46e17c615061f9351d0a85348dbe1de/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/64e129821a36ba34f705c428ff496c6c8538fd145c418647b6b72191896452f8/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/64269d15b792312e7f4ee877d7279b9fc2af40dd443dadb09cfcf6b39eecea4e/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/1f799911a85e65473e232f3f563b199b905434f63e02e3be562b081b7e463bcd/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/ba073c7692c06b8702b1cc9bc507e89d2037a482427266a70ab99035c7528db3/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/400cb90ba2b7601524b98a1567d85dc9613402de9d704b5439001b7000bc6735/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/4b38bdfb785c35811e5203d9bc16dcd1177bb67df590a333088ace0b832669c4/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/710d1c094a116e4a1bf4df79858bbadfcb3c0756961eb96a03de50aa8edb0331/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/a460e5ce64b666fa0755f5db23666926cf0244b9983a6597ebfde3ca40be7a35/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/5e0d65a513a1c281877f19a8b046fcfe4303a1a1ccf852c7190cb19d7aff7562/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/d67f2bbb096319fc33de6ebc99aa4350a19ef177f54b2d1ed79c857bf86d60e3/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/f7a743dd5abccf4a2024fabed33ccf0ce32b48bac666f44d65f25f5d5956d5f5/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/6be4a4090366e5edd7d8ce9a0f6397ef5e4a8beead36796c4dd2da344f4e681c/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/8c8c0ba3376716554f02c916dc279288a0b9fd6b6c7683f193868495af75ba4d/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/6b5ef333c779587863a5b6e619e16fba0c69982cf5d699af1f4a9b114272603b/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/222878fea426826eb4653a57e74318f3825cdcde783b2ff941d88d81fc42aa45/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/471057615b54024b68e5f8ff1d339f44607399ebe6520b516093b18448011ae3/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/f26db405d12d3d7db84ae6b9281a3b8eefa0779b940348224d608f2bcae0a969/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/3bae98051ffad8ba7122f873af6d4a38df916a6542db9f89546312e040d01201/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/43ea536c3107f6830cd7b64029a7125fe106929d3f8803227ed151d0ecd79ff1/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/a1f3b497f73afc4b1151dae36c85c3884ea123b619a55c3555125c453656f991/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/2b9cfacaa8201bd771e85eaf158c91c56dae4261545f7502b93ca1aa96f04213/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/e9853917579d436b09d1a441a1e9faece686785d21f157f96a9e591bfa880796/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 12:45:05.031896 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.031634    4544 manager.go:217] Machine: {Timestamp:2026-10-30 12:45:05.031147488 +0000 UTC m=+0.078281433 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104416 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2eceb28fd290b835c4179577a50dd6 SystemUUID:ec2eceb2-8fd2-90b8-35c4-179577a50dd6 BootID:7f0d79fc-cd47-41be-b9d6-3dc3a093680c Filesystems:[{Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b35d2751-096e-4a78-b32c-2d15f831954c/volumes/kubernetes.io~projected/kube-api-access-5gxqq DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/852721cfc7b104180559ad14aac4ef585c465890e93f883a8991738a9b3e8bbf/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/48212ac1-a400-4369-87dd-07b4d5317399/volumes/kubernetes.io~projected/kube-api-access-ldcfg DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/41f57abb74c1a669c430f0dbb87718baa6489308d0cc793d9ccb1db942687041/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7c8d17a8-9a82-4a4f-a791-bf72f59fd756/volumes/kubernetes.io~projected/kube-api-access-d4v98 DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2/volumes/kubernetes.io~projected/kube-api-access-xffpq DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/184ffc49-7c79-41c2-b5b8-66e219e4f23e/volumes/kubernetes.io~projected/kube-api-access-98j7q DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5f33217b-afa5-4fff-a284-5fe1be413d5c/volumes/kubernetes.io~projected/kube-api-access-w2jqh DeviceMajor:0 DeviceMinor:228 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a98741e4e980f2331f8ad5c157a7062e44ab9d43ad76901e9f5e1473c7878a75/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b08436c9-9897-4b2a-b59f-40c7f26ac3c8/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ada0dc5751c1b2c179d3403d8ebca87eb8e0f244a186544542c715f38395c108/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/abe27783-da2a-46b9-81f6-cf551787b1bc/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2bc309e11242418011070b5c2efea6b00aa043f633edc74ae4d4b9631abca96d/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623/volumes/kubernetes.io~projected/kube-api-access-nkk44 DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4102dfcdc76e5ce0a289515ac852a4a0d1b2a458279518b5c53e0e846c82bd7c/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/940fa76d-d452-41fc-99ad-0f16f51c842e/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee/volumes/kubernetes.io~projected/kube-api-access-p8ns9 DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80aa64600269b92137dd94b6ffe57210ba58dc9cf999790ad098f9f7502399d9/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d57e941d-1795-48f2-9c7a-7d04c84d0078/volumes/kubernetes.io~projected/kube-api-access-l42fz DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2eef9c4d-7079-47ad-9fad-c89c16ac69b4/volumes/kubernetes.io~projected/kube-api-access-kfhxh DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7bb8182c-0949-40e4-adeb-b8254e44922e/volumes/kubernetes.io~projected/kube-api-access-jlrgs DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ccba6cd87c2d004d3980a01255dde1f600c6883887a957ee1ebb161c1c73df5e/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/73c9689bdccb305b8354b2a73384d8f7fe052a3fbbc05fd6f2a3346267cf689b/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7fb7a7df-9a89-4cb7-9bef-bfcf542ed28e/volumes/kubernetes.io~projected/kube-api-access-t58hp DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/696affb719b735bd04e03d28e25d2958829aeaff062c9fe97e096a2b8197acbd/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2db7f273c7518b0745c5e8bae9f6e47ed93e507e118eda0fab3419ec56a49269/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-163 DeviceMajor:0 DeviceMinor:163 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/503de692d00b91b8d502a7076ac6017b75167b1d394a025a0440d568beb9b66c/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b08436c9-9897-4b2a-b59f-40c7f26ac3c8/volumes/kubernetes.io~projected/kube-api-access-h2bmk DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b9736332-b3de-4fda-8372-880e2196a136/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/52d3557ffaa9dcb0c60e23002732a5fee588b9bc148de34a7d48651b523141f4/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1712802f314501d149a03ffad2781f32b71ddf66a385dd3026d1ffe2585c8157/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7c8d17a8-9a82-4a4f-a791-bf72f59fd756/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a36ceaac-5967-4321-802e-42f4c4b37032/volumes/kubernetes.io~projected/kube-api-access-87d2p DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b607752-d51b-4781-a57f-99e95883d236/volumes/kubernetes.io~projected/kube-api-access-ncgsv DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/940fa76d-d452-41fc-99ad-0f16f51c842e/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/940fa76d-d452-41fc-99ad-0f16f51c842e/volumes/kubernetes.io~projected/kube-api-access-5z5dz DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e0008197a374974d904d3d54eccbdb31bb9ee232b6ec21ada276f6249614ca2a/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-165 DeviceMajor:0 DeviceMinor:165 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/76e24688a1d436fedf2f2af4d99baf4a345cd30c5ae2918d5d1cf21da12a4af0/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:4102dfcdc76e5ce MacAddress:96:74:ce:6e:d0:fb Speed:10000 Mtu:8901} {Name:41f57abb74c1a66 MacAddress:02:df:ef:2d:ea:32 Speed:10000 Mtu:8901} {Name:503de692d00b91b MacAddress:16:ef:78:e3:79:4f Speed:10000 Mtu:8901} {Name:52d3557ffaa9dcb MacAddress:9e:c8:1f:41:e6:fe Speed:10000 Mtu:8901} {Name:76e24688a1d436f MacAddress:ae:26:a6:a5:54:90 Speed:10000 Mtu:8901} {Name:852721cfc7b1041 MacAddress:46:1c:8b:55:a5:13 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f9:4e:d3:89 Speed:0 Mtu:9001} {Name:br-int MacAddress:e2:7a:3e:98:33:3b Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:f9:4e:d3:89 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:56:6b:6a:e9:67:b9 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:7a:2a:9d:8f:80:3e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 12:45:05.031896 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.031897    4544 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 12:45:05.032174 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.031945    4544 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 12:45:05.032250 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.032212    4544 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 12:45:05.032382 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.032251    4544 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-38.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 12:45:05.032433 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.032385    4544 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 12:45:05.032433 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.032394    4544 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 12:45:05.032433 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.032405    4544 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 12:45:05.032433 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.032426    4544 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 12:45:05.032533 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.032525    4544 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 12:45:05.032705 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.032695    4544 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 12:45:05.032822 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.032813    4544 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 12:45:05.032856 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.032852    4544 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 12:45:05.032887 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.032866    4544 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 12:45:05.032887 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.032877    4544 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 12:45:05.032887 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.032886    4544 kubelet.go:405] "Adding apiserver pod source"
Oct 30 12:45:05.032978 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.032894    4544 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 12:45:05.033587 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.033569    4544 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 12:45:05.033587 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.033582    4544 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 12:45:05.034209 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.034191    4544 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 12:45:05.036198 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.036181    4544 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 12:45:05.036198 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.036196    4544 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 12:45:05.036354 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.036342    4544 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 12:45:05.036580 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.036565    4544 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 12:45:05.036652 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.036606    4544 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 12:45:05.036731 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.036720    4544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 12:45:05.036790 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.036740    4544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 12:45:05.036790 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.036750    4544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 12:45:05.036790 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.036759    4544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 12:45:05.036790 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.036769    4544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 12:45:05.036790 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.036778    4544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 12:45:05.036790 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.036789    4544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 12:45:05.037049 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.036798    4544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 12:45:05.037049 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.036810    4544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 12:45:05.037049 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.036820    4544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 12:45:05.037049 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.036840    4544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 12:45:05.037049 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.036854    4544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 12:45:05.037049 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.036889    4544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 12:45:05.037049 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.036899    4544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 12:45:05.040692 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.040675    4544 server.go:1263] "Started kubelet"
Oct 30 12:45:05.040769 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.040747    4544 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 12:45:05.040880 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.040831    4544 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 12:45:05.040928 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.040908    4544 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 12:45:05.041195 ip-10-0-143-38 systemd[1]: Started Kubernetes Kubelet.
Oct 30 12:45:05.041355 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.041269    4544 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 12:45:05.046235 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.043760    4544 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 12:45:05.046235 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.044830    4544 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 12:45:05.046235 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.045281    4544 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 12:45:05.046235 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:45:05.045865    4544 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-38.ec2.internal\" not found"
Oct 30 12:45:05.046235 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.045931    4544 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 12:45:05.046235 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.045940    4544 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 12:45:05.046235 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.046076    4544 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 12:45:05.047604 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.046805    4544 factory.go:55] Registering systemd factory
Oct 30 12:45:05.047604 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.046829    4544 factory.go:223] Registration of the systemd container factory successfully
Oct 30 12:45:05.047604 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.047235    4544 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:39:27 +0000 UTC" deadline="2028-08-06 02:32:41.51700579 +0000 UTC"
Oct 30 12:45:05.047604 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.047279    4544 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15493h47m36.46973147s"
Oct 30 12:45:05.049021 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.048983    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d57e941d-1795-48f2-9c7a-7d04c84d0078" volumeName="kubernetes.io/projected/d57e941d-1795-48f2-9c7a-7d04c84d0078-kube-api-access-l42fz" seLinuxMountContext=""
Oct 30 12:45:05.049021 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049020    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2" volumeName="kubernetes.io/configmap/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-cni-binary-copy" seLinuxMountContext=""
Oct 30 12:45:05.049134 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049033    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a36ceaac-5967-4321-802e-42f4c4b37032" volumeName="kubernetes.io/projected/a36ceaac-5967-4321-802e-42f4c4b37032-kube-api-access-87d2p" seLinuxMountContext=""
Oct 30 12:45:05.049134 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049043    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abe27783-da2a-46b9-81f6-cf551787b1bc" volumeName="kubernetes.io/secret/abe27783-da2a-46b9-81f6-cf551787b1bc-agent-certs" seLinuxMountContext=""
Oct 30 12:45:05.049134 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049053    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="184ffc49-7c79-41c2-b5b8-66e219e4f23e" volumeName="kubernetes.io/configmap/184ffc49-7c79-41c2-b5b8-66e219e4f23e-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 12:45:05.049134 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049062    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2eef9c4d-7079-47ad-9fad-c89c16ac69b4" volumeName="kubernetes.io/configmap/2eef9c4d-7079-47ad-9fad-c89c16ac69b4-serviceca" seLinuxMountContext=""
Oct 30 12:45:05.049134 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049070    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48212ac1-a400-4369-87dd-07b4d5317399" volumeName="kubernetes.io/projected/48212ac1-a400-4369-87dd-07b4d5317399-kube-api-access-ldcfg" seLinuxMountContext=""
Oct 30 12:45:05.049134 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049078    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b607752-d51b-4781-a57f-99e95883d236" volumeName="kubernetes.io/projected/7b607752-d51b-4781-a57f-99e95883d236-kube-api-access-ncgsv" seLinuxMountContext=""
Oct 30 12:45:05.049134 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049086    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b08436c9-9897-4b2a-b59f-40c7f26ac3c8" volumeName="kubernetes.io/projected/b08436c9-9897-4b2a-b59f-40c7f26ac3c8-kube-api-access-h2bmk" seLinuxMountContext=""
Oct 30 12:45:05.049134 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049094    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48212ac1-a400-4369-87dd-07b4d5317399" volumeName="kubernetes.io/configmap/48212ac1-a400-4369-87dd-07b4d5317399-iptables-alerter-script" seLinuxMountContext=""
Oct 30 12:45:05.049134 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049102    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623" volumeName="kubernetes.io/configmap/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-ovnkube-config" seLinuxMountContext=""
Oct 30 12:45:05.049134 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049109    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623" volumeName="kubernetes.io/projected/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-kube-api-access-nkk44" seLinuxMountContext=""
Oct 30 12:45:05.049134 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049117    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2eef9c4d-7079-47ad-9fad-c89c16ac69b4" volumeName="kubernetes.io/projected/2eef9c4d-7079-47ad-9fad-c89c16ac69b4-kube-api-access-kfhxh" seLinuxMountContext=""
Oct 30 12:45:05.049134 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049124    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fb7a7df-9a89-4cb7-9bef-bfcf542ed28e" volumeName="kubernetes.io/projected/7fb7a7df-9a89-4cb7-9bef-bfcf542ed28e-kube-api-access-t58hp" seLinuxMountContext=""
Oct 30 12:45:05.049502 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049141    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="940fa76d-d452-41fc-99ad-0f16f51c842e" volumeName="kubernetes.io/empty-dir/940fa76d-d452-41fc-99ad-0f16f51c842e-tmp" seLinuxMountContext=""
Oct 30 12:45:05.049502 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049149    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="940fa76d-d452-41fc-99ad-0f16f51c842e" volumeName="kubernetes.io/empty-dir/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-tuned" seLinuxMountContext=""
Oct 30 12:45:05.049502 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049157    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abe27783-da2a-46b9-81f6-cf551787b1bc" volumeName="kubernetes.io/configmap/abe27783-da2a-46b9-81f6-cf551787b1bc-konnectivity-ca" seLinuxMountContext=""
Oct 30 12:45:05.049502 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049165    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623" volumeName="kubernetes.io/configmap/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 12:45:05.049502 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049172    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b607752-d51b-4781-a57f-99e95883d236" volumeName="kubernetes.io/empty-dir/7b607752-d51b-4781-a57f-99e95883d236-tmp-dir" seLinuxMountContext=""
Oct 30 12:45:05.049502 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049179    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c8d17a8-9a82-4a4f-a791-bf72f59fd756" volumeName="kubernetes.io/projected/7c8d17a8-9a82-4a4f-a791-bf72f59fd756-kube-api-access-d4v98" seLinuxMountContext=""
Oct 30 12:45:05.049502 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049187    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2" volumeName="kubernetes.io/configmap/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-multus-daemon-config" seLinuxMountContext=""
Oct 30 12:45:05.049502 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049194    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b35d2751-096e-4a78-b32c-2d15f831954c" volumeName="kubernetes.io/empty-dir/b35d2751-096e-4a78-b32c-2d15f831954c-data-volume" seLinuxMountContext=""
Oct 30 12:45:05.049502 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049201    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee" volumeName="kubernetes.io/configmap/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 12:45:05.049502 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049208    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee" volumeName="kubernetes.io/configmap/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 12:45:05.049502 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049215    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee" volumeName="kubernetes.io/projected/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-kube-api-access-p8ns9" seLinuxMountContext=""
Oct 30 12:45:05.049502 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049248    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623" volumeName="kubernetes.io/configmap/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-env-overrides" seLinuxMountContext=""
Oct 30 12:45:05.049502 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049256    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623" volumeName="kubernetes.io/secret/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 12:45:05.049502 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049263    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="184ffc49-7c79-41c2-b5b8-66e219e4f23e" volumeName="kubernetes.io/configmap/184ffc49-7c79-41c2-b5b8-66e219e4f23e-service-ca-bundle" seLinuxMountContext=""
Oct 30 12:45:05.049502 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049273    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="184ffc49-7c79-41c2-b5b8-66e219e4f23e" volumeName="kubernetes.io/projected/184ffc49-7c79-41c2-b5b8-66e219e4f23e-kube-api-access-98j7q" seLinuxMountContext=""
Oct 30 12:45:05.049502 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049281    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="940fa76d-d452-41fc-99ad-0f16f51c842e" volumeName="kubernetes.io/projected/940fa76d-d452-41fc-99ad-0f16f51c842e-kube-api-access-5z5dz" seLinuxMountContext=""
Oct 30 12:45:05.049502 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049289    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a36ceaac-5967-4321-802e-42f4c4b37032" volumeName="kubernetes.io/empty-dir/a36ceaac-5967-4321-802e-42f4c4b37032-tmp-dir" seLinuxMountContext=""
Oct 30 12:45:05.049502 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049296    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bb8182c-0949-40e4-adeb-b8254e44922e" volumeName="kubernetes.io/projected/7bb8182c-0949-40e4-adeb-b8254e44922e-kube-api-access-jlrgs" seLinuxMountContext=""
Oct 30 12:45:05.049502 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049303    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2" volumeName="kubernetes.io/projected/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-kube-api-access-xffpq" seLinuxMountContext=""
Oct 30 12:45:05.049502 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049311    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a36ceaac-5967-4321-802e-42f4c4b37032" volumeName="kubernetes.io/configmap/a36ceaac-5967-4321-802e-42f4c4b37032-config-volume" seLinuxMountContext=""
Oct 30 12:45:05.049502 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049317    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b08436c9-9897-4b2a-b59f-40c7f26ac3c8" volumeName="kubernetes.io/configmap/b08436c9-9897-4b2a-b59f-40c7f26ac3c8-signing-cabundle" seLinuxMountContext=""
Oct 30 12:45:05.049502 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049325    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b35d2751-096e-4a78-b32c-2d15f831954c" volumeName="kubernetes.io/projected/b35d2751-096e-4a78-b32c-2d15f831954c-kube-api-access-5gxqq" seLinuxMountContext=""
Oct 30 12:45:05.049502 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049332    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee" volumeName="kubernetes.io/configmap/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-cni-binary-copy" seLinuxMountContext=""
Oct 30 12:45:05.049502 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049339    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="184ffc49-7c79-41c2-b5b8-66e219e4f23e" volumeName="kubernetes.io/empty-dir/184ffc49-7c79-41c2-b5b8-66e219e4f23e-archives-path" seLinuxMountContext=""
Oct 30 12:45:05.049502 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049347    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f33217b-afa5-4fff-a284-5fe1be413d5c" volumeName="kubernetes.io/projected/5f33217b-afa5-4fff-a284-5fe1be413d5c-kube-api-access-w2jqh" seLinuxMountContext=""
Oct 30 12:45:05.049502 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049355    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c8d17a8-9a82-4a4f-a791-bf72f59fd756" volumeName="kubernetes.io/secret/7c8d17a8-9a82-4a4f-a791-bf72f59fd756-cert" seLinuxMountContext=""
Oct 30 12:45:05.049502 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049363    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b08436c9-9897-4b2a-b59f-40c7f26ac3c8" volumeName="kubernetes.io/secret/b08436c9-9897-4b2a-b59f-40c7f26ac3c8-signing-key" seLinuxMountContext=""
Oct 30 12:45:05.049502 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049370    4544 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9736332-b3de-4fda-8372-880e2196a136" volumeName="kubernetes.io/secret/b9736332-b3de-4fda-8372-880e2196a136-original-pull-secret" seLinuxMountContext=""
Oct 30 12:45:05.049502 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049379    4544 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 12:45:05.049502 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049384    4544 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 12:45:05.050598 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049553    4544 factory.go:153] Registering CRI-O factory
Oct 30 12:45:05.050598 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049569    4544 factory.go:223] Registration of the crio container factory successfully
Oct 30 12:45:05.050598 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049620    4544 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 12:45:05.050598 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049634    4544 factory.go:103] Registering Raw factory
Oct 30 12:45:05.050598 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049649    4544 manager.go:1199] Started watching for new ooms in manager
Oct 30 12:45:05.050598 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.049955    4544 manager.go:319] Starting recovery of all containers
Oct 30 12:45:05.075954 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.075904    4544 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 12:45:05.077363 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.077340    4544 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 12:45:05.077363 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.077361    4544 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 12:45:05.077556 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.077382    4544 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 12:45:05.077556 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:45:05.077416    4544 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 12:45:05.085940 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.085894    4544 generic.go:356] "Generic (PLEG): container finished" podID="36805beca2c15cc87891009bcced9111" containerID="003de8c9f3729006608a735e75fe63ed3672163a4cc7c43545e62b12571bc4ae" exitCode=0
Oct 30 12:45:05.096641 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.096606    4544 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 12:45:05.097560 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.097526    4544 generic.go:356] "Generic (PLEG): container finished" podID="d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee" containerID="23aa074fb005b07b4d4eadbbe2f595c35882d05b74ed6ed251d4709db86a7c5e" exitCode=0
Oct 30 12:45:05.097560 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.097560    4544 generic.go:356] "Generic (PLEG): container finished" podID="d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee" containerID="fb7f83e67a2edff97cc95a525267f920e60a53ecf68976b6004e03763d151d9e" exitCode=0
Oct 30 12:45:05.097757 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.097570    4544 generic.go:356] "Generic (PLEG): container finished" podID="d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee" containerID="c9340509c033232136b344617f71d9b83197a6c8465dc6dd7fc94e68365556d5" exitCode=0
Oct 30 12:45:05.097757 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.097582    4544 generic.go:356] "Generic (PLEG): container finished" podID="d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee" containerID="8e8016a8675a6ee0b98efa262607299bd2b276940e405755f81c0fc4e8bf875c" exitCode=0
Oct 30 12:45:05.097757 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.097590    4544 generic.go:356] "Generic (PLEG): container finished" podID="d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee" containerID="4a7415d816c2731872e0e631e1e1c610a96624124a3f67671163761821276404" exitCode=0
Oct 30 12:45:05.097757 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.097598    4544 generic.go:356] "Generic (PLEG): container finished" podID="d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee" containerID="b0cba6a6d2adf75e1212e74825f2c39040ded3545f0d335cce3fe447d00bf5c6" exitCode=0
Oct 30 12:45:05.098002 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.097919    4544 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:45:05.110463 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.110430    4544 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:45:05.110654 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.110546    4544 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:45:05.127174 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.127093    4544 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:45:05.131798 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.131777    4544 manager.go:324] Recovery completed
Oct 30 12:45:05.146779 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.146755    4544 cpu_manager.go:226] "Starting" policy="none"
Oct 30 12:45:05.146779 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.146772    4544 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 12:45:05.146989 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.146797    4544 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 12:45:05.146989 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.146948    4544 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 12:45:05.146989 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.146960    4544 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 12:45:05.146989 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.146977    4544 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 12:45:05.146989 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.146983    4544 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 12:45:05.146989 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.146988    4544 policy_none.go:50] "Start"
Oct 30 12:45:05.147177 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.146996    4544 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 12:45:05.147177 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.147005    4544 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 12:45:05.147177 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.147131    4544 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 12:45:05.147177 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.147138    4544 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 12:45:05.147177 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.147143    4544 policy_none.go:44] "Start"
Oct 30 12:45:05.151408 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.151365    4544 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 12:45:05.154039 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.154023    4544 manager.go:341] "Starting Device Plugin manager"
Oct 30 12:45:05.154773 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:45:05.154052    4544 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 12:45:05.154849 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.154781    4544 server.go:87] "Starting device plugin registration server"
Oct 30 12:45:05.155155 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.155140    4544 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 12:45:05.155252 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.155159    4544 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 12:45:05.155405 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.155388    4544 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 12:45:05.156639 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.155487    4544 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 12:45:05.156639 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.155523    4544 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 12:45:05.156639 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.155527    4544 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 12:45:05.156639 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.155598    4544 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T12:45:05.155515377Z","UUID":"9faaf4c5-f7a9-4727-96f9-c0c5c2c92684","Handler":null,"Name":"","Endpoint":""}
Oct 30 12:45:05.156843 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:45:05.156780    4544 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 12:45:05.157078 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.157061    4544 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 12:45:05.157134 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.157088    4544 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 12:45:05.177526 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.177462    4544 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-38.ec2.internal"]
Oct 30 12:45:05.178067 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.177984    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal" event={"ID":"36805beca2c15cc87891009bcced9111","Type":"ContainerStarted","Data":"283bea144822dced6fd6272652a0256a02f6690eeef3340e4043badd276bcfba"}
Oct 30 12:45:05.178176 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.178082    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal" event={"ID":"36805beca2c15cc87891009bcced9111","Type":"ContainerDied","Data":"003de8c9f3729006608a735e75fe63ed3672163a4cc7c43545e62b12571bc4ae"}
Oct 30 12:45:05.178176 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.178106    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal" event={"ID":"36805beca2c15cc87891009bcced9111","Type":"ContainerStarted","Data":"ada0dc5751c1b2c179d3403d8ebca87eb8e0f244a186544542c715f38395c108"}
Oct 30 12:45:05.178176 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.178125    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-38.ec2.internal" event={"ID":"6d2044c13bd181c94b6dd5acd1654d5c","Type":"ContainerStarted","Data":"4fb53e3788edcdcb66639c8d45b625a6dcebb33ddc596ba9704b6474ee39abfc"}
Oct 30 12:45:05.178176 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.178137    4544 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal"
Oct 30 12:45:05.178350 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.178145    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-38.ec2.internal" event={"ID":"6d2044c13bd181c94b6dd5acd1654d5c","Type":"ContainerStarted","Data":"1712802f314501d149a03ffad2781f32b71ddf66a385dd3026d1ffe2585c8157"}
Oct 30 12:45:05.178350 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.178241    4544 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-38.ec2.internal"
Oct 30 12:45:05.202470 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.202434    4544 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:45:05.202690 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:45:05.202635    4544 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-38.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-38.ec2.internal"
Oct 30 12:45:05.202894 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.202874    4544 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:45:05.202944 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:45:05.202928    4544 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal"
Oct 30 12:45:05.256206 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.256169    4544 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:45:05.257602 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.257579    4544 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:45:05.257697 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.257616    4544 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:45:05.257697 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.257630    4544 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-38.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:45:05.257697 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.257666    4544 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-38.ec2.internal"
Oct 30 12:45:05.291752 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.291719    4544 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-38.ec2.internal"
Oct 30 12:45:05.291909 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.291797    4544 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-38.ec2.internal"
Oct 30 12:45:05.297632 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.297600    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/36805beca2c15cc87891009bcced9111-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal\" (UID: \"36805beca2c15cc87891009bcced9111\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal"
Oct 30 12:45:05.297748 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.297635    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36805beca2c15cc87891009bcced9111-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal\" (UID: \"36805beca2c15cc87891009bcced9111\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal"
Oct 30 12:45:05.297748 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.297653    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6d2044c13bd181c94b6dd5acd1654d5c-config\") pod \"kube-apiserver-proxy-ip-10-0-143-38.ec2.internal\" (UID: \"6d2044c13bd181c94b6dd5acd1654d5c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-38.ec2.internal"
Oct 30 12:45:05.397879 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.397780    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/36805beca2c15cc87891009bcced9111-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal\" (UID: \"36805beca2c15cc87891009bcced9111\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal"
Oct 30 12:45:05.397879 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.397827    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36805beca2c15cc87891009bcced9111-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal\" (UID: \"36805beca2c15cc87891009bcced9111\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal"
Oct 30 12:45:05.397879 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.397851    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6d2044c13bd181c94b6dd5acd1654d5c-config\") pod \"kube-apiserver-proxy-ip-10-0-143-38.ec2.internal\" (UID: \"6d2044c13bd181c94b6dd5acd1654d5c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-38.ec2.internal"
Oct 30 12:45:05.398115 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.397902    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6d2044c13bd181c94b6dd5acd1654d5c-config\") pod \"kube-apiserver-proxy-ip-10-0-143-38.ec2.internal\" (UID: \"6d2044c13bd181c94b6dd5acd1654d5c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-38.ec2.internal"
Oct 30 12:45:05.398115 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.397912    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/36805beca2c15cc87891009bcced9111-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal\" (UID: \"36805beca2c15cc87891009bcced9111\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal"
Oct 30 12:45:05.398115 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:05.397912    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36805beca2c15cc87891009bcced9111-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal\" (UID: \"36805beca2c15cc87891009bcced9111\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal"
Oct 30 12:45:06.033781 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.033742    4544 apiserver.go:52] "Watching apiserver"
Oct 30 12:45:06.090148 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.090113    4544 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 12:45:06.090606 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.090586    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-v68vl","openshift-insights/periodic-gathering-sb2kl-jnhml","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal","openshift-network-diagnostics/network-check-target-jt6tp","openshift-network-operator/iptables-alerter-f82zx","openshift-insights/insights-runtime-extractor-q6dlw","kube-system/global-pull-secret-syncer-xd7lj","kube-system/kube-apiserver-proxy-ip-10-0-143-38.ec2.internal","openshift-multus/network-metrics-daemon-mgctc","openshift-kube-storage-version-migrator/migrator-66df5d496b-r8xn7","openshift-multus/multus-additional-cni-plugins-bdjsz","openshift-multus/multus-n8f6q","kube-system/konnectivity-agent-9sprl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx","openshift-ovn-kubernetes/ovnkube-node-9zfnt","openshift-image-registry/node-ca-v9khv","openshift-service-ca/service-ca-5fffc97bdf-ft2cr","openshift-dns/dns-default-xmn5t","openshift-ingress-canary/ingress-canary-bgxhx","openshift-cluster-node-tuning-operator/tuned-79zhh"]
Oct 30 12:45:06.092213 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.092184    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgctc"
Oct 30 12:45:06.095648 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.093084    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:06.097800 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.097774    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xmn5t"
Oct 30 12:45:06.098582 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.098561    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:45:06.105881 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.105853    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 12:45:06.105881 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.105853    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-l246t\""
Oct 30 12:45:06.106070 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.105922    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-svz4t\""
Oct 30 12:45:06.106346 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.106327    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 12:45:06.106420 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.106331    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 12:45:06.106420 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.106352    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 12:45:06.106420 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.106370    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 12:45:06.106420 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.106353    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:45:06.106420 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.106393    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 12:45:06.106565 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.106426    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 12:45:06.106565 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.106429    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 12:45:06.106778 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.106757    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 12:45:06.106778 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.106760    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 12:45:06.107384 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.107368    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 12:45:06.107906 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.107886    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zngfb\""
Oct 30 12:45:06.107964 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.107925    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 12:45:06.107964 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.107930    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5fhhc\""
Oct 30 12:45:06.108072 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.107961    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6g9fh\""
Oct 30 12:45:06.108072 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.108042    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-w75sr\""
Oct 30 12:45:06.108129 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.108077    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-trr9l\""
Oct 30 12:45:06.118114 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.118081    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 12:45:06.118250 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.118083    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 12:45:06.118334 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.118319    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xlfj2\""
Oct 30 12:45:06.118507 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.118494    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 12:45:06.118573 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.118497    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 12:45:06.118573 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.118540    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k9vxw\""
Oct 30 12:45:06.118573 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.118543    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-b79n8\""
Oct 30 12:45:06.118573 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.118568    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mxwzk\""
Oct 30 12:45:06.118743 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.118731    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-k9mqg\""
Oct 30 12:45:06.118808 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.118769    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mwttw\""
Oct 30 12:45:06.118808 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.118777    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 12:45:06.118867 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.118808    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6f4z2\""
Oct 30 12:45:06.118867 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.118814    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 12:45:06.118867 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.118810    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pglr2\""
Oct 30 12:45:06.118867 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.118844    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g92fm\""
Oct 30 12:45:06.118867 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.118861    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-wp9fl\""
Oct 30 12:45:06.119010 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.118886    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 12:45:06.119010 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.118896    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9fv4n\""
Oct 30 12:45:06.130673 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.130647    4544 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-38.ec2.internal"
Oct 30 12:45:06.130841 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.130780    4544 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal"
Oct 30 12:45:06.147822 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.147795    4544 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 12:45:06.153445 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.153428    4544 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:45:06.153511 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:45:06.153482    4544 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-38.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-38.ec2.internal"
Oct 30 12:45:06.154002 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.153991    4544 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:45:06.154048 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:45:06.154023    4544 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-38.ec2.internal"
Oct 30 12:45:06.172514 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.172485    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-r8xn7_7fb7a7df-9a89-4cb7-9bef-bfcf542ed28e/migrator/0.log"
Oct 30 12:45:06.200381 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.200348    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-hostroot\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.200527 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.200384    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c8d17a8-9a82-4a4f-a791-bf72f59fd756-cert\") pod \"ingress-canary-bgxhx\" (UID: \"7c8d17a8-9a82-4a4f-a791-bf72f59fd756\") " pod="openshift-ingress-canary/ingress-canary-bgxhx"
Oct 30 12:45:06.200527 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.200405    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-node-log\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.200527 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.200424    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-kubernetes\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.200527 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.200492    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfhxh\" (UniqueName: \"kubernetes.io/projected/2eef9c4d-7079-47ad-9fad-c89c16ac69b4-kube-api-access-kfhxh\") pod \"node-ca-v9khv\" (UID: \"2eef9c4d-7079-47ad-9fad-c89c16ac69b4\") " pod="openshift-image-registry/node-ca-v9khv"
Oct 30 12:45:06.200664 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.200568    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-multus-daemon-config\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.200664 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.200625    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-run\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.200747 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.200729    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:45:06.200779 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.200760    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t58hp\" (UniqueName: \"kubernetes.io/projected/7fb7a7df-9a89-4cb7-9bef-bfcf542ed28e-kube-api-access-t58hp\") pod \"migrator-66df5d496b-r8xn7\" (UID: \"7fb7a7df-9a89-4cb7-9bef-bfcf542ed28e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-r8xn7"
Oct 30 12:45:06.200807 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.200796    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7b607752-d51b-4781-a57f-99e95883d236-hosts-file\") pod \"node-resolver-v68vl\" (UID: \"7b607752-d51b-4781-a57f-99e95883d236\") " pod="openshift-dns/node-resolver-v68vl"
Oct 30 12:45:06.200838 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.200800    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c8d17a8-9a82-4a4f-a791-bf72f59fd756-cert\") pod \"ingress-canary-bgxhx\" (UID: \"7c8d17a8-9a82-4a4f-a791-bf72f59fd756\") " pod="openshift-ingress-canary/ingress-canary-bgxhx"
Oct 30 12:45:06.200838 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.200816    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a36ceaac-5967-4321-802e-42f4c4b37032-metrics-tls\") pod \"dns-default-xmn5t\" (UID: \"a36ceaac-5967-4321-802e-42f4c4b37032\") " pod="openshift-dns/dns-default-xmn5t"
Oct 30 12:45:06.200899 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.200838    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-socket-dir\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:45:06.200899 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.200879    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-etc-kubernetes\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.200972 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.200955    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xffpq\" (UniqueName: \"kubernetes.io/projected/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-kube-api-access-xffpq\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.201115 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.201096    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-cni-bin\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.201163 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.201143    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z5dz\" (UniqueName: \"kubernetes.io/projected/940fa76d-d452-41fc-99ad-0f16f51c842e-kube-api-access-5z5dz\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.201196 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.201181    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2jqh\" (UniqueName: \"kubernetes.io/projected/5f33217b-afa5-4fff-a284-5fe1be413d5c-kube-api-access-w2jqh\") pod \"network-check-target-jt6tp\" (UID: \"5f33217b-afa5-4fff-a284-5fe1be413d5c\") " pod="openshift-network-diagnostics/network-check-target-jt6tp"
Oct 30 12:45:06.201243 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.201208    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-multus-conf-dir\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.201272 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.201264    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldcfg\" (UniqueName: \"kubernetes.io/projected/48212ac1-a400-4369-87dd-07b4d5317399-kube-api-access-ldcfg\") pod \"iptables-alerter-f82zx\" (UID: \"48212ac1-a400-4369-87dd-07b4d5317399\") " pod="openshift-network-operator/iptables-alerter-f82zx"
Oct 30 12:45:06.201306 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.201285    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-run-ovn-kubernetes\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.201346 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.201309    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-cni-netd\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.201377 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.201345    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-modprobe-d\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.201416 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.201402    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-system-cni-dir\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:45:06.201458 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.201441    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:45:06.201554 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.201500    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:45:06.201554 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.201548    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/abe27783-da2a-46b9-81f6-cf551787b1bc-konnectivity-ca\") pod \"konnectivity-agent-9sprl\" (UID: \"abe27783-da2a-46b9-81f6-cf551787b1bc\") " pod="kube-system/konnectivity-agent-9sprl"
Oct 30 12:45:06.201641 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.201619    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-multus-cni-dir\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.201673 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.201641    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-kubelet\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.201673 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.201658    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-ovnkube-script-lib\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.201756 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.201674    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-cnibin\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:45:06.201756 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.201690    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8ns9\" (UniqueName: \"kubernetes.io/projected/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-kube-api-access-p8ns9\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:45:06.201756 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.201739    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b08436c9-9897-4b2a-b59f-40c7f26ac3c8-signing-cabundle\") pod \"service-ca-5fffc97bdf-ft2cr\" (UID: \"b08436c9-9897-4b2a-b59f-40c7f26ac3c8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ft2cr"
Oct 30 12:45:06.201861 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.201812    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b35d2751-096e-4a78-b32c-2d15f831954c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:45:06.201861 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.201837    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/abe27783-da2a-46b9-81f6-cf551787b1bc-konnectivity-ca\") pod \"konnectivity-agent-9sprl\" (UID: \"abe27783-da2a-46b9-81f6-cf551787b1bc\") " pod="kube-system/konnectivity-agent-9sprl"
Oct 30 12:45:06.201861 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.201852    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/184ffc49-7c79-41c2-b5b8-66e219e4f23e-archives-path\") pod \"periodic-gathering-sb2kl-jnhml\" (UID: \"184ffc49-7c79-41c2-b5b8-66e219e4f23e\") " pod="openshift-insights/periodic-gathering-sb2kl-jnhml"
Oct 30 12:45:06.201977 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.201898    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98j7q\" (UniqueName: \"kubernetes.io/projected/184ffc49-7c79-41c2-b5b8-66e219e4f23e-kube-api-access-98j7q\") pod \"periodic-gathering-sb2kl-jnhml\" (UID: \"184ffc49-7c79-41c2-b5b8-66e219e4f23e\") " pod="openshift-insights/periodic-gathering-sb2kl-jnhml"
Oct 30 12:45:06.201977 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.201960    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/184ffc49-7c79-41c2-b5b8-66e219e4f23e-archives-path\") pod \"periodic-gathering-sb2kl-jnhml\" (UID: \"184ffc49-7c79-41c2-b5b8-66e219e4f23e\") " pod="openshift-insights/periodic-gathering-sb2kl-jnhml"
Oct 30 12:45:06.202067 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.201971    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-registration-dir\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:45:06.202067 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.202054    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlrgs\" (UniqueName: \"kubernetes.io/projected/7bb8182c-0949-40e4-adeb-b8254e44922e-kube-api-access-jlrgs\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:45:06.202163 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.202087    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-var-lib-cni-bin\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.202163 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.202124    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-slash\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.202260 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.202161    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-etc-openvswitch\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.202260 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.202183    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-device-dir\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:45:06.202260 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.202214    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-cnibin\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.202395 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.202277    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-sysctl-conf\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.202395 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.202309    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-lib-modules\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.202395 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.202334    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-var-lib-kubelet\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.202395 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.202361    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b9736332-b3de-4fda-8372-880e2196a136-dbus\") pod \"global-pull-secret-syncer-xd7lj\" (UID: \"b9736332-b3de-4fda-8372-880e2196a136\") " pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:45:06.202574 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.202393    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9736332-b3de-4fda-8372-880e2196a136-original-pull-secret\") pod \"global-pull-secret-syncer-xd7lj\" (UID: \"b9736332-b3de-4fda-8372-880e2196a136\") " pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:45:06.202574 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.202413    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/abe27783-da2a-46b9-81f6-cf551787b1bc-agent-certs\") pod \"konnectivity-agent-9sprl\" (UID: \"abe27783-da2a-46b9-81f6-cf551787b1bc\") " pod="kube-system/konnectivity-agent-9sprl"
Oct 30 12:45:06.202574 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.202546    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d57e941d-1795-48f2-9c7a-7d04c84d0078-metrics-certs\") pod \"network-metrics-daemon-mgctc\" (UID: \"d57e941d-1795-48f2-9c7a-7d04c84d0078\") " pod="openshift-multus/network-metrics-daemon-mgctc"
Oct 30 12:45:06.202720 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.202580    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/48212ac1-a400-4369-87dd-07b4d5317399-host-slash\") pod \"iptables-alerter-f82zx\" (UID: \"48212ac1-a400-4369-87dd-07b4d5317399\") " pod="openshift-network-operator/iptables-alerter-f82zx"
Oct 30 12:45:06.202720 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.202603    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-run-openvswitch\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.202720 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.202628    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-ovnkube-config\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.202720 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.202662    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b35d2751-096e-4a78-b32c-2d15f831954c-crio-socket\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:45:06.202720 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.202673    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9736332-b3de-4fda-8372-880e2196a136-original-pull-secret\") pod \"global-pull-secret-syncer-xd7lj\" (UID: \"b9736332-b3de-4fda-8372-880e2196a136\") " pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:45:06.202720 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.202686    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a36ceaac-5967-4321-802e-42f4c4b37032-config-volume\") pod \"dns-default-xmn5t\" (UID: \"a36ceaac-5967-4321-802e-42f4c4b37032\") " pod="openshift-dns/dns-default-xmn5t"
Oct 30 12:45:06.202720 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.202689    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/abe27783-da2a-46b9-81f6-cf551787b1bc-agent-certs\") pod \"konnectivity-agent-9sprl\" (UID: \"abe27783-da2a-46b9-81f6-cf551787b1bc\") " pod="kube-system/konnectivity-agent-9sprl"
Oct 30 12:45:06.202910 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.202733    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-etc-selinux\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:45:06.202910 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.202756    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-var-lib-kubelet\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.202910 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.202775    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-run-multus-certs\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.202910 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.202813    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/48212ac1-a400-4369-87dd-07b4d5317399-iptables-alerter-script\") pod \"iptables-alerter-f82zx\" (UID: \"48212ac1-a400-4369-87dd-07b4d5317399\") " pod="openshift-network-operator/iptables-alerter-f82zx"
Oct 30 12:45:06.202910 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.202844    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.202910 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.202875    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkk44\" (UniqueName: \"kubernetes.io/projected/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-kube-api-access-nkk44\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.202910 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.202891    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b35d2751-096e-4a78-b32c-2d15f831954c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:45:06.203172 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.202924    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a36ceaac-5967-4321-802e-42f4c4b37032-tmp-dir\") pod \"dns-default-xmn5t\" (UID: \"a36ceaac-5967-4321-802e-42f4c4b37032\") " pod="openshift-dns/dns-default-xmn5t"
Oct 30 12:45:06.203172 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.202954    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/184ffc49-7c79-41c2-b5b8-66e219e4f23e-service-ca-bundle\") pod \"periodic-gathering-sb2kl-jnhml\" (UID: \"184ffc49-7c79-41c2-b5b8-66e219e4f23e\") " pod="openshift-insights/periodic-gathering-sb2kl-jnhml"
Oct 30 12:45:06.203172 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.202976    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:45:06.203172 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.203003    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b08436c9-9897-4b2a-b59f-40c7f26ac3c8-signing-key\") pod \"service-ca-5fffc97bdf-ft2cr\" (UID: \"b08436c9-9897-4b2a-b59f-40c7f26ac3c8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ft2cr"
Oct 30 12:45:06.203172 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.203005    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/48212ac1-a400-4369-87dd-07b4d5317399-iptables-alerter-script\") pod \"iptables-alerter-f82zx\" (UID: \"48212ac1-a400-4369-87dd-07b4d5317399\") " pod="openshift-network-operator/iptables-alerter-f82zx"
Oct 30 12:45:06.203172 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.203043    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a36ceaac-5967-4321-802e-42f4c4b37032-tmp-dir\") pod \"dns-default-xmn5t\" (UID: \"a36ceaac-5967-4321-802e-42f4c4b37032\") " pod="openshift-dns/dns-default-xmn5t"
Oct 30 12:45:06.203172 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.203050    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2bmk\" (UniqueName: \"kubernetes.io/projected/b08436c9-9897-4b2a-b59f-40c7f26ac3c8-kube-api-access-h2bmk\") pod \"service-ca-5fffc97bdf-ft2cr\" (UID: \"b08436c9-9897-4b2a-b59f-40c7f26ac3c8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ft2cr"
Oct 30 12:45:06.203172 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.203161    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b35d2751-096e-4a78-b32c-2d15f831954c-data-volume\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:45:06.203405 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.203208    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b08436c9-9897-4b2a-b59f-40c7f26ac3c8-signing-key\") pod \"service-ca-5fffc97bdf-ft2cr\" (UID: \"b08436c9-9897-4b2a-b59f-40c7f26ac3c8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ft2cr"
Oct 30 12:45:06.203405 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.203244    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b35d2751-096e-4a78-b32c-2d15f831954c-data-volume\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:45:06.203405 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.203239    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b607752-d51b-4781-a57f-99e95883d236-tmp-dir\") pod \"node-resolver-v68vl\" (UID: \"7b607752-d51b-4781-a57f-99e95883d236\") " pod="openshift-dns/node-resolver-v68vl"
Oct 30 12:45:06.203405 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.203284    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2eef9c4d-7079-47ad-9fad-c89c16ac69b4-serviceca\") pod \"node-ca-v9khv\" (UID: \"2eef9c4d-7079-47ad-9fad-c89c16ac69b4\") " pod="openshift-image-registry/node-ca-v9khv"
Oct 30 12:45:06.203405 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.203312    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b607752-d51b-4781-a57f-99e95883d236-tmp-dir\") pod \"node-resolver-v68vl\" (UID: \"7b607752-d51b-4781-a57f-99e95883d236\") " pod="openshift-dns/node-resolver-v68vl"
Oct 30 12:45:06.203405 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.203315    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-run-k8s-cni-cncf-io\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.203563 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.203408    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-run-netns\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.203563 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.203451    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-sysconfig\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.203563 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.203459    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2eef9c4d-7079-47ad-9fad-c89c16ac69b4-serviceca\") pod \"node-ca-v9khv\" (UID: \"2eef9c4d-7079-47ad-9fad-c89c16ac69b4\") " pod="openshift-image-registry/node-ca-v9khv"
Oct 30 12:45:06.203563 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.203472    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4v98\" (UniqueName: \"kubernetes.io/projected/7c8d17a8-9a82-4a4f-a791-bf72f59fd756-kube-api-access-d4v98\") pod \"ingress-canary-bgxhx\" (UID: \"7c8d17a8-9a82-4a4f-a791-bf72f59fd756\") " pod="openshift-ingress-canary/ingress-canary-bgxhx"
Oct 30 12:45:06.203563 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.203501    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncgsv\" (UniqueName: \"kubernetes.io/projected/7b607752-d51b-4781-a57f-99e95883d236-kube-api-access-ncgsv\") pod \"node-resolver-v68vl\" (UID: \"7b607752-d51b-4781-a57f-99e95883d236\") " pod="openshift-dns/node-resolver-v68vl"
Oct 30 12:45:06.203702 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.203564    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-multus-socket-dir-parent\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.203702 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.203660    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-systemd-units\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.203702 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.203676    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-run-systemd\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.203784 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.203722    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-run-ovn\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.203784 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.203756    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-env-overrides\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.203836 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.203795    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/940fa76d-d452-41fc-99ad-0f16f51c842e-tmp\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.203836 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.203821    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b9736332-b3de-4fda-8372-880e2196a136-kubelet-config\") pod \"global-pull-secret-syncer-xd7lj\" (UID: \"b9736332-b3de-4fda-8372-880e2196a136\") " pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:45:06.203890 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.203865    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-sys\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.203921 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.203898    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/940fa76d-d452-41fc-99ad-0f16f51c842e-tmp\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.203921 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.203904    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gxqq\" (UniqueName: \"kubernetes.io/projected/b35d2751-096e-4a78-b32c-2d15f831954c-kube-api-access-5gxqq\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:45:06.203973 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.203935    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-var-lib-cni-multus\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.203973 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.203959    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-log-socket\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.204060 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.204044    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-ovn-node-metrics-cert\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.204090 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.204080    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-host\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.204139 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.204123    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-tuned\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.204177 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.204165    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-cni-binary-copy\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:45:06.204232 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.204201    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-tuned\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.204270 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.204251    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-ovn-node-metrics-cert\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.204270 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.204257    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-var-lib-openvswitch\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.204327 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.204287    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l42fz\" (UniqueName: \"kubernetes.io/projected/d57e941d-1795-48f2-9c7a-7d04c84d0078-kube-api-access-l42fz\") pod \"network-metrics-daemon-mgctc\" (UID: \"d57e941d-1795-48f2-9c7a-7d04c84d0078\") " pod="openshift-multus/network-metrics-daemon-mgctc"
Oct 30 12:45:06.204360 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.204325    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87d2p\" (UniqueName: \"kubernetes.io/projected/a36ceaac-5967-4321-802e-42f4c4b37032-kube-api-access-87d2p\") pod \"dns-default-xmn5t\" (UID: \"a36ceaac-5967-4321-802e-42f4c4b37032\") " pod="openshift-dns/dns-default-xmn5t"
Oct 30 12:45:06.204360 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.204350    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/184ffc49-7c79-41c2-b5b8-66e219e4f23e-trusted-ca-bundle\") pod \"periodic-gathering-sb2kl-jnhml\" (UID: \"184ffc49-7c79-41c2-b5b8-66e219e4f23e\") " pod="openshift-insights/periodic-gathering-sb2kl-jnhml"
Oct 30 12:45:06.204457 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.204422    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-sys-fs\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:45:06.204538 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.204523    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-sysctl-d\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.204584 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.204563    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-systemd\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.204613 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.204592    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2eef9c4d-7079-47ad-9fad-c89c16ac69b4-host\") pod \"node-ca-v9khv\" (UID: \"2eef9c4d-7079-47ad-9fad-c89c16ac69b4\") " pod="openshift-image-registry/node-ca-v9khv"
Oct 30 12:45:06.204641 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.204613    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-system-cni-dir\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.204674 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.204647    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-cni-binary-copy\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.204702 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.204692    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-run-netns\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.206167 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.206147    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 12:45:06.206277 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.206183    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 12:45:06.206489 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.206476    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:45:06.206572 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.206478    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 12:45:06.206572 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.206499    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 12:45:06.206572 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.206508    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 12:45:06.206572 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.206529    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 12:45:06.206692 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.206587    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 12:45:06.206738 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.206714    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 12:45:06.206738 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.206734    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 12:45:06.206836 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.206752    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 12:45:06.206836 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.206751    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 12:45:06.206836 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.206774    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 12:45:06.206836 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.206779    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 12:45:06.206836 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.206828    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 12:45:06.207021 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.206859    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 12:45:06.207021 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.206863    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 12:45:06.207021 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.206975    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 12:45:06.207021 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.206985    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 12:45:06.207021 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.206990    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 12:45:06.207021 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.207000    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 12:45:06.207021 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.207007    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 12:45:06.207021 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.207010    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 12:45:06.207274 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.207025    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 12:45:06.207274 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.206983    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 12:45:06.207274 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.207149    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 12:45:06.209249 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.207809    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 12:45:06.209249 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.209064    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 12:45:06.211209 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.211186    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-multus-daemon-config\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.211940 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.211908    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:45:06.211940 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.211908    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:45:06.212066 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.212051    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b08436c9-9897-4b2a-b59f-40c7f26ac3c8-signing-cabundle\") pod \"service-ca-5fffc97bdf-ft2cr\" (UID: \"b08436c9-9897-4b2a-b59f-40c7f26ac3c8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ft2cr"
Oct 30 12:45:06.212110 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.212057    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-ovnkube-script-lib\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.213164 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.213149    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a36ceaac-5967-4321-802e-42f4c4b37032-config-volume\") pod \"dns-default-xmn5t\" (UID: \"a36ceaac-5967-4321-802e-42f4c4b37032\") " pod="openshift-dns/dns-default-xmn5t"
Oct 30 12:45:06.213245 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.213232    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-ovnkube-config\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.213300 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.213216    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/184ffc49-7c79-41c2-b5b8-66e219e4f23e-service-ca-bundle\") pod \"periodic-gathering-sb2kl-jnhml\" (UID: \"184ffc49-7c79-41c2-b5b8-66e219e4f23e\") " pod="openshift-insights/periodic-gathering-sb2kl-jnhml"
Oct 30 12:45:06.213963 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.213949    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-env-overrides\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.215116 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.215097    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-cni-binary-copy\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.215190 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.215176    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-cni-binary-copy\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:45:06.215283 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.215266    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/184ffc49-7c79-41c2-b5b8-66e219e4f23e-trusted-ca-bundle\") pod \"periodic-gathering-sb2kl-jnhml\" (UID: \"184ffc49-7c79-41c2-b5b8-66e219e4f23e\") " pod="openshift-insights/periodic-gathering-sb2kl-jnhml"
Oct 30 12:45:06.222443 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.222293    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t58hp\" (UniqueName: \"kubernetes.io/projected/7fb7a7df-9a89-4cb7-9bef-bfcf542ed28e-kube-api-access-t58hp\") pod \"migrator-66df5d496b-r8xn7\" (UID: \"7fb7a7df-9a89-4cb7-9bef-bfcf542ed28e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-r8xn7"
Oct 30 12:45:06.222443 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.222382    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xffpq\" (UniqueName: \"kubernetes.io/projected/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-kube-api-access-xffpq\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.222603 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.222447    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z5dz\" (UniqueName: \"kubernetes.io/projected/940fa76d-d452-41fc-99ad-0f16f51c842e-kube-api-access-5z5dz\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.222603 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.222485    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldcfg\" (UniqueName: \"kubernetes.io/projected/48212ac1-a400-4369-87dd-07b4d5317399-kube-api-access-ldcfg\") pod \"iptables-alerter-f82zx\" (UID: \"48212ac1-a400-4369-87dd-07b4d5317399\") " pod="openshift-network-operator/iptables-alerter-f82zx"
Oct 30 12:45:06.222690 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.222668    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfhxh\" (UniqueName: \"kubernetes.io/projected/2eef9c4d-7079-47ad-9fad-c89c16ac69b4-kube-api-access-kfhxh\") pod \"node-ca-v9khv\" (UID: \"2eef9c4d-7079-47ad-9fad-c89c16ac69b4\") " pod="openshift-image-registry/node-ca-v9khv"
Oct 30 12:45:06.222789 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.222772    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98j7q\" (UniqueName: \"kubernetes.io/projected/184ffc49-7c79-41c2-b5b8-66e219e4f23e-kube-api-access-98j7q\") pod \"periodic-gathering-sb2kl-jnhml\" (UID: \"184ffc49-7c79-41c2-b5b8-66e219e4f23e\") " pod="openshift-insights/periodic-gathering-sb2kl-jnhml"
Oct 30 12:45:06.222890 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.222866    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlrgs\" (UniqueName: \"kubernetes.io/projected/7bb8182c-0949-40e4-adeb-b8254e44922e-kube-api-access-jlrgs\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:45:06.222925 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.222892    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2jqh\" (UniqueName: \"kubernetes.io/projected/5f33217b-afa5-4fff-a284-5fe1be413d5c-kube-api-access-w2jqh\") pod \"network-check-target-jt6tp\" (UID: \"5f33217b-afa5-4fff-a284-5fe1be413d5c\") " pod="openshift-network-diagnostics/network-check-target-jt6tp"
Oct 30 12:45:06.223201 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.223184    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8ns9\" (UniqueName: \"kubernetes.io/projected/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-kube-api-access-p8ns9\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:45:06.229994 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.229964    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gxqq\" (UniqueName: \"kubernetes.io/projected/b35d2751-096e-4a78-b32c-2d15f831954c-kube-api-access-5gxqq\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:45:06.230139 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.230037    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87d2p\" (UniqueName: \"kubernetes.io/projected/a36ceaac-5967-4321-802e-42f4c4b37032-kube-api-access-87d2p\") pod \"dns-default-xmn5t\" (UID: \"a36ceaac-5967-4321-802e-42f4c4b37032\") " pod="openshift-dns/dns-default-xmn5t"
Oct 30 12:45:06.230139 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.230048    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l42fz\" (UniqueName: \"kubernetes.io/projected/d57e941d-1795-48f2-9c7a-7d04c84d0078-kube-api-access-l42fz\") pod \"network-metrics-daemon-mgctc\" (UID: \"d57e941d-1795-48f2-9c7a-7d04c84d0078\") " pod="openshift-multus/network-metrics-daemon-mgctc"
Oct 30 12:45:06.230139 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.230049    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkk44\" (UniqueName: \"kubernetes.io/projected/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-kube-api-access-nkk44\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.230310 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.230238    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4v98\" (UniqueName: \"kubernetes.io/projected/7c8d17a8-9a82-4a4f-a791-bf72f59fd756-kube-api-access-d4v98\") pod \"ingress-canary-bgxhx\" (UID: \"7c8d17a8-9a82-4a4f-a791-bf72f59fd756\") " pod="openshift-ingress-canary/ingress-canary-bgxhx"
Oct 30 12:45:06.230310 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.230276    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2bmk\" (UniqueName: \"kubernetes.io/projected/b08436c9-9897-4b2a-b59f-40c7f26ac3c8-kube-api-access-h2bmk\") pod \"service-ca-5fffc97bdf-ft2cr\" (UID: \"b08436c9-9897-4b2a-b59f-40c7f26ac3c8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ft2cr"
Oct 30 12:45:06.230366 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.230276    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncgsv\" (UniqueName: \"kubernetes.io/projected/7b607752-d51b-4781-a57f-99e95883d236-kube-api-access-ncgsv\") pod \"node-resolver-v68vl\" (UID: \"7b607752-d51b-4781-a57f-99e95883d236\") " pod="openshift-dns/node-resolver-v68vl"
Oct 30 12:45:06.305250 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305132    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-multus-cni-dir\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.305250 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305171    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-kubelet\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.305250 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305191    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-cnibin\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:45:06.305487 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305259    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-cnibin\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:45:06.305487 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305271    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b35d2751-096e-4a78-b32c-2d15f831954c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:45:06.305487 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305289    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-multus-cni-dir\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.305487 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305309    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-registration-dir\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:45:06.305487 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305275    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-kubelet\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.305487 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305333    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-var-lib-cni-bin\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.305487 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305351    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-slash\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.305487 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305356    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-registration-dir\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:45:06.305487 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:45:06.305360    4544 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 12:45:06.305487 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305368    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-etc-openvswitch\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.305487 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305382    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-var-lib-cni-bin\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.305487 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305386    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-device-dir\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:45:06.305487 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305410    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-cnibin\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.305487 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305419    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-etc-openvswitch\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.305487 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305425    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-sysctl-conf\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.305487 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305425    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-slash\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.305487 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:45:06.305449    4544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b35d2751-096e-4a78-b32c-2d15f831954c-insights-runtime-extractor-tls podName:b35d2751-096e-4a78-b32c-2d15f831954c nodeName:}" failed. No retries permitted until 2026-10-30 12:45:06.805429001 +0000 UTC m=+1.852562953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b35d2751-096e-4a78-b32c-2d15f831954c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q6dlw" (UID: "b35d2751-096e-4a78-b32c-2d15f831954c") : secret "insights-runtime-extractor-tls" not found
Oct 30 12:45:06.305487 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305466    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-cnibin\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.305487 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305490    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-device-dir\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:45:06.305487 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305499    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-lib-modules\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305521    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-sysctl-conf\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305526    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-var-lib-kubelet\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305549    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b9736332-b3de-4fda-8372-880e2196a136-dbus\") pod \"global-pull-secret-syncer-xd7lj\" (UID: \"b9736332-b3de-4fda-8372-880e2196a136\") " pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305575    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d57e941d-1795-48f2-9c7a-7d04c84d0078-metrics-certs\") pod \"network-metrics-daemon-mgctc\" (UID: \"d57e941d-1795-48f2-9c7a-7d04c84d0078\") " pod="openshift-multus/network-metrics-daemon-mgctc"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305584    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b9736332-b3de-4fda-8372-880e2196a136-dbus\") pod \"global-pull-secret-syncer-xd7lj\" (UID: \"b9736332-b3de-4fda-8372-880e2196a136\") " pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305553    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-var-lib-kubelet\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305599    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/48212ac1-a400-4369-87dd-07b4d5317399-host-slash\") pod \"iptables-alerter-f82zx\" (UID: \"48212ac1-a400-4369-87dd-07b4d5317399\") " pod="openshift-network-operator/iptables-alerter-f82zx"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305602    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-lib-modules\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305623    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-run-openvswitch\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305639    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/48212ac1-a400-4369-87dd-07b4d5317399-host-slash\") pod \"iptables-alerter-f82zx\" (UID: \"48212ac1-a400-4369-87dd-07b4d5317399\") " pod="openshift-network-operator/iptables-alerter-f82zx"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305649    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b35d2751-096e-4a78-b32c-2d15f831954c-crio-socket\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305670    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-run-openvswitch\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305674    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-etc-selinux\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305712    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-var-lib-kubelet\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305729    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-run-multus-certs\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305733    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-etc-selinux\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305737    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b35d2751-096e-4a78-b32c-2d15f831954c-crio-socket\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305746    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305770    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-run-multus-certs\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305782    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b35d2751-096e-4a78-b32c-2d15f831954c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305813    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305817    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305784    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-var-lib-kubelet\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305844    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-run-k8s-cni-cncf-io\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305871    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-run-netns\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305893    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-sysconfig\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305898    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-run-k8s-cni-cncf-io\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305872    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305917    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-multus-socket-dir-parent\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305945    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-systemd-units\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305951    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-sysconfig\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305954    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-run-netns\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305967    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-run-systemd\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305986    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-systemd-units\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305992    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-run-ovn\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306004    4544 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306020    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b9736332-b3de-4fda-8372-880e2196a136-kubelet-config\") pod \"global-pull-secret-syncer-xd7lj\" (UID: \"b9736332-b3de-4fda-8372-880e2196a136\") " pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.305997    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-multus-socket-dir-parent\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306031    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-run-ovn\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306020    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-run-systemd\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306054    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-sys\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306079    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b9736332-b3de-4fda-8372-880e2196a136-kubelet-config\") pod \"global-pull-secret-syncer-xd7lj\" (UID: \"b9736332-b3de-4fda-8372-880e2196a136\") " pod="kube-system/global-pull-secret-syncer-xd7lj"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306085    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-var-lib-cni-multus\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306099    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-log-socket\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306133    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-log-socket\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306133    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-sys\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306162    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-var-lib-cni-multus\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306187    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-host\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306269    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-host\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306271    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-var-lib-openvswitch\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306292    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-var-lib-openvswitch\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306309    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-sys-fs\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306332    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-sysctl-d\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306348    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-systemd\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306361    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2eef9c4d-7079-47ad-9fad-c89c16ac69b4-host\") pod \"node-ca-v9khv\" (UID: \"2eef9c4d-7079-47ad-9fad-c89c16ac69b4\") " pod="openshift-image-registry/node-ca-v9khv"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306380    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-system-cni-dir\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306403    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-run-netns\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306409    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-sys-fs\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306413    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-systemd\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306407    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b35d2751-096e-4a78-b32c-2d15f831954c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306436    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-host-run-netns\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306443    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-hostroot\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306460    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-system-cni-dir\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306471    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-node-log\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.306438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306493    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-kubernetes\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.308692 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306496    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-hostroot\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.308692 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306527    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-sysctl-d\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.308692 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306540    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-kubernetes\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.308692 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306547    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-run\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.308692 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306577    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:45:06.308692 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306586    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-run\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.308692 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306593    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7b607752-d51b-4781-a57f-99e95883d236-hosts-file\") pod \"node-resolver-v68vl\" (UID: \"7b607752-d51b-4781-a57f-99e95883d236\") " pod="openshift-dns/node-resolver-v68vl"
Oct 30 12:45:06.308692 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306531    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-node-log\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.308692 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306609    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a36ceaac-5967-4321-802e-42f4c4b37032-metrics-tls\") pod \"dns-default-xmn5t\" (UID: \"a36ceaac-5967-4321-802e-42f4c4b37032\") " pod="openshift-dns/dns-default-xmn5t"
Oct 30 12:45:06.308692 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306631    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-socket-dir\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:45:06.308692 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306627    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:45:06.308692 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306647    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7b607752-d51b-4781-a57f-99e95883d236-hosts-file\") pod \"node-resolver-v68vl\" (UID: \"7b607752-d51b-4781-a57f-99e95883d236\") " pod="openshift-dns/node-resolver-v68vl"
Oct 30 12:45:06.308692 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306655    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2eef9c4d-7079-47ad-9fad-c89c16ac69b4-host\") pod \"node-ca-v9khv\" (UID: \"2eef9c4d-7079-47ad-9fad-c89c16ac69b4\") " pod="openshift-image-registry/node-ca-v9khv"
Oct 30 12:45:06.308692 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306649    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-etc-kubernetes\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.308692 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306708    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7bb8182c-0949-40e4-adeb-b8254e44922e-socket-dir\") pod \"aws-ebs-csi-driver-node-jq8kx\" (UID: \"7bb8182c-0949-40e4-adeb-b8254e44922e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq8kx"
Oct 30 12:45:06.308692 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:45:06.306716    4544 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:45:06.308692 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306718    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-cni-bin\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.308692 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306708    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-etc-kubernetes\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.308692 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306751    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-multus-conf-dir\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.308692 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306746    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-cni-bin\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.308692 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:45:06.306764    4544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a36ceaac-5967-4321-802e-42f4c4b37032-metrics-tls podName:a36ceaac-5967-4321-802e-42f4c4b37032 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:06.806748536 +0000 UTC m=+1.853882473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a36ceaac-5967-4321-802e-42f4c4b37032-metrics-tls") pod "dns-default-xmn5t" (UID: "a36ceaac-5967-4321-802e-42f4c4b37032") : secret "dns-default-metrics-tls" not found
Oct 30 12:45:06.308692 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306807    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2-multus-conf-dir\") pod \"multus-n8f6q\" (UID: \"836d7dec-cc0f-49ae-b1d5-d07a8dcefaf2\") " pod="openshift-multus/multus-n8f6q"
Oct 30 12:45:06.308692 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306844    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-run-ovn-kubernetes\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.308692 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306871    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-cni-netd\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.308692 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306902    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-run-ovn-kubernetes\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.308692 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306920    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-modprobe-d\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.308692 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306930    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623-host-cni-netd\") pod \"ovnkube-node-9zfnt\" (UID: \"0b0e3727-f71f-4edd-bfc3-0b1ef8c4f623\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:06.308692 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.306947    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-system-cni-dir\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:45:06.308692 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.307002    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/940fa76d-d452-41fc-99ad-0f16f51c842e-etc-modprobe-d\") pod \"tuned-79zhh\" (UID: \"940fa76d-d452-41fc-99ad-0f16f51c842e\") " pod="openshift-cluster-node-tuning-operator/tuned-79zhh"
Oct 30 12:45:06.308692 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.307028    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee-system-cni-dir\") pod \"multus-additional-cni-plugins-bdjsz\" (UID: \"d9397b73-2dc3-41bb-b46d-0dd4daaeb3ee\") " pod="openshift-multus/multus-additional-cni-plugins-bdjsz"
Oct 30 12:45:06.308692 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.308349    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d57e941d-1795-48f2-9c7a-7d04c84d0078-metrics-certs\") pod \"network-metrics-daemon-mgctc\" (UID: \"d57e941d-1795-48f2-9c7a-7d04c84d0078\") " pod="openshift-multus/network-metrics-daemon-mgctc"
Oct 30 12:45:06.392858 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.392819    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgctc"
Oct 30 12:45:06.446664 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.446633    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jt6tp"
Oct 30 12:45:06.706612 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.706572    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mgctc"]
Oct 30 12:45:06.712949 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:06.712916    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd57e941d_1795_48f2_9c7a_7d04c84d0078.slice/crio-69089fadcfe25a57711f4065f0688a7e1161fda16abf79b08323a4718a3c97ae WatchSource:0}: Error finding container 69089fadcfe25a57711f4065f0688a7e1161fda16abf79b08323a4718a3c97ae: Status 404 returned error can't find the container with id 69089fadcfe25a57711f4065f0688a7e1161fda16abf79b08323a4718a3c97ae
Oct 30 12:45:06.715553 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.715528    4544 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 12:45:06.771951 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.771919    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:06.811378 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.811335    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b35d2751-096e-4a78-b32c-2d15f831954c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:45:06.811528 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.811404    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a36ceaac-5967-4321-802e-42f4c4b37032-metrics-tls\") pod \"dns-default-xmn5t\" (UID: \"a36ceaac-5967-4321-802e-42f4c4b37032\") " pod="openshift-dns/dns-default-xmn5t"
Oct 30 12:45:06.811528 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:45:06.811497    4544 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:45:06.811588 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:45:06.811557    4544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a36ceaac-5967-4321-802e-42f4c4b37032-metrics-tls podName:a36ceaac-5967-4321-802e-42f4c4b37032 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:07.811541511 +0000 UTC m=+2.858675456 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a36ceaac-5967-4321-802e-42f4c4b37032-metrics-tls") pod "dns-default-xmn5t" (UID: "a36ceaac-5967-4321-802e-42f4c4b37032") : secret "dns-default-metrics-tls" not found
Oct 30 12:45:06.811624 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:45:06.811498    4544 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 12:45:06.811653 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:45:06.811642    4544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b35d2751-096e-4a78-b32c-2d15f831954c-insights-runtime-extractor-tls podName:b35d2751-096e-4a78-b32c-2d15f831954c nodeName:}" failed. No retries permitted until 2026-10-30 12:45:07.811623624 +0000 UTC m=+2.858757557 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b35d2751-096e-4a78-b32c-2d15f831954c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q6dlw" (UID: "b35d2751-096e-4a78-b32c-2d15f831954c") : secret "insights-runtime-extractor-tls" not found
Oct 30 12:45:06.933070 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:06.933017    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:07.067119 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:07.067044    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:07.133787 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:07.133750    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:07.134586 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:07.134550    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mgctc" event={"ID":"d57e941d-1795-48f2-9c7a-7d04c84d0078","Type":"ContainerStarted","Data":"69089fadcfe25a57711f4065f0688a7e1161fda16abf79b08323a4718a3c97ae"}
Oct 30 12:45:07.252986 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:07.252957    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:07.375351 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:07.375314    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:07.479084 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:07.479039    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:07.587462 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:07.587423    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:07.820254 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:07.820153    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a36ceaac-5967-4321-802e-42f4c4b37032-metrics-tls\") pod \"dns-default-xmn5t\" (UID: \"a36ceaac-5967-4321-802e-42f4c4b37032\") " pod="openshift-dns/dns-default-xmn5t"
Oct 30 12:45:07.820254 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:07.820208    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b35d2751-096e-4a78-b32c-2d15f831954c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:45:07.820438 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:45:07.820337    4544 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:45:07.820438 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:45:07.820418    4544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a36ceaac-5967-4321-802e-42f4c4b37032-metrics-tls podName:a36ceaac-5967-4321-802e-42f4c4b37032 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:09.820403826 +0000 UTC m=+4.867537769 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a36ceaac-5967-4321-802e-42f4c4b37032-metrics-tls") pod "dns-default-xmn5t" (UID: "a36ceaac-5967-4321-802e-42f4c4b37032") : secret "dns-default-metrics-tls" not found
Oct 30 12:45:07.820438 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:45:07.820337    4544 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 12:45:07.820561 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:45:07.820484    4544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b35d2751-096e-4a78-b32c-2d15f831954c-insights-runtime-extractor-tls podName:b35d2751-096e-4a78-b32c-2d15f831954c nodeName:}" failed. No retries permitted until 2026-10-30 12:45:09.820472373 +0000 UTC m=+4.867606312 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b35d2751-096e-4a78-b32c-2d15f831954c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q6dlw" (UID: "b35d2751-096e-4a78-b32c-2d15f831954c") : secret "insights-runtime-extractor-tls" not found
Oct 30 12:45:07.970634 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:07.970594    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:08.138587 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:08.138561    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mgctc" event={"ID":"d57e941d-1795-48f2-9c7a-7d04c84d0078","Type":"ContainerStarted","Data":"1565af0cd14a2c942542ec9c3ddfc507e632119de4a41e6ba5990dc656591259"}
Oct 30 12:45:08.430117 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:08.430032    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:08.768387 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:08.768307    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:09.144136 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:09.144099    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mgctc" event={"ID":"d57e941d-1795-48f2-9c7a-7d04c84d0078","Type":"ContainerStarted","Data":"26f3b3c6439442d4fa25285b54669127cfca38ce60f8a8341238f482b74fe716"}
Oct 30 12:45:09.169464 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:09.169431    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:09.365778 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:09.365744    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:09.571151 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:09.571068    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:09.837691 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:09.837652    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a36ceaac-5967-4321-802e-42f4c4b37032-metrics-tls\") pod \"dns-default-xmn5t\" (UID: \"a36ceaac-5967-4321-802e-42f4c4b37032\") " pod="openshift-dns/dns-default-xmn5t"
Oct 30 12:45:09.837952 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:09.837701    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b35d2751-096e-4a78-b32c-2d15f831954c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:45:09.837952 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:45:09.837811    4544 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 12:45:09.837952 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:45:09.837815    4544 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:45:09.837952 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:45:09.837873    4544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b35d2751-096e-4a78-b32c-2d15f831954c-insights-runtime-extractor-tls podName:b35d2751-096e-4a78-b32c-2d15f831954c nodeName:}" failed. No retries permitted until 2026-10-30 12:45:13.837860115 +0000 UTC m=+8.884994045 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b35d2751-096e-4a78-b32c-2d15f831954c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q6dlw" (UID: "b35d2751-096e-4a78-b32c-2d15f831954c") : secret "insights-runtime-extractor-tls" not found
Oct 30 12:45:09.837952 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:45:09.837887    4544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a36ceaac-5967-4321-802e-42f4c4b37032-metrics-tls podName:a36ceaac-5967-4321-802e-42f4c4b37032 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:13.837881361 +0000 UTC m=+8.885015293 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a36ceaac-5967-4321-802e-42f4c4b37032-metrics-tls") pod "dns-default-xmn5t" (UID: "a36ceaac-5967-4321-802e-42f4c4b37032") : secret "dns-default-metrics-tls" not found
Oct 30 12:45:09.971726 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:09.971687    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:10.770700 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:10.770660    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:10.878892 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:10.878859    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:11.168017 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:11.167982    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:11.373579 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:11.373542    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:11.568557 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:11.568460    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:11.978383 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:11.978349    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:12.092780 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:12.092748    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:12.563125 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:12.563094    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:12.563350 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:12.563314    4544 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 12:45:12.782872 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:12.782838    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:13.175935 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:13.175900    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:13.369910 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:13.369878    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:13.574366 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:13.574328    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:13.760652 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:13.760614    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:13.868045 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:13.867951    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a36ceaac-5967-4321-802e-42f4c4b37032-metrics-tls\") pod \"dns-default-xmn5t\" (UID: \"a36ceaac-5967-4321-802e-42f4c4b37032\") " pod="openshift-dns/dns-default-xmn5t"
Oct 30 12:45:13.868045 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:13.868000    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b35d2751-096e-4a78-b32c-2d15f831954c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:45:13.870433 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:13.870404    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b35d2751-096e-4a78-b32c-2d15f831954c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6dlw\" (UID: \"b35d2751-096e-4a78-b32c-2d15f831954c\") " pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:45:13.870433 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:13.870431    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a36ceaac-5967-4321-802e-42f4c4b37032-metrics-tls\") pod \"dns-default-xmn5t\" (UID: \"a36ceaac-5967-4321-802e-42f4c4b37032\") " pod="openshift-dns/dns-default-xmn5t"
Oct 30 12:45:13.891561 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:13.891527    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:13.898621 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:13.898593    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xmn5t"
Oct 30 12:45:13.899690 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:13.899671    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q6dlw"
Oct 30 12:45:14.004544 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:14.004506    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:14.185712 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:14.185617    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:14.204439 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:14.204404    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q6dlw"]
Oct 30 12:45:14.210794 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:14.210765    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb35d2751_096e_4a78_b32c_2d15f831954c.slice/crio-9bb3f2f4a2d3f7f3eef0d94f8d8f9fbd1156fb338eb77b34c1142a8cb036208a WatchSource:0}: Error finding container 9bb3f2f4a2d3f7f3eef0d94f8d8f9fbd1156fb338eb77b34c1142a8cb036208a: Status 404 returned error can't find the container with id 9bb3f2f4a2d3f7f3eef0d94f8d8f9fbd1156fb338eb77b34c1142a8cb036208a
Oct 30 12:45:14.217069 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:14.217047    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xmn5t"]
Oct 30 12:45:14.223500 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:14.223475    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda36ceaac_5967_4321_802e_42f4c4b37032.slice/crio-64e6c479251b5e7a5d02060712dcd87fac7e7d203c185f91ddd742b3f4a22655 WatchSource:0}: Error finding container 64e6c479251b5e7a5d02060712dcd87fac7e7d203c185f91ddd742b3f4a22655: Status 404 returned error can't find the container with id 64e6c479251b5e7a5d02060712dcd87fac7e7d203c185f91ddd742b3f4a22655
Oct 30 12:45:14.293772 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:14.293739    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:14.410630 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:14.410593    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:14.550405 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:14.550317    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:14.661604 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:14.661570    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:14.778021 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:14.777982    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:14.961736 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:14.961696    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:15.169187 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:15.169142    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q6dlw" event={"ID":"b35d2751-096e-4a78-b32c-2d15f831954c","Type":"ContainerStarted","Data":"0c034e368de321ad2290fbd10979db08497ab7c42812f1231cb5bbbb6f54dd2b"}
Oct 30 12:45:15.169369 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:15.169246    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q6dlw" event={"ID":"b35d2751-096e-4a78-b32c-2d15f831954c","Type":"ContainerStarted","Data":"e452faa9d6bb2ef5ed6411f31001bf25ce2b0eaa5216a154bfcb3635ee6b6009"}
Oct 30 12:45:15.169369 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:15.169281    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q6dlw" event={"ID":"b35d2751-096e-4a78-b32c-2d15f831954c","Type":"ContainerStarted","Data":"9bb3f2f4a2d3f7f3eef0d94f8d8f9fbd1156fb338eb77b34c1142a8cb036208a"}
Oct 30 12:45:15.170456 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:15.170425    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xmn5t" event={"ID":"a36ceaac-5967-4321-802e-42f4c4b37032","Type":"ContainerStarted","Data":"64e6c479251b5e7a5d02060712dcd87fac7e7d203c185f91ddd742b3f4a22655"}
Oct 30 12:45:15.213804 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:15.213669    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:16.176872 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:16.176824    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xmn5t" event={"ID":"a36ceaac-5967-4321-802e-42f4c4b37032","Type":"ContainerStarted","Data":"cd9ab63eeaa191ac2cb0ee6d99c643ea3fbace62319e5062e8958e9bc88ba225"}
Oct 30 12:45:16.177054 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:16.176887    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xmn5t" event={"ID":"a36ceaac-5967-4321-802e-42f4c4b37032","Type":"ContainerStarted","Data":"2b77f0a0ecccfc1076b4db4d791cd48eab385a73e042b0eb3f16cbdb0e884d54"}
Oct 30 12:45:16.177054 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:16.176992    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xmn5t"
Oct 30 12:45:16.225387 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:16.225340    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:16.339452 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:16.339407    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:17.182600 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:17.182562    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q6dlw" event={"ID":"b35d2751-096e-4a78-b32c-2d15f831954c","Type":"ContainerStarted","Data":"3b792c4d835a9c8445e026fef9ce573a2567abf7bfdec72c1ea5448c81e7964a"}
Oct 30 12:45:20.465984 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.465949    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qvzmt"]
Oct 30 12:45:20.469063 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.469038    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qvzmt"
Oct 30 12:45:20.479412 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.479385    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-km7rm"]
Oct 30 12:45:20.482195 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.482169    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-km7rm"
Oct 30 12:45:20.491631 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.491604    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-bf67cd67d-sm5rz"]
Oct 30 12:45:20.494427 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.494409    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bf67cd67d-sm5rz"
Oct 30 12:45:20.501194 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.501131    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 12:45:20.501320 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.501237    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-d2zth\""
Oct 30 12:45:20.503702 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.503683    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qvzmt"]
Oct 30 12:45:20.522268 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.522235    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 12:45:20.522760 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.522741    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 12:45:20.528211 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.528189    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 12:45:20.528306 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.528257    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 12:45:20.528359 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.528347    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 12:45:20.538624 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.538595    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-km7rm"]
Oct 30 12:45:20.539088 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.539067    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bf67cd67d-sm5rz"]
Oct 30 12:45:20.575572 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.575537    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5cc44fbdc6-z6lmh"]
Oct 30 12:45:20.578670 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.578643    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cc44fbdc6-z6lmh"
Oct 30 12:45:20.613082 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.613006    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 12:45:20.613082 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.613006    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-xs6lh\""
Oct 30 12:45:20.613082 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.613040    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 12:45:20.613082 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.613070    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 12:45:20.613379 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.613106    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 12:45:20.615346 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.615318    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8b8c8\""
Oct 30 12:45:20.615346 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.615321    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 12:45:20.615534 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.615324    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jpdrb\""
Oct 30 12:45:20.615534 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.615392    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 12:45:20.618034 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.618016    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05cbf75e-a21f-475f-afef-6e7c55cf0551-tmp\") pod \"console-bf67cd67d-sm5rz\" (UID: \"05cbf75e-a21f-475f-afef-6e7c55cf0551\") " pod="openshift-console/console-bf67cd67d-sm5rz"
Oct 30 12:45:20.618081 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.618046    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28d577e6-7b12-40a1-98b4-4bc41023ec17-tmp\") pod \"downloads-6dd789c8b7-km7rm\" (UID: \"28d577e6-7b12-40a1-98b4-4bc41023ec17\") " pod="openshift-console/downloads-6dd789c8b7-km7rm"
Oct 30 12:45:20.618081 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.618066    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05cbf75e-a21f-475f-afef-6e7c55cf0551-console-oauth-config\") pod \"console-bf67cd67d-sm5rz\" (UID: \"05cbf75e-a21f-475f-afef-6e7c55cf0551\") " pod="openshift-console/console-bf67cd67d-sm5rz"
Oct 30 12:45:20.618146 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.618085    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05cbf75e-a21f-475f-afef-6e7c55cf0551-console-config\") pod \"console-bf67cd67d-sm5rz\" (UID: \"05cbf75e-a21f-475f-afef-6e7c55cf0551\") " pod="openshift-console/console-bf67cd67d-sm5rz"
Oct 30 12:45:20.618188 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.618141    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05cbf75e-a21f-475f-afef-6e7c55cf0551-service-ca\") pod \"console-bf67cd67d-sm5rz\" (UID: \"05cbf75e-a21f-475f-afef-6e7c55cf0551\") " pod="openshift-console/console-bf67cd67d-sm5rz"
Oct 30 12:45:20.618348 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.618329    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0f5859be-f23e-4aa3-8f6b-df7c77f1caa4-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-qvzmt\" (UID: \"0f5859be-f23e-4aa3-8f6b-df7c77f1caa4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qvzmt"
Oct 30 12:45:20.618393 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.618368    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05cbf75e-a21f-475f-afef-6e7c55cf0551-oauth-serving-cert\") pod \"console-bf67cd67d-sm5rz\" (UID: \"05cbf75e-a21f-475f-afef-6e7c55cf0551\") " pod="openshift-console/console-bf67cd67d-sm5rz"
Oct 30 12:45:20.618393 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.618390    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29wwj\" (UniqueName: \"kubernetes.io/projected/05cbf75e-a21f-475f-afef-6e7c55cf0551-kube-api-access-29wwj\") pod \"console-bf67cd67d-sm5rz\" (UID: \"05cbf75e-a21f-475f-afef-6e7c55cf0551\") " pod="openshift-console/console-bf67cd67d-sm5rz"
Oct 30 12:45:20.618468 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.618453    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rkcr\" (UniqueName: \"kubernetes.io/projected/28d577e6-7b12-40a1-98b4-4bc41023ec17-kube-api-access-6rkcr\") pod \"downloads-6dd789c8b7-km7rm\" (UID: \"28d577e6-7b12-40a1-98b4-4bc41023ec17\") " pod="openshift-console/downloads-6dd789c8b7-km7rm"
Oct 30 12:45:20.618503 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.618494    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05cbf75e-a21f-475f-afef-6e7c55cf0551-console-serving-cert\") pod \"console-bf67cd67d-sm5rz\" (UID: \"05cbf75e-a21f-475f-afef-6e7c55cf0551\") " pod="openshift-console/console-bf67cd67d-sm5rz"
Oct 30 12:45:20.625885 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.625860    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cc44fbdc6-z6lmh"]
Oct 30 12:45:20.643047 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.643009    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:20.719235 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.719139    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0f5859be-f23e-4aa3-8f6b-df7c77f1caa4-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-qvzmt\" (UID: \"0f5859be-f23e-4aa3-8f6b-df7c77f1caa4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qvzmt"
Oct 30 12:45:20.719235 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.719180    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxpdt\" (UniqueName: \"kubernetes.io/projected/4df72e77-b759-4a2a-b9c1-808f30a3a1ae-kube-api-access-qxpdt\") pod \"image-registry-5cc44fbdc6-z6lmh\" (UID: \"4df72e77-b759-4a2a-b9c1-808f30a3a1ae\") " pod="openshift-image-registry/image-registry-5cc44fbdc6-z6lmh"
Oct 30 12:45:20.719235 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.719198    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4df72e77-b759-4a2a-b9c1-808f30a3a1ae-bound-sa-token\") pod \"image-registry-5cc44fbdc6-z6lmh\" (UID: \"4df72e77-b759-4a2a-b9c1-808f30a3a1ae\") " pod="openshift-image-registry/image-registry-5cc44fbdc6-z6lmh"
Oct 30 12:45:20.719458 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.719330    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4df72e77-b759-4a2a-b9c1-808f30a3a1ae-installation-pull-secrets\") pod \"image-registry-5cc44fbdc6-z6lmh\" (UID: \"4df72e77-b759-4a2a-b9c1-808f30a3a1ae\") " pod="openshift-image-registry/image-registry-5cc44fbdc6-z6lmh"
Oct 30 12:45:20.719458 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.719371    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4df72e77-b759-4a2a-b9c1-808f30a3a1ae-trusted-ca\") pod \"image-registry-5cc44fbdc6-z6lmh\" (UID: \"4df72e77-b759-4a2a-b9c1-808f30a3a1ae\") " pod="openshift-image-registry/image-registry-5cc44fbdc6-z6lmh"
Oct 30 12:45:20.719458 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.719393    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05cbf75e-a21f-475f-afef-6e7c55cf0551-oauth-serving-cert\") pod \"console-bf67cd67d-sm5rz\" (UID: \"05cbf75e-a21f-475f-afef-6e7c55cf0551\") " pod="openshift-console/console-bf67cd67d-sm5rz"
Oct 30 12:45:20.719458 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.719446    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29wwj\" (UniqueName: \"kubernetes.io/projected/05cbf75e-a21f-475f-afef-6e7c55cf0551-kube-api-access-29wwj\") pod \"console-bf67cd67d-sm5rz\" (UID: \"05cbf75e-a21f-475f-afef-6e7c55cf0551\") " pod="openshift-console/console-bf67cd67d-sm5rz"
Oct 30 12:45:20.719591 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.719561    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rkcr\" (UniqueName: \"kubernetes.io/projected/28d577e6-7b12-40a1-98b4-4bc41023ec17-kube-api-access-6rkcr\") pod \"downloads-6dd789c8b7-km7rm\" (UID: \"28d577e6-7b12-40a1-98b4-4bc41023ec17\") " pod="openshift-console/downloads-6dd789c8b7-km7rm"
Oct 30 12:45:20.719626 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.719594    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05cbf75e-a21f-475f-afef-6e7c55cf0551-console-serving-cert\") pod \"console-bf67cd67d-sm5rz\" (UID: \"05cbf75e-a21f-475f-afef-6e7c55cf0551\") " pod="openshift-console/console-bf67cd67d-sm5rz"
Oct 30 12:45:20.719672 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.719626    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05cbf75e-a21f-475f-afef-6e7c55cf0551-tmp\") pod \"console-bf67cd67d-sm5rz\" (UID: \"05cbf75e-a21f-475f-afef-6e7c55cf0551\") " pod="openshift-console/console-bf67cd67d-sm5rz"
Oct 30 12:45:20.719672 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.719658    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4df72e77-b759-4a2a-b9c1-808f30a3a1ae-ca-trust-extracted\") pod \"image-registry-5cc44fbdc6-z6lmh\" (UID: \"4df72e77-b759-4a2a-b9c1-808f30a3a1ae\") " pod="openshift-image-registry/image-registry-5cc44fbdc6-z6lmh"
Oct 30 12:45:20.719779 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.719756    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4df72e77-b759-4a2a-b9c1-808f30a3a1ae-registry-certificates\") pod \"image-registry-5cc44fbdc6-z6lmh\" (UID: \"4df72e77-b759-4a2a-b9c1-808f30a3a1ae\") " pod="openshift-image-registry/image-registry-5cc44fbdc6-z6lmh"
Oct 30 12:45:20.719851 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.719836    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28d577e6-7b12-40a1-98b4-4bc41023ec17-tmp\") pod \"downloads-6dd789c8b7-km7rm\" (UID: \"28d577e6-7b12-40a1-98b4-4bc41023ec17\") " pod="openshift-console/downloads-6dd789c8b7-km7rm"
Oct 30 12:45:20.719889 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.719867    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05cbf75e-a21f-475f-afef-6e7c55cf0551-console-oauth-config\") pod \"console-bf67cd67d-sm5rz\" (UID: \"05cbf75e-a21f-475f-afef-6e7c55cf0551\") " pod="openshift-console/console-bf67cd67d-sm5rz"
Oct 30 12:45:20.719925 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.719893    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05cbf75e-a21f-475f-afef-6e7c55cf0551-console-config\") pod \"console-bf67cd67d-sm5rz\" (UID: \"05cbf75e-a21f-475f-afef-6e7c55cf0551\") " pod="openshift-console/console-bf67cd67d-sm5rz"
Oct 30 12:45:20.719972 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.719921    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4df72e77-b759-4a2a-b9c1-808f30a3a1ae-image-registry-private-configuration\") pod \"image-registry-5cc44fbdc6-z6lmh\" (UID: \"4df72e77-b759-4a2a-b9c1-808f30a3a1ae\") " pod="openshift-image-registry/image-registry-5cc44fbdc6-z6lmh"
Oct 30 12:45:20.719972 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.719950    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05cbf75e-a21f-475f-afef-6e7c55cf0551-service-ca\") pod \"console-bf67cd67d-sm5rz\" (UID: \"05cbf75e-a21f-475f-afef-6e7c55cf0551\") " pod="openshift-console/console-bf67cd67d-sm5rz"
Oct 30 12:45:20.720068 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.719978    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4df72e77-b759-4a2a-b9c1-808f30a3a1ae-registry-tls\") pod \"image-registry-5cc44fbdc6-z6lmh\" (UID: \"4df72e77-b759-4a2a-b9c1-808f30a3a1ae\") " pod="openshift-image-registry/image-registry-5cc44fbdc6-z6lmh"
Oct 30 12:45:20.720117 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.720091    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05cbf75e-a21f-475f-afef-6e7c55cf0551-tmp\") pod \"console-bf67cd67d-sm5rz\" (UID: \"05cbf75e-a21f-475f-afef-6e7c55cf0551\") " pod="openshift-console/console-bf67cd67d-sm5rz"
Oct 30 12:45:20.720205 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.720162    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05cbf75e-a21f-475f-afef-6e7c55cf0551-oauth-serving-cert\") pod \"console-bf67cd67d-sm5rz\" (UID: \"05cbf75e-a21f-475f-afef-6e7c55cf0551\") " pod="openshift-console/console-bf67cd67d-sm5rz"
Oct 30 12:45:20.720344 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.720325    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28d577e6-7b12-40a1-98b4-4bc41023ec17-tmp\") pod \"downloads-6dd789c8b7-km7rm\" (UID: \"28d577e6-7b12-40a1-98b4-4bc41023ec17\") " pod="openshift-console/downloads-6dd789c8b7-km7rm"
Oct 30 12:45:20.720701 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.720681    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05cbf75e-a21f-475f-afef-6e7c55cf0551-console-config\") pod \"console-bf67cd67d-sm5rz\" (UID: \"05cbf75e-a21f-475f-afef-6e7c55cf0551\") " pod="openshift-console/console-bf67cd67d-sm5rz"
Oct 30 12:45:20.720799 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.720776    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05cbf75e-a21f-475f-afef-6e7c55cf0551-service-ca\") pod \"console-bf67cd67d-sm5rz\" (UID: \"05cbf75e-a21f-475f-afef-6e7c55cf0551\") " pod="openshift-console/console-bf67cd67d-sm5rz"
Oct 30 12:45:20.722036 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.722010    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0f5859be-f23e-4aa3-8f6b-df7c77f1caa4-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-qvzmt\" (UID: \"0f5859be-f23e-4aa3-8f6b-df7c77f1caa4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qvzmt"
Oct 30 12:45:20.722270 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.722253    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05cbf75e-a21f-475f-afef-6e7c55cf0551-console-serving-cert\") pod \"console-bf67cd67d-sm5rz\" (UID: \"05cbf75e-a21f-475f-afef-6e7c55cf0551\") " pod="openshift-console/console-bf67cd67d-sm5rz"
Oct 30 12:45:20.722368 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.722352    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05cbf75e-a21f-475f-afef-6e7c55cf0551-console-oauth-config\") pod \"console-bf67cd67d-sm5rz\" (UID: \"05cbf75e-a21f-475f-afef-6e7c55cf0551\") " pod="openshift-console/console-bf67cd67d-sm5rz"
Oct 30 12:45:20.748929 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.748898    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29wwj\" (UniqueName: \"kubernetes.io/projected/05cbf75e-a21f-475f-afef-6e7c55cf0551-kube-api-access-29wwj\") pod \"console-bf67cd67d-sm5rz\" (UID: \"05cbf75e-a21f-475f-afef-6e7c55cf0551\") " pod="openshift-console/console-bf67cd67d-sm5rz"
Oct 30 12:45:20.751811 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.751782    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rkcr\" (UniqueName: \"kubernetes.io/projected/28d577e6-7b12-40a1-98b4-4bc41023ec17-kube-api-access-6rkcr\") pod \"downloads-6dd789c8b7-km7rm\" (UID: \"28d577e6-7b12-40a1-98b4-4bc41023ec17\") " pod="openshift-console/downloads-6dd789c8b7-km7rm"
Oct 30 12:45:20.780147 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.780112    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qvzmt"
Oct 30 12:45:20.793466 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.793434    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-km7rm"
Oct 30 12:45:20.805749 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.805721    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bf67cd67d-sm5rz"
Oct 30 12:45:20.823690 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.823634    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4df72e77-b759-4a2a-b9c1-808f30a3a1ae-ca-trust-extracted\") pod \"image-registry-5cc44fbdc6-z6lmh\" (UID: \"4df72e77-b759-4a2a-b9c1-808f30a3a1ae\") " pod="openshift-image-registry/image-registry-5cc44fbdc6-z6lmh"
Oct 30 12:45:20.823817 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.823703    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4df72e77-b759-4a2a-b9c1-808f30a3a1ae-registry-certificates\") pod \"image-registry-5cc44fbdc6-z6lmh\" (UID: \"4df72e77-b759-4a2a-b9c1-808f30a3a1ae\") " pod="openshift-image-registry/image-registry-5cc44fbdc6-z6lmh"
Oct 30 12:45:20.823817 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.823749    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4df72e77-b759-4a2a-b9c1-808f30a3a1ae-image-registry-private-configuration\") pod \"image-registry-5cc44fbdc6-z6lmh\" (UID: \"4df72e77-b759-4a2a-b9c1-808f30a3a1ae\") " pod="openshift-image-registry/image-registry-5cc44fbdc6-z6lmh"
Oct 30 12:45:20.823817 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.823784    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4df72e77-b759-4a2a-b9c1-808f30a3a1ae-registry-tls\") pod \"image-registry-5cc44fbdc6-z6lmh\" (UID: \"4df72e77-b759-4a2a-b9c1-808f30a3a1ae\") " pod="openshift-image-registry/image-registry-5cc44fbdc6-z6lmh"
Oct 30 12:45:20.823928 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.823815    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxpdt\" (UniqueName: \"kubernetes.io/projected/4df72e77-b759-4a2a-b9c1-808f30a3a1ae-kube-api-access-qxpdt\") pod \"image-registry-5cc44fbdc6-z6lmh\" (UID: \"4df72e77-b759-4a2a-b9c1-808f30a3a1ae\") " pod="openshift-image-registry/image-registry-5cc44fbdc6-z6lmh"
Oct 30 12:45:20.823928 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.823833    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4df72e77-b759-4a2a-b9c1-808f30a3a1ae-bound-sa-token\") pod \"image-registry-5cc44fbdc6-z6lmh\" (UID: \"4df72e77-b759-4a2a-b9c1-808f30a3a1ae\") " pod="openshift-image-registry/image-registry-5cc44fbdc6-z6lmh"
Oct 30 12:45:20.823928 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.823856    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4df72e77-b759-4a2a-b9c1-808f30a3a1ae-installation-pull-secrets\") pod \"image-registry-5cc44fbdc6-z6lmh\" (UID: \"4df72e77-b759-4a2a-b9c1-808f30a3a1ae\") " pod="openshift-image-registry/image-registry-5cc44fbdc6-z6lmh"
Oct 30 12:45:20.823928 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.823881    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4df72e77-b759-4a2a-b9c1-808f30a3a1ae-trusted-ca\") pod \"image-registry-5cc44fbdc6-z6lmh\" (UID: \"4df72e77-b759-4a2a-b9c1-808f30a3a1ae\") " pod="openshift-image-registry/image-registry-5cc44fbdc6-z6lmh"
Oct 30 12:45:20.824846 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.824816    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4df72e77-b759-4a2a-b9c1-808f30a3a1ae-trusted-ca\") pod \"image-registry-5cc44fbdc6-z6lmh\" (UID: \"4df72e77-b759-4a2a-b9c1-808f30a3a1ae\") " pod="openshift-image-registry/image-registry-5cc44fbdc6-z6lmh"
Oct 30 12:45:20.824919 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.824894    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4df72e77-b759-4a2a-b9c1-808f30a3a1ae-ca-trust-extracted\") pod \"image-registry-5cc44fbdc6-z6lmh\" (UID: \"4df72e77-b759-4a2a-b9c1-808f30a3a1ae\") " pod="openshift-image-registry/image-registry-5cc44fbdc6-z6lmh"
Oct 30 12:45:20.825693 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.825638    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4df72e77-b759-4a2a-b9c1-808f30a3a1ae-registry-certificates\") pod \"image-registry-5cc44fbdc6-z6lmh\" (UID: \"4df72e77-b759-4a2a-b9c1-808f30a3a1ae\") " pod="openshift-image-registry/image-registry-5cc44fbdc6-z6lmh"
Oct 30 12:45:20.828168 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.828144    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4df72e77-b759-4a2a-b9c1-808f30a3a1ae-image-registry-private-configuration\") pod \"image-registry-5cc44fbdc6-z6lmh\" (UID: \"4df72e77-b759-4a2a-b9c1-808f30a3a1ae\") " pod="openshift-image-registry/image-registry-5cc44fbdc6-z6lmh"
Oct 30 12:45:20.828552 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.828534    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4df72e77-b759-4a2a-b9c1-808f30a3a1ae-installation-pull-secrets\") pod \"image-registry-5cc44fbdc6-z6lmh\" (UID: \"4df72e77-b759-4a2a-b9c1-808f30a3a1ae\") " pod="openshift-image-registry/image-registry-5cc44fbdc6-z6lmh"
Oct 30 12:45:20.829643 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.829543    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4df72e77-b759-4a2a-b9c1-808f30a3a1ae-registry-tls\") pod \"image-registry-5cc44fbdc6-z6lmh\" (UID: \"4df72e77-b759-4a2a-b9c1-808f30a3a1ae\") " pod="openshift-image-registry/image-registry-5cc44fbdc6-z6lmh"
Oct 30 12:45:20.849532 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.849501    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxpdt\" (UniqueName: \"kubernetes.io/projected/4df72e77-b759-4a2a-b9c1-808f30a3a1ae-kube-api-access-qxpdt\") pod \"image-registry-5cc44fbdc6-z6lmh\" (UID: \"4df72e77-b759-4a2a-b9c1-808f30a3a1ae\") " pod="openshift-image-registry/image-registry-5cc44fbdc6-z6lmh"
Oct 30 12:45:20.854828 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.854764    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4df72e77-b759-4a2a-b9c1-808f30a3a1ae-bound-sa-token\") pod \"image-registry-5cc44fbdc6-z6lmh\" (UID: \"4df72e77-b759-4a2a-b9c1-808f30a3a1ae\") " pod="openshift-image-registry/image-registry-5cc44fbdc6-z6lmh"
Oct 30 12:45:20.861735 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.861640    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:20.890621 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.890381    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cc44fbdc6-z6lmh"
Oct 30 12:45:20.989606 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:20.989520    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:21.084773 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:21.084741    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qvzmt"]
Oct 30 12:45:21.091989 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:21.091949    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f5859be_f23e_4aa3_8f6b_df7c77f1caa4.slice/crio-cf9d50061e822c84f0d19d0130a047d4697c97767bc1187f3d311c5969ae3edd WatchSource:0}: Error finding container cf9d50061e822c84f0d19d0130a047d4697c97767bc1187f3d311c5969ae3edd: Status 404 returned error can't find the container with id cf9d50061e822c84f0d19d0130a047d4697c97767bc1187f3d311c5969ae3edd
Oct 30 12:45:21.101472 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:21.101437    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-km7rm"]
Oct 30 12:45:21.106802 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:21.106736    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28d577e6_7b12_40a1_98b4_4bc41023ec17.slice/crio-1c32a1b15ff10bbc3f34c1cbeaa32e9da98c13f52fe48216e6c3958c7082710f WatchSource:0}: Error finding container 1c32a1b15ff10bbc3f34c1cbeaa32e9da98c13f52fe48216e6c3958c7082710f: Status 404 returned error can't find the container with id 1c32a1b15ff10bbc3f34c1cbeaa32e9da98c13f52fe48216e6c3958c7082710f
Oct 30 12:45:21.126537 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:21.126503    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bf67cd67d-sm5rz"]
Oct 30 12:45:21.134852 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:21.134822    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05cbf75e_a21f_475f_afef_6e7c55cf0551.slice/crio-bc014583e7d4b114fcb4b61de3a744003feba3ae621ad8bc049f8f0a62f93d43 WatchSource:0}: Error finding container bc014583e7d4b114fcb4b61de3a744003feba3ae621ad8bc049f8f0a62f93d43: Status 404 returned error can't find the container with id bc014583e7d4b114fcb4b61de3a744003feba3ae621ad8bc049f8f0a62f93d43
Oct 30 12:45:21.195904 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:21.195864    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cc44fbdc6-z6lmh"]
Oct 30 12:45:21.198882 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:21.198814    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-km7rm" event={"ID":"28d577e6-7b12-40a1-98b4-4bc41023ec17","Type":"ContainerStarted","Data":"1c32a1b15ff10bbc3f34c1cbeaa32e9da98c13f52fe48216e6c3958c7082710f"}
Oct 30 12:45:21.200236 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:21.200190    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bf67cd67d-sm5rz" event={"ID":"05cbf75e-a21f-475f-afef-6e7c55cf0551","Type":"ContainerStarted","Data":"bc014583e7d4b114fcb4b61de3a744003feba3ae621ad8bc049f8f0a62f93d43"}
Oct 30 12:45:21.201183 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:21.201157    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df72e77_b759_4a2a_b9c1_808f30a3a1ae.slice/crio-33c7d7a17ad18a36330230ef46522b6f92ffbb35ad847e99397df86193b67736 WatchSource:0}: Error finding container 33c7d7a17ad18a36330230ef46522b6f92ffbb35ad847e99397df86193b67736: Status 404 returned error can't find the container with id 33c7d7a17ad18a36330230ef46522b6f92ffbb35ad847e99397df86193b67736
Oct 30 12:45:21.205334 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:21.201385    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qvzmt" event={"ID":"0f5859be-f23e-4aa3-8f6b-df7c77f1caa4","Type":"ContainerStarted","Data":"cf9d50061e822c84f0d19d0130a047d4697c97767bc1187f3d311c5969ae3edd"}
Oct 30 12:45:22.207961 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:22.207923    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cc44fbdc6-z6lmh" event={"ID":"4df72e77-b759-4a2a-b9c1-808f30a3a1ae","Type":"ContainerStarted","Data":"0d76106197defb9985b5963f2db137e27b8c047804248156095eacec5eedbd70"}
Oct 30 12:45:22.208462 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:22.207990    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cc44fbdc6-z6lmh" event={"ID":"4df72e77-b759-4a2a-b9c1-808f30a3a1ae","Type":"ContainerStarted","Data":"33c7d7a17ad18a36330230ef46522b6f92ffbb35ad847e99397df86193b67736"}
Oct 30 12:45:22.208462 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:22.208248    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5cc44fbdc6-z6lmh"
Oct 30 12:45:22.255726 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:22.255672    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5cc44fbdc6-z6lmh" podStartSLOduration=2.255655189 podStartE2EDuration="2.255655189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:22.25500227 +0000 UTC m=+17.302136224" watchObservedRunningTime="2026-10-30 12:45:22.255655189 +0000 UTC m=+17.302789137"
Oct 30 12:45:23.213873 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:23.213824    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qvzmt" event={"ID":"0f5859be-f23e-4aa3-8f6b-df7c77f1caa4","Type":"ContainerStarted","Data":"67c9fb07d93e0b8fd6df8d55ed84af1ad8dcca345e8caa684c44f821cfa0241a"}
Oct 30 12:45:23.254746 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:23.254674    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qvzmt" podStartSLOduration=2.083281997 podStartE2EDuration="3.254654349s" totalImagesPullingTime="1.171372352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:23.253840245 +0000 UTC m=+18.300974202" watchObservedRunningTime="2026-10-30 12:45:23.254654349 +0000 UTC m=+18.301788304"
Oct 30 12:45:24.218562 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:24.218462    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qvzmt"
Oct 30 12:45:24.224866 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:24.224790    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qvzmt"
Oct 30 12:45:24.482080 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:24.482047    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:24.595390 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:24.595352    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-2w7nn"]
Oct 30 12:45:24.599023 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:24.598990    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2w7nn"
Oct 30 12:45:24.602644 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:24.602607    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:24.615908 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:24.615868    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 12:45:24.616153 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:24.616131    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 12:45:24.616212 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:24.616192    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 12:45:24.616301 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:24.616292    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 12:45:24.616342 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:24.616302    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 12:45:24.616449 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:24.616437    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-jjjh6\""
Oct 30 12:45:24.629094 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:24.629061    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-2w7nn"]
Oct 30 12:45:24.738100 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:24.738012    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:24.757064 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:24.757025    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4npf6\" (UniqueName: \"kubernetes.io/projected/b3b4295c-1ef2-400d-a7e6-f2cc60f39cb6-kube-api-access-4npf6\") pod \"prometheus-operator-9cd89b74c-2w7nn\" (UID: \"b3b4295c-1ef2-400d-a7e6-f2cc60f39cb6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2w7nn"
Oct 30 12:45:24.757260 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:24.757069    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3b4295c-1ef2-400d-a7e6-f2cc60f39cb6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-2w7nn\" (UID: \"b3b4295c-1ef2-400d-a7e6-f2cc60f39cb6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2w7nn"
Oct 30 12:45:24.757260 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:24.757113    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3b4295c-1ef2-400d-a7e6-f2cc60f39cb6-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-2w7nn\" (UID: \"b3b4295c-1ef2-400d-a7e6-f2cc60f39cb6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2w7nn"
Oct 30 12:45:24.757260 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:24.757209    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3b4295c-1ef2-400d-a7e6-f2cc60f39cb6-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-2w7nn\" (UID: \"b3b4295c-1ef2-400d-a7e6-f2cc60f39cb6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2w7nn"
Oct 30 12:45:24.858489 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:24.858450    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3b4295c-1ef2-400d-a7e6-f2cc60f39cb6-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-2w7nn\" (UID: \"b3b4295c-1ef2-400d-a7e6-f2cc60f39cb6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2w7nn"
Oct 30 12:45:24.858691 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:24.858510    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3b4295c-1ef2-400d-a7e6-f2cc60f39cb6-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-2w7nn\" (UID: \"b3b4295c-1ef2-400d-a7e6-f2cc60f39cb6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2w7nn"
Oct 30 12:45:24.858691 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:24.858585    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4npf6\" (UniqueName: \"kubernetes.io/projected/b3b4295c-1ef2-400d-a7e6-f2cc60f39cb6-kube-api-access-4npf6\") pod \"prometheus-operator-9cd89b74c-2w7nn\" (UID: \"b3b4295c-1ef2-400d-a7e6-f2cc60f39cb6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2w7nn"
Oct 30 12:45:24.858691 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:24.858621    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3b4295c-1ef2-400d-a7e6-f2cc60f39cb6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-2w7nn\" (UID: \"b3b4295c-1ef2-400d-a7e6-f2cc60f39cb6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2w7nn"
Oct 30 12:45:24.859393 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:24.859367    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3b4295c-1ef2-400d-a7e6-f2cc60f39cb6-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-2w7nn\" (UID: \"b3b4295c-1ef2-400d-a7e6-f2cc60f39cb6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2w7nn"
Oct 30 12:45:24.861274 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:24.861251    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3b4295c-1ef2-400d-a7e6-f2cc60f39cb6-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-2w7nn\" (UID: \"b3b4295c-1ef2-400d-a7e6-f2cc60f39cb6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2w7nn"
Oct 30 12:45:24.861397 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:24.861293    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3b4295c-1ef2-400d-a7e6-f2cc60f39cb6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-2w7nn\" (UID: \"b3b4295c-1ef2-400d-a7e6-f2cc60f39cb6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2w7nn"
Oct 30 12:45:24.882767 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:24.882731    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4npf6\" (UniqueName: \"kubernetes.io/projected/b3b4295c-1ef2-400d-a7e6-f2cc60f39cb6-kube-api-access-4npf6\") pod \"prometheus-operator-9cd89b74c-2w7nn\" (UID: \"b3b4295c-1ef2-400d-a7e6-f2cc60f39cb6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2w7nn"
Oct 30 12:45:24.915207 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:24.915171    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2w7nn"
Oct 30 12:45:25.108408 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:25.108367    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9zfnt"
Oct 30 12:45:25.226266 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:25.226203    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bf67cd67d-sm5rz" event={"ID":"05cbf75e-a21f-475f-afef-6e7c55cf0551","Type":"ContainerStarted","Data":"1722daa2614ed284003bfb572360d8abe6c618a3ef969becba8962044e0e6d65"}
Oct 30 12:45:25.231450 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:25.231419    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-2w7nn"]
Oct 30 12:45:25.237117 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:25.237086    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3b4295c_1ef2_400d_a7e6_f2cc60f39cb6.slice/crio-294bf8bac912b7e272e4e76f90ee29f7e134f0c5efdf30cfe377b8efd238a0e8 WatchSource:0}: Error finding container 294bf8bac912b7e272e4e76f90ee29f7e134f0c5efdf30cfe377b8efd238a0e8: Status 404 returned error can't find the container with id 294bf8bac912b7e272e4e76f90ee29f7e134f0c5efdf30cfe377b8efd238a0e8
Oct 30 12:45:25.268201 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:25.268141    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-bf67cd67d-sm5rz" podStartSLOduration=1.9934870070000001 podStartE2EDuration="5.268124512s" totalImagesPullingTime="3.274637505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:25.267997902 +0000 UTC m=+20.315131857" watchObservedRunningTime="2026-10-30 12:45:25.268124512 +0000 UTC m=+20.315258466"
Oct 30 12:45:26.186046 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:26.185939    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xmn5t"
Oct 30 12:45:26.234553 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:26.234456    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2w7nn" event={"ID":"b3b4295c-1ef2-400d-a7e6-f2cc60f39cb6","Type":"ContainerStarted","Data":"294bf8bac912b7e272e4e76f90ee29f7e134f0c5efdf30cfe377b8efd238a0e8"}
Oct 30 12:45:26.970076 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:26.970028    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:27.079568 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:27.079533    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:27.242383 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:27.242298    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2w7nn" event={"ID":"b3b4295c-1ef2-400d-a7e6-f2cc60f39cb6","Type":"ContainerStarted","Data":"ce4abb3f195264b42ee1af7aebac0b24931a58960d95ea023ef867a0b99674cf"}
Oct 30 12:45:27.242383 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:27.242367    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2w7nn" event={"ID":"b3b4295c-1ef2-400d-a7e6-f2cc60f39cb6","Type":"ContainerStarted","Data":"83c18dcf8860d1ae0258ca797918502bfe3d482cd80f45073a9125448eb9148f"}
Oct 30 12:45:27.284362 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:27.284302    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2w7nn" podStartSLOduration=2.014701925 podStartE2EDuration="3.284284764s" totalImagesPullingTime="1.269582839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:27.284002215 +0000 UTC m=+22.331136169" watchObservedRunningTime="2026-10-30 12:45:27.284284764 +0000 UTC m=+22.331418720"
Oct 30 12:45:27.360716 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:27.360679    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:27.679879 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:27.679848    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:28.956330 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:28.956289    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:29.013262 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:45:29.013210    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 12:45:29.145960 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.145914    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:29.244947 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.243422    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kg8q6"]
Oct 30 12:45:29.249910 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.249873    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kg8q6"
Oct 30 12:45:29.263940 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.263914    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:29.298322 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.298281    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e9723bc6-9b25-43ec-b210-7e231e8bc8af-node-exporter-tls\") pod \"node-exporter-kg8q6\" (UID: \"e9723bc6-9b25-43ec-b210-7e231e8bc8af\") " pod="openshift-monitoring/node-exporter-kg8q6"
Oct 30 12:45:29.298523 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.298342    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e9723bc6-9b25-43ec-b210-7e231e8bc8af-node-exporter-wtmp\") pod \"node-exporter-kg8q6\" (UID: \"e9723bc6-9b25-43ec-b210-7e231e8bc8af\") " pod="openshift-monitoring/node-exporter-kg8q6"
Oct 30 12:45:29.298523 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.298436    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e9723bc6-9b25-43ec-b210-7e231e8bc8af-node-exporter-accelerators-collector-config\") pod \"node-exporter-kg8q6\" (UID: \"e9723bc6-9b25-43ec-b210-7e231e8bc8af\") " pod="openshift-monitoring/node-exporter-kg8q6"
Oct 30 12:45:29.298523 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.298471    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq6kk\" (UniqueName: \"kubernetes.io/projected/e9723bc6-9b25-43ec-b210-7e231e8bc8af-kube-api-access-nq6kk\") pod \"node-exporter-kg8q6\" (UID: \"e9723bc6-9b25-43ec-b210-7e231e8bc8af\") " pod="openshift-monitoring/node-exporter-kg8q6"
Oct 30 12:45:29.298730 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.298571    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9723bc6-9b25-43ec-b210-7e231e8bc8af-metrics-client-ca\") pod \"node-exporter-kg8q6\" (UID: \"e9723bc6-9b25-43ec-b210-7e231e8bc8af\") " pod="openshift-monitoring/node-exporter-kg8q6"
Oct 30 12:45:29.298730 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.298596    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e9723bc6-9b25-43ec-b210-7e231e8bc8af-node-exporter-textfile\") pod \"node-exporter-kg8q6\" (UID: \"e9723bc6-9b25-43ec-b210-7e231e8bc8af\") " pod="openshift-monitoring/node-exporter-kg8q6"
Oct 30 12:45:29.298730 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.298646    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e9723bc6-9b25-43ec-b210-7e231e8bc8af-root\") pod \"node-exporter-kg8q6\" (UID: \"e9723bc6-9b25-43ec-b210-7e231e8bc8af\") " pod="openshift-monitoring/node-exporter-kg8q6"
Oct 30 12:45:29.298730 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.298702    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9723bc6-9b25-43ec-b210-7e231e8bc8af-sys\") pod \"node-exporter-kg8q6\" (UID: \"e9723bc6-9b25-43ec-b210-7e231e8bc8af\") " pod="openshift-monitoring/node-exporter-kg8q6"
Oct 30 12:45:29.298856 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.298783    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e9723bc6-9b25-43ec-b210-7e231e8bc8af-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kg8q6\" (UID: \"e9723bc6-9b25-43ec-b210-7e231e8bc8af\") " pod="openshift-monitoring/node-exporter-kg8q6"
Oct 30 12:45:29.310335 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.310300    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 12:45:29.310542 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.310521    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 12:45:29.310685 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.310656    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-njv68\""
Oct 30 12:45:29.326130 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.326094    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 12:45:29.373769 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.373716    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:29.399963 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.399920    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9723bc6-9b25-43ec-b210-7e231e8bc8af-sys\") pod \"node-exporter-kg8q6\" (UID: \"e9723bc6-9b25-43ec-b210-7e231e8bc8af\") " pod="openshift-monitoring/node-exporter-kg8q6"
Oct 30 12:45:29.400157 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.399980    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e9723bc6-9b25-43ec-b210-7e231e8bc8af-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kg8q6\" (UID: \"e9723bc6-9b25-43ec-b210-7e231e8bc8af\") " pod="openshift-monitoring/node-exporter-kg8q6"
Oct 30 12:45:29.400157 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.399991    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9723bc6-9b25-43ec-b210-7e231e8bc8af-sys\") pod \"node-exporter-kg8q6\" (UID: \"e9723bc6-9b25-43ec-b210-7e231e8bc8af\") " pod="openshift-monitoring/node-exporter-kg8q6"
Oct 30 12:45:29.400157 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.400018    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e9723bc6-9b25-43ec-b210-7e231e8bc8af-node-exporter-tls\") pod \"node-exporter-kg8q6\" (UID: \"e9723bc6-9b25-43ec-b210-7e231e8bc8af\") " pod="openshift-monitoring/node-exporter-kg8q6"
Oct 30 12:45:29.400157 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.400064    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e9723bc6-9b25-43ec-b210-7e231e8bc8af-node-exporter-wtmp\") pod \"node-exporter-kg8q6\" (UID: \"e9723bc6-9b25-43ec-b210-7e231e8bc8af\") " pod="openshift-monitoring/node-exporter-kg8q6"
Oct 30 12:45:29.400157 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.400105    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e9723bc6-9b25-43ec-b210-7e231e8bc8af-node-exporter-accelerators-collector-config\") pod \"node-exporter-kg8q6\" (UID: \"e9723bc6-9b25-43ec-b210-7e231e8bc8af\") " pod="openshift-monitoring/node-exporter-kg8q6"
Oct 30 12:45:29.400157 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.400134    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq6kk\" (UniqueName: \"kubernetes.io/projected/e9723bc6-9b25-43ec-b210-7e231e8bc8af-kube-api-access-nq6kk\") pod \"node-exporter-kg8q6\" (UID: \"e9723bc6-9b25-43ec-b210-7e231e8bc8af\") " pod="openshift-monitoring/node-exporter-kg8q6"
Oct 30 12:45:29.400497 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.400168    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9723bc6-9b25-43ec-b210-7e231e8bc8af-metrics-client-ca\") pod \"node-exporter-kg8q6\" (UID: \"e9723bc6-9b25-43ec-b210-7e231e8bc8af\") " pod="openshift-monitoring/node-exporter-kg8q6"
Oct 30 12:45:29.400497 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.400193    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e9723bc6-9b25-43ec-b210-7e231e8bc8af-node-exporter-textfile\") pod \"node-exporter-kg8q6\" (UID: \"e9723bc6-9b25-43ec-b210-7e231e8bc8af\") " pod="openshift-monitoring/node-exporter-kg8q6"
Oct 30 12:45:29.400497 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.400249    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e9723bc6-9b25-43ec-b210-7e231e8bc8af-root\") pod \"node-exporter-kg8q6\" (UID: \"e9723bc6-9b25-43ec-b210-7e231e8bc8af\") " pod="openshift-monitoring/node-exporter-kg8q6"
Oct 30 12:45:29.400497 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.400294    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e9723bc6-9b25-43ec-b210-7e231e8bc8af-node-exporter-wtmp\") pod \"node-exporter-kg8q6\" (UID: \"e9723bc6-9b25-43ec-b210-7e231e8bc8af\") " pod="openshift-monitoring/node-exporter-kg8q6"
Oct 30 12:45:29.400497 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.400319    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e9723bc6-9b25-43ec-b210-7e231e8bc8af-root\") pod \"node-exporter-kg8q6\" (UID: \"e9723bc6-9b25-43ec-b210-7e231e8bc8af\") " pod="openshift-monitoring/node-exporter-kg8q6"
Oct 30 12:45:29.400747 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.400653    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e9723bc6-9b25-43ec-b210-7e231e8bc8af-node-exporter-textfile\") pod \"node-exporter-kg8q6\" (UID: \"e9723bc6-9b25-43ec-b210-7e231e8bc8af\") " pod="openshift-monitoring/node-exporter-kg8q6"
Oct 30 12:45:29.400849 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.400827    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e9723bc6-9b25-43ec-b210-7e231e8bc8af-node-exporter-accelerators-collector-config\") pod \"node-exporter-kg8q6\" (UID: \"e9723bc6-9b25-43ec-b210-7e231e8bc8af\") " pod="openshift-monitoring/node-exporter-kg8q6"
Oct 30 12:45:29.400998 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.400980    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9723bc6-9b25-43ec-b210-7e231e8bc8af-metrics-client-ca\") pod \"node-exporter-kg8q6\" (UID: \"e9723bc6-9b25-43ec-b210-7e231e8bc8af\") " pod="openshift-monitoring/node-exporter-kg8q6"
Oct 30 12:45:29.403071 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.403046    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e9723bc6-9b25-43ec-b210-7e231e8bc8af-node-exporter-tls\") pod \"node-exporter-kg8q6\" (UID: \"e9723bc6-9b25-43ec-b210-7e231e8bc8af\") " pod="openshift-monitoring/node-exporter-kg8q6"
Oct 30 12:45:29.405630 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.405604    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e9723bc6-9b25-43ec-b210-7e231e8bc8af-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kg8q6\" (UID: \"e9723bc6-9b25-43ec-b210-7e231e8bc8af\") " pod="openshift-monitoring/node-exporter-kg8q6"
Oct 30 12:45:29.423500 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.423462    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq6kk\" (UniqueName: \"kubernetes.io/projected/e9723bc6-9b25-43ec-b210-7e231e8bc8af-kube-api-access-nq6kk\") pod \"node-exporter-kg8q6\" (UID: \"e9723bc6-9b25-43ec-b210-7e231e8bc8af\") " pod="openshift-monitoring/node-exporter-kg8q6"
Oct 30 12:45:29.489754 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.489702    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:29.573594 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.573558    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kg8q6"
Oct 30 12:45:29.601160 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.601121    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:29.616813 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:29.616777    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9723bc6_9b25_43ec_b210_7e231e8bc8af.slice/crio-f0bf1e9cf8b508d182cf0934ae2bbd56a8888ee59f1ada52c9af0d9c540a0e4c WatchSource:0}: Error finding container f0bf1e9cf8b508d182cf0934ae2bbd56a8888ee59f1ada52c9af0d9c540a0e4c: Status 404 returned error can't find the container with id f0bf1e9cf8b508d182cf0934ae2bbd56a8888ee59f1ada52c9af0d9c540a0e4c
Oct 30 12:45:29.721503 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:29.721466    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:30.257890 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:30.257841    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kg8q6" event={"ID":"e9723bc6-9b25-43ec-b210-7e231e8bc8af","Type":"ContainerStarted","Data":"f0bf1e9cf8b508d182cf0934ae2bbd56a8888ee59f1ada52c9af0d9c540a0e4c"}
Oct 30 12:45:30.451798 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:30.451764    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:30.607104 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:30.607060    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:30.778804 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:30.778772    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:30.806888 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:30.806843    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bf67cd67d-sm5rz"
Oct 30 12:45:30.807069 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:30.806942    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bf67cd67d-sm5rz"
Oct 30 12:45:30.812558 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:30.812525    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-bf67cd67d-sm5rz"
Oct 30 12:45:31.226187 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:31.226104    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:31.263744 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:31.263701    4544 generic.go:356] "Generic (PLEG): container finished" podID="e9723bc6-9b25-43ec-b210-7e231e8bc8af" containerID="1f20eb3cd62a9f0de14832a6da519cb96b2edc94eaf84407582834c1f565cb08" exitCode=0
Oct 30 12:45:31.264247 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:31.263790    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kg8q6" event={"ID":"e9723bc6-9b25-43ec-b210-7e231e8bc8af","Type":"ContainerDied","Data":"1f20eb3cd62a9f0de14832a6da519cb96b2edc94eaf84407582834c1f565cb08"}
Oct 30 12:45:31.269694 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:31.269666    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-bf67cd67d-sm5rz"
Oct 30 12:45:31.351831 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:31.351797    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:31.864989 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:31.864944    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:31.974244 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:31.974182    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:32.270361 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:32.270252    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kg8q6" event={"ID":"e9723bc6-9b25-43ec-b210-7e231e8bc8af","Type":"ContainerStarted","Data":"2b6cca148fa7324d0404f096e413d4957103b8f91cdc532a8f18ae3d05f01a1b"}
Oct 30 12:45:32.270844 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:32.270370    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kg8q6" event={"ID":"e9723bc6-9b25-43ec-b210-7e231e8bc8af","Type":"ContainerStarted","Data":"5a96b0c3aad3596a3ba360f78946fdaed681385d7873c66733a8d7d467dda8fe"}
Oct 30 12:45:32.341633 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:32.341578    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kg8q6" podStartSLOduration=1.908245049 podStartE2EDuration="3.34156127s" totalImagesPullingTime="1.351778811s" totalInitContainerRuntime="81.53741ms" isStatefulPod=true podCreationTimestamp="2026-10-30 12:45:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:32.328675144 +0000 UTC m=+27.375809110" watchObservedRunningTime="2026-10-30 12:45:32.34156127 +0000 UTC m=+27.388695267"
Oct 30 12:45:32.431672 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:32.431628    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:32.548779 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:32.548691    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:32.669737 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:32.669703    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:32.895129 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:32.895095    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:33.326554 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:33.326525    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:33.438905 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:33.438865    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:34.034397 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:34.034358    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:34.149365 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:34.149336    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:34.273415 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:34.273377    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:36.135388 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:36.135349    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9sprl"
Oct 30 12:45:36.227290 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:36.227253    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:36.337252 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:36.337192    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:36.448237 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:36.448100    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:37.138483 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:37.138452    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jt6tp"
Oct 30 12:45:38.296926 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:38.296841    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-km7rm" event={"ID":"28d577e6-7b12-40a1-98b4-4bc41023ec17","Type":"ContainerStarted","Data":"4cb48d6249d426d94865c3b9dd665a987b18b3ecb43977e36c5a3032c6e2aa6a"}
Oct 30 12:45:38.297402 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:38.297104    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-km7rm"
Oct 30 12:45:38.298812 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:38.298784    4544 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-km7rm container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.13:8080/\": dial tcp 10.133.0.13:8080: connect: connection refused" start-of-body=
Oct 30 12:45:38.298924 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:38.298842    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-km7rm" podUID="28d577e6-7b12-40a1-98b4-4bc41023ec17" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.13:8080/\": dial tcp 10.133.0.13:8080: connect: connection refused"
Oct 30 12:45:38.354479 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:38.354418    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-km7rm" podStartSLOduration=1.549155904 podStartE2EDuration="18.354399913s" totalImagesPullingTime="16.805244009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:38.349423935 +0000 UTC m=+33.396557889" watchObservedRunningTime="2026-10-30 12:45:38.354399913 +0000 UTC m=+33.401533868"
Oct 30 12:45:38.623984 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:38.623941    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:38.730882 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:38.730784    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:39.316469 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:39.316437    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-km7rm"
Oct 30 12:45:39.319601 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:39.319568    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:39.438732 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:39.438694    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:39.892369 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:39.892333    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-76bcb6b7cd-hw2zh"]
Oct 30 12:45:39.916528 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:39.916487    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76bcb6b7cd-hw2zh"
Oct 30 12:45:39.919138 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:39.919110    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76bcb6b7cd-hw2zh"]
Oct 30 12:45:39.956244 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:39.956188    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 12:45:39.988460 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:39.988426    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-trusted-ca-bundle\") pod \"console-76bcb6b7cd-hw2zh\" (UID: \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\") " pod="openshift-console/console-76bcb6b7cd-hw2zh"
Oct 30 12:45:39.988460 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:39.988471    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-tmp\") pod \"console-76bcb6b7cd-hw2zh\" (UID: \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\") " pod="openshift-console/console-76bcb6b7cd-hw2zh"
Oct 30 12:45:39.988747 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:39.988623    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-console-oauth-config\") pod \"console-76bcb6b7cd-hw2zh\" (UID: \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\") " pod="openshift-console/console-76bcb6b7cd-hw2zh"
Oct 30 12:45:39.988747 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:39.988663    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-oauth-serving-cert\") pod \"console-76bcb6b7cd-hw2zh\" (UID: \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\") " pod="openshift-console/console-76bcb6b7cd-hw2zh"
Oct 30 12:45:39.988832 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:39.988738    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-service-ca\") pod \"console-76bcb6b7cd-hw2zh\" (UID: \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\") " pod="openshift-console/console-76bcb6b7cd-hw2zh"
Oct 30 12:45:39.988832 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:39.988785    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-console-serving-cert\") pod \"console-76bcb6b7cd-hw2zh\" (UID: \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\") " pod="openshift-console/console-76bcb6b7cd-hw2zh"
Oct 30 12:45:39.988832 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:39.988819    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-console-config\") pod \"console-76bcb6b7cd-hw2zh\" (UID: \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\") " pod="openshift-console/console-76bcb6b7cd-hw2zh"
Oct 30 12:45:39.988956 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:39.988845    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4h2v\" (UniqueName: \"kubernetes.io/projected/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-kube-api-access-c4h2v\") pod \"console-76bcb6b7cd-hw2zh\" (UID: \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\") " pod="openshift-console/console-76bcb6b7cd-hw2zh"
Oct 30 12:45:40.089276 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:40.089210    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-console-oauth-config\") pod \"console-76bcb6b7cd-hw2zh\" (UID: \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\") " pod="openshift-console/console-76bcb6b7cd-hw2zh"
Oct 30 12:45:40.089276 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:40.089282    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-oauth-serving-cert\") pod \"console-76bcb6b7cd-hw2zh\" (UID: \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\") " pod="openshift-console/console-76bcb6b7cd-hw2zh"
Oct 30 12:45:40.089519 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:40.089316    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-service-ca\") pod \"console-76bcb6b7cd-hw2zh\" (UID: \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\") " pod="openshift-console/console-76bcb6b7cd-hw2zh"
Oct 30 12:45:40.089519 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:40.089356    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-console-serving-cert\") pod \"console-76bcb6b7cd-hw2zh\" (UID: \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\") " pod="openshift-console/console-76bcb6b7cd-hw2zh"
Oct 30 12:45:40.089519 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:40.089395    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-console-config\") pod \"console-76bcb6b7cd-hw2zh\" (UID: \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\") " pod="openshift-console/console-76bcb6b7cd-hw2zh"
Oct 30 12:45:40.089519 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:40.089417    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4h2v\" (UniqueName: \"kubernetes.io/projected/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-kube-api-access-c4h2v\") pod \"console-76bcb6b7cd-hw2zh\" (UID: \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\") " pod="openshift-console/console-76bcb6b7cd-hw2zh"
Oct 30 12:45:40.089519 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:40.089440    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-trusted-ca-bundle\") pod \"console-76bcb6b7cd-hw2zh\" (UID: \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\") " pod="openshift-console/console-76bcb6b7cd-hw2zh"
Oct 30 12:45:40.089519 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:40.089474    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-tmp\") pod \"console-76bcb6b7cd-hw2zh\" (UID: \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\") " pod="openshift-console/console-76bcb6b7cd-hw2zh"
Oct 30 12:45:40.089965 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:40.089940    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-tmp\") pod \"console-76bcb6b7cd-hw2zh\" (UID: \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\") " pod="openshift-console/console-76bcb6b7cd-hw2zh"
Oct 30 12:45:40.090213 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:40.090185    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-oauth-serving-cert\") pod \"console-76bcb6b7cd-hw2zh\" (UID: \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\") " pod="openshift-console/console-76bcb6b7cd-hw2zh"
Oct 30 12:45:40.090528 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:40.090496    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-console-config\") pod \"console-76bcb6b7cd-hw2zh\" (UID: \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\") " pod="openshift-console/console-76bcb6b7cd-hw2zh"
Oct 30 12:45:40.091111 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:40.091087    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-service-ca\") pod \"console-76bcb6b7cd-hw2zh\" (UID: \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\") " pod="openshift-console/console-76bcb6b7cd-hw2zh"
Oct 30 12:45:40.091246 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:40.091180    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-trusted-ca-bundle\") pod \"console-76bcb6b7cd-hw2zh\" (UID: \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\") " pod="openshift-console/console-76bcb6b7cd-hw2zh"
Oct 30 12:45:40.092504 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:40.092484    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-console-oauth-config\") pod \"console-76bcb6b7cd-hw2zh\" (UID: \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\") " pod="openshift-console/console-76bcb6b7cd-hw2zh"
Oct 30 12:45:40.093464 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:40.093438    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-console-serving-cert\") pod \"console-76bcb6b7cd-hw2zh\" (UID: \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\") " pod="openshift-console/console-76bcb6b7cd-hw2zh"
Oct 30 12:45:40.114182 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:40.114141    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4h2v\" (UniqueName: \"kubernetes.io/projected/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-kube-api-access-c4h2v\") pod \"console-76bcb6b7cd-hw2zh\" (UID: \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\") " pod="openshift-console/console-76bcb6b7cd-hw2zh"
Oct 30 12:45:40.246428 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:40.246350    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76bcb6b7cd-hw2zh"
Oct 30 12:45:40.579239 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:40.579191    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76bcb6b7cd-hw2zh"]
Oct 30 12:45:40.586215 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:45:40.586129    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0283ec2a_eab1_483b_bf7f_4830ff7c4e28.slice/crio-a2ed1e0360ff141aa6aa994040e91c20479ad1890143e47386adffc14b29a04c WatchSource:0}: Error finding container a2ed1e0360ff141aa6aa994040e91c20479ad1890143e47386adffc14b29a04c: Status 404 returned error can't find the container with id a2ed1e0360ff141aa6aa994040e91c20479ad1890143e47386adffc14b29a04c
Oct 30 12:45:40.878680 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:40.878596    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:41.084306 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:41.084269    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:41.310910 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:41.310813    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76bcb6b7cd-hw2zh" event={"ID":"0283ec2a-eab1-483b-bf7f-4830ff7c4e28","Type":"ContainerStarted","Data":"f98db6976acdfc68a94be76f68abb05993fd1ab81e0732ee5a4520bd73a2e213"}
Oct 30 12:45:41.310910 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:41.310885    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76bcb6b7cd-hw2zh" event={"ID":"0283ec2a-eab1-483b-bf7f-4830ff7c4e28","Type":"ContainerStarted","Data":"a2ed1e0360ff141aa6aa994040e91c20479ad1890143e47386adffc14b29a04c"}
Oct 30 12:45:41.355272 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:41.355195    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-76bcb6b7cd-hw2zh" podStartSLOduration=2.355175708 podStartE2EDuration="2.355175708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:41.353873505 +0000 UTC m=+36.401007459" watchObservedRunningTime="2026-10-30 12:45:41.355175708 +0000 UTC m=+36.402309661"
Oct 30 12:45:43.219141 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:43.219114    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5cc44fbdc6-z6lmh"
Oct 30 12:45:43.584920 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:43.584884    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:43.831134 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:43.831084    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:44.683232 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:44.683196    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:44.956868 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:44.956782    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:47.815152 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:47.815072    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:47.959147 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:47.959112    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:50.199347 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:50.199308    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:50.247359 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:50.247313    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76bcb6b7cd-hw2zh"
Oct 30 12:45:50.247563 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:50.247441    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76bcb6b7cd-hw2zh"
Oct 30 12:45:50.254351 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:50.254320    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76bcb6b7cd-hw2zh"
Oct 30 12:45:50.349045 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:50.349019    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76bcb6b7cd-hw2zh"
Oct 30 12:45:50.356406 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:50.356375    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:50.430941 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:50.430908    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bf67cd67d-sm5rz"]
Oct 30 12:45:51.018288 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:51.018256    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:51.164794 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:51.164757    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:55.906501 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:55.906469    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:56.020492 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:56.020457    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:56.979237 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:56.979190    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:57.091011 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:45:57.090981    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:59.005537 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:45:59.005505    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 12:46:06.135996 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:06.135955    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="kube-system/konnectivity-agent-9sprl" podUID="abe27783-da2a-46b9-81f6-cf551787b1bc" containerName="konnectivity-agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:46:06.136499 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:06.136060    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9sprl"
Oct 30 12:46:06.136664 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:06.136636    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="kube-system/konnectivity-agent-9sprl" podUID="abe27783-da2a-46b9-81f6-cf551787b1bc" containerName="konnectivity-agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:46:06.258727 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:06.258691    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:07.245091 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:07.245053    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:07.368401 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:07.368369    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:07.476682 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:07.476648    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:07.875785 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:07.875749    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:12.010458 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:12.010426    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:12.126413 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:12.126373    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:12.388388 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:12.388356    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-sb2kl-jnhml"]
Oct 30 12:46:12.388597 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:12.388576    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-sb2kl-jnhml" podUID="184ffc49-7c79-41c2-b5b8-66e219e4f23e" containerName="insights-gathering" containerID="cri-o://4d78dbd1484fd3c59b6a1bd4ea09b644028395f9fc8a6872c3a61f763abe0f5d" gracePeriod=30
Oct 30 12:46:12.556407 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:12.556376    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-5djl4-d9vnz"]
Oct 30 12:46:12.574619 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:12.574589    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5djl4-d9vnz"
Oct 30 12:46:12.584655 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:12.584626    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-5djl4-d9vnz"]
Oct 30 12:46:12.664706 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:12.664611    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74bb378c-e8ce-4261-9ce4-4eea638acd5d-trusted-ca-bundle\") pod \"periodic-gathering-5djl4-d9vnz\" (UID: \"74bb378c-e8ce-4261-9ce4-4eea638acd5d\") " pod="openshift-insights/periodic-gathering-5djl4-d9vnz"
Oct 30 12:46:12.664706 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:12.664663    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74bb378c-e8ce-4261-9ce4-4eea638acd5d-service-ca-bundle\") pod \"periodic-gathering-5djl4-d9vnz\" (UID: \"74bb378c-e8ce-4261-9ce4-4eea638acd5d\") " pod="openshift-insights/periodic-gathering-5djl4-d9vnz"
Oct 30 12:46:12.664943 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:12.664763    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/74bb378c-e8ce-4261-9ce4-4eea638acd5d-archives-path\") pod \"periodic-gathering-5djl4-d9vnz\" (UID: \"74bb378c-e8ce-4261-9ce4-4eea638acd5d\") " pod="openshift-insights/periodic-gathering-5djl4-d9vnz"
Oct 30 12:46:12.664943 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:12.664832    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6lb8\" (UniqueName: \"kubernetes.io/projected/74bb378c-e8ce-4261-9ce4-4eea638acd5d-kube-api-access-v6lb8\") pod \"periodic-gathering-5djl4-d9vnz\" (UID: \"74bb378c-e8ce-4261-9ce4-4eea638acd5d\") " pod="openshift-insights/periodic-gathering-5djl4-d9vnz"
Oct 30 12:46:12.703361 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:12.703332    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-sb2kl-jnhml"
Oct 30 12:46:12.765971 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:12.765933    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/74bb378c-e8ce-4261-9ce4-4eea638acd5d-archives-path\") pod \"periodic-gathering-5djl4-d9vnz\" (UID: \"74bb378c-e8ce-4261-9ce4-4eea638acd5d\") " pod="openshift-insights/periodic-gathering-5djl4-d9vnz"
Oct 30 12:46:12.765971 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:12.765979    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6lb8\" (UniqueName: \"kubernetes.io/projected/74bb378c-e8ce-4261-9ce4-4eea638acd5d-kube-api-access-v6lb8\") pod \"periodic-gathering-5djl4-d9vnz\" (UID: \"74bb378c-e8ce-4261-9ce4-4eea638acd5d\") " pod="openshift-insights/periodic-gathering-5djl4-d9vnz"
Oct 30 12:46:12.766250 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:12.766014    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74bb378c-e8ce-4261-9ce4-4eea638acd5d-trusted-ca-bundle\") pod \"periodic-gathering-5djl4-d9vnz\" (UID: \"74bb378c-e8ce-4261-9ce4-4eea638acd5d\") " pod="openshift-insights/periodic-gathering-5djl4-d9vnz"
Oct 30 12:46:12.766250 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:12.766049    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74bb378c-e8ce-4261-9ce4-4eea638acd5d-service-ca-bundle\") pod \"periodic-gathering-5djl4-d9vnz\" (UID: \"74bb378c-e8ce-4261-9ce4-4eea638acd5d\") " pod="openshift-insights/periodic-gathering-5djl4-d9vnz"
Oct 30 12:46:12.766442 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:12.766421    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/74bb378c-e8ce-4261-9ce4-4eea638acd5d-archives-path\") pod \"periodic-gathering-5djl4-d9vnz\" (UID: \"74bb378c-e8ce-4261-9ce4-4eea638acd5d\") " pod="openshift-insights/periodic-gathering-5djl4-d9vnz"
Oct 30 12:46:12.766732 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:12.766714    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74bb378c-e8ce-4261-9ce4-4eea638acd5d-service-ca-bundle\") pod \"periodic-gathering-5djl4-d9vnz\" (UID: \"74bb378c-e8ce-4261-9ce4-4eea638acd5d\") " pod="openshift-insights/periodic-gathering-5djl4-d9vnz"
Oct 30 12:46:12.767006 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:12.766986    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74bb378c-e8ce-4261-9ce4-4eea638acd5d-trusted-ca-bundle\") pod \"periodic-gathering-5djl4-d9vnz\" (UID: \"74bb378c-e8ce-4261-9ce4-4eea638acd5d\") " pod="openshift-insights/periodic-gathering-5djl4-d9vnz"
Oct 30 12:46:12.789935 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:12.789905    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6lb8\" (UniqueName: \"kubernetes.io/projected/74bb378c-e8ce-4261-9ce4-4eea638acd5d-kube-api-access-v6lb8\") pod \"periodic-gathering-5djl4-d9vnz\" (UID: \"74bb378c-e8ce-4261-9ce4-4eea638acd5d\") " pod="openshift-insights/periodic-gathering-5djl4-d9vnz"
Oct 30 12:46:12.866691 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:12.866655    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/184ffc49-7c79-41c2-b5b8-66e219e4f23e-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/184ffc49-7c79-41c2-b5b8-66e219e4f23e-service-ca-bundle\") pod \"184ffc49-7c79-41c2-b5b8-66e219e4f23e\" (UID: \"184ffc49-7c79-41c2-b5b8-66e219e4f23e\") "
Oct 30 12:46:12.866869 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:12.866732    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/184ffc49-7c79-41c2-b5b8-66e219e4f23e-kube-api-access-98j7q\" (UniqueName: \"kubernetes.io/projected/184ffc49-7c79-41c2-b5b8-66e219e4f23e-kube-api-access-98j7q\") pod \"184ffc49-7c79-41c2-b5b8-66e219e4f23e\" (UID: \"184ffc49-7c79-41c2-b5b8-66e219e4f23e\") "
Oct 30 12:46:12.866923 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:12.866865    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/184ffc49-7c79-41c2-b5b8-66e219e4f23e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/184ffc49-7c79-41c2-b5b8-66e219e4f23e-trusted-ca-bundle\") pod \"184ffc49-7c79-41c2-b5b8-66e219e4f23e\" (UID: \"184ffc49-7c79-41c2-b5b8-66e219e4f23e\") "
Oct 30 12:46:12.866923 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:12.866904    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/184ffc49-7c79-41c2-b5b8-66e219e4f23e-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/184ffc49-7c79-41c2-b5b8-66e219e4f23e-archives-path\") pod \"184ffc49-7c79-41c2-b5b8-66e219e4f23e\" (UID: \"184ffc49-7c79-41c2-b5b8-66e219e4f23e\") "
Oct 30 12:46:12.867142 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:12.867108    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/184ffc49-7c79-41c2-b5b8-66e219e4f23e-service-ca-bundle" pod "184ffc49-7c79-41c2-b5b8-66e219e4f23e" (UID: "184ffc49-7c79-41c2-b5b8-66e219e4f23e"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:12.867280 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:12.867255    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/184ffc49-7c79-41c2-b5b8-66e219e4f23e-trusted-ca-bundle" pod "184ffc49-7c79-41c2-b5b8-66e219e4f23e" (UID: "184ffc49-7c79-41c2-b5b8-66e219e4f23e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:12.867280 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:12.867265    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/184ffc49-7c79-41c2-b5b8-66e219e4f23e-archives-path" pod "184ffc49-7c79-41c2-b5b8-66e219e4f23e" (UID: "184ffc49-7c79-41c2-b5b8-66e219e4f23e"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:12.869087 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:12.869061    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/184ffc49-7c79-41c2-b5b8-66e219e4f23e-kube-api-access-98j7q" pod "184ffc49-7c79-41c2-b5b8-66e219e4f23e" (UID: "184ffc49-7c79-41c2-b5b8-66e219e4f23e"). InnerVolumeSpecName "kube-api-access-98j7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:12.899134 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:12.899101    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5djl4-d9vnz"
Oct 30 12:46:12.968196 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:12.968157    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98j7q\" (UniqueName: \"kubernetes.io/projected/184ffc49-7c79-41c2-b5b8-66e219e4f23e-kube-api-access-98j7q\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:46:12.968196 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:12.968199    4544 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/184ffc49-7c79-41c2-b5b8-66e219e4f23e-trusted-ca-bundle\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:46:12.968466 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:12.968215    4544 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/184ffc49-7c79-41c2-b5b8-66e219e4f23e-archives-path\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:46:12.968466 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:12.968250    4544 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/184ffc49-7c79-41c2-b5b8-66e219e4f23e-service-ca-bundle\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:46:13.230796 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:13.230687    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-5djl4-d9vnz"]
Oct 30 12:46:13.235216 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:46:13.235191    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74bb378c_e8ce_4261_9ce4_4eea638acd5d.slice/crio-e301d2780404e50db4af3ec28dd6771c03971d9afc3eb353201bc5e163b70c1d WatchSource:0}: Error finding container e301d2780404e50db4af3ec28dd6771c03971d9afc3eb353201bc5e163b70c1d: Status 404 returned error can't find the container with id e301d2780404e50db4af3ec28dd6771c03971d9afc3eb353201bc5e163b70c1d
Oct 30 12:46:13.426169 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:13.426136    4544 generic.go:356] "Generic (PLEG): container finished" podID="184ffc49-7c79-41c2-b5b8-66e219e4f23e" containerID="4d78dbd1484fd3c59b6a1bd4ea09b644028395f9fc8a6872c3a61f763abe0f5d" exitCode=2
Oct 30 12:46:13.426371 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:13.426179    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-sb2kl-jnhml"
Oct 30 12:46:13.426371 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:13.426232    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-sb2kl-jnhml" event={"ID":"184ffc49-7c79-41c2-b5b8-66e219e4f23e","Type":"ContainerDied","Data":"4d78dbd1484fd3c59b6a1bd4ea09b644028395f9fc8a6872c3a61f763abe0f5d"}
Oct 30 12:46:13.426371 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:13.426297    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-sb2kl-jnhml" event={"ID":"184ffc49-7c79-41c2-b5b8-66e219e4f23e","Type":"ContainerDied","Data":"41f57abb74c1a669c430f0dbb87718baa6489308d0cc793d9ccb1db942687041"}
Oct 30 12:46:13.426371 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:13.426315    4544 scope.go:122] "RemoveContainer" containerID="4d78dbd1484fd3c59b6a1bd4ea09b644028395f9fc8a6872c3a61f763abe0f5d"
Oct 30 12:46:13.427894 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:13.427869    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5djl4-d9vnz" event={"ID":"74bb378c-e8ce-4261-9ce4-4eea638acd5d","Type":"ContainerStarted","Data":"49db8b5fe3d513e430b6e60ce696f3a4c3b9459135ac817a74749fd6bc39d3c3"}
Oct 30 12:46:13.428006 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:13.427915    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5djl4-d9vnz" event={"ID":"74bb378c-e8ce-4261-9ce4-4eea638acd5d","Type":"ContainerStarted","Data":"e301d2780404e50db4af3ec28dd6771c03971d9afc3eb353201bc5e163b70c1d"}
Oct 30 12:46:13.433990 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:13.433971    4544 scope.go:122] "RemoveContainer" containerID="4d78dbd1484fd3c59b6a1bd4ea09b644028395f9fc8a6872c3a61f763abe0f5d"
Oct 30 12:46:13.434290 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:46:13.434259    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d78dbd1484fd3c59b6a1bd4ea09b644028395f9fc8a6872c3a61f763abe0f5d\": container with ID starting with 4d78dbd1484fd3c59b6a1bd4ea09b644028395f9fc8a6872c3a61f763abe0f5d not found: ID does not exist" containerID="4d78dbd1484fd3c59b6a1bd4ea09b644028395f9fc8a6872c3a61f763abe0f5d"
Oct 30 12:46:13.434377 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:13.434294    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d78dbd1484fd3c59b6a1bd4ea09b644028395f9fc8a6872c3a61f763abe0f5d"} err="failed to get container status \"4d78dbd1484fd3c59b6a1bd4ea09b644028395f9fc8a6872c3a61f763abe0f5d\": rpc error: code = NotFound desc = could not find container \"4d78dbd1484fd3c59b6a1bd4ea09b644028395f9fc8a6872c3a61f763abe0f5d\": container with ID starting with 4d78dbd1484fd3c59b6a1bd4ea09b644028395f9fc8a6872c3a61f763abe0f5d not found: ID does not exist"
Oct 30 12:46:13.481684 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:13.481602    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-sb2kl-jnhml"]
Oct 30 12:46:13.490571 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:13.490536    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-sb2kl-jnhml"]
Oct 30 12:46:13.546651 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:13.546599    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-5djl4-d9vnz" podStartSLOduration=1.546585088 podStartE2EDuration="1.546585088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:13.545858045 +0000 UTC m=+68.592992000" watchObservedRunningTime="2026-10-30 12:46:13.546585088 +0000 UTC m=+68.593719041"
Oct 30 12:46:15.083476 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:15.083440    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="184ffc49-7c79-41c2-b5b8-66e219e4f23e" path="/var/lib/kubelet/pods/184ffc49-7c79-41c2-b5b8-66e219e4f23e/volumes"
Oct 30 12:46:15.452996 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:15.452869    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-bf67cd67d-sm5rz" podUID="05cbf75e-a21f-475f-afef-6e7c55cf0551" containerName="console" containerID="cri-o://1722daa2614ed284003bfb572360d8abe6c618a3ef969becba8962044e0e6d65" gracePeriod=15
Oct 30 12:46:15.732262 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:15.732205    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bf67cd67d-sm5rz"
Oct 30 12:46:15.891187 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:15.891137    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05cbf75e-a21f-475f-afef-6e7c55cf0551-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05cbf75e-a21f-475f-afef-6e7c55cf0551-oauth-serving-cert\") pod \"05cbf75e-a21f-475f-afef-6e7c55cf0551\" (UID: \"05cbf75e-a21f-475f-afef-6e7c55cf0551\") "
Oct 30 12:46:15.891187 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:15.891192    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05cbf75e-a21f-475f-afef-6e7c55cf0551-kube-api-access-29wwj\" (UniqueName: \"kubernetes.io/projected/05cbf75e-a21f-475f-afef-6e7c55cf0551-kube-api-access-29wwj\") pod \"05cbf75e-a21f-475f-afef-6e7c55cf0551\" (UID: \"05cbf75e-a21f-475f-afef-6e7c55cf0551\") "
Oct 30 12:46:15.891448 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:15.891242    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05cbf75e-a21f-475f-afef-6e7c55cf0551-service-ca\" (UniqueName: \"kubernetes.io/configmap/05cbf75e-a21f-475f-afef-6e7c55cf0551-service-ca\") pod \"05cbf75e-a21f-475f-afef-6e7c55cf0551\" (UID: \"05cbf75e-a21f-475f-afef-6e7c55cf0551\") "
Oct 30 12:46:15.891448 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:15.891296    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05cbf75e-a21f-475f-afef-6e7c55cf0551-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05cbf75e-a21f-475f-afef-6e7c55cf0551-console-serving-cert\") pod \"05cbf75e-a21f-475f-afef-6e7c55cf0551\" (UID: \"05cbf75e-a21f-475f-afef-6e7c55cf0551\") "
Oct 30 12:46:15.891448 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:15.891352    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05cbf75e-a21f-475f-afef-6e7c55cf0551-tmp\" (UniqueName: \"kubernetes.io/empty-dir/05cbf75e-a21f-475f-afef-6e7c55cf0551-tmp\") pod \"05cbf75e-a21f-475f-afef-6e7c55cf0551\" (UID: \"05cbf75e-a21f-475f-afef-6e7c55cf0551\") "
Oct 30 12:46:15.891448 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:15.891397    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05cbf75e-a21f-475f-afef-6e7c55cf0551-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05cbf75e-a21f-475f-afef-6e7c55cf0551-console-oauth-config\") pod \"05cbf75e-a21f-475f-afef-6e7c55cf0551\" (UID: \"05cbf75e-a21f-475f-afef-6e7c55cf0551\") "
Oct 30 12:46:15.891662 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:15.891449    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05cbf75e-a21f-475f-afef-6e7c55cf0551-console-config\" (UniqueName: \"kubernetes.io/configmap/05cbf75e-a21f-475f-afef-6e7c55cf0551-console-config\") pod \"05cbf75e-a21f-475f-afef-6e7c55cf0551\" (UID: \"05cbf75e-a21f-475f-afef-6e7c55cf0551\") "
Oct 30 12:46:15.891724 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:15.891683    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05cbf75e-a21f-475f-afef-6e7c55cf0551-tmp" pod "05cbf75e-a21f-475f-afef-6e7c55cf0551" (UID: "05cbf75e-a21f-475f-afef-6e7c55cf0551"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:15.891778 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:15.891693    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05cbf75e-a21f-475f-afef-6e7c55cf0551-service-ca" pod "05cbf75e-a21f-475f-afef-6e7c55cf0551" (UID: "05cbf75e-a21f-475f-afef-6e7c55cf0551"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:15.891778 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:15.891688    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05cbf75e-a21f-475f-afef-6e7c55cf0551-oauth-serving-cert" pod "05cbf75e-a21f-475f-afef-6e7c55cf0551" (UID: "05cbf75e-a21f-475f-afef-6e7c55cf0551"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:15.891971 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:15.891948    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05cbf75e-a21f-475f-afef-6e7c55cf0551-console-config" pod "05cbf75e-a21f-475f-afef-6e7c55cf0551" (UID: "05cbf75e-a21f-475f-afef-6e7c55cf0551"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:15.893766 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:15.893731    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05cbf75e-a21f-475f-afef-6e7c55cf0551-console-oauth-config" pod "05cbf75e-a21f-475f-afef-6e7c55cf0551" (UID: "05cbf75e-a21f-475f-afef-6e7c55cf0551"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:15.893766 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:15.893750    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05cbf75e-a21f-475f-afef-6e7c55cf0551-kube-api-access-29wwj" pod "05cbf75e-a21f-475f-afef-6e7c55cf0551" (UID: "05cbf75e-a21f-475f-afef-6e7c55cf0551"). InnerVolumeSpecName "kube-api-access-29wwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:15.893960 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:15.893800    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05cbf75e-a21f-475f-afef-6e7c55cf0551-console-serving-cert" pod "05cbf75e-a21f-475f-afef-6e7c55cf0551" (UID: "05cbf75e-a21f-475f-afef-6e7c55cf0551"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:15.992709 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:15.992623    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29wwj\" (UniqueName: \"kubernetes.io/projected/05cbf75e-a21f-475f-afef-6e7c55cf0551-kube-api-access-29wwj\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:46:15.992709 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:15.992657    4544 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05cbf75e-a21f-475f-afef-6e7c55cf0551-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:46:15.992709 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:15.992667    4544 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05cbf75e-a21f-475f-afef-6e7c55cf0551-console-serving-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:46:15.992709 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:15.992677    4544 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05cbf75e-a21f-475f-afef-6e7c55cf0551-tmp\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:46:15.992709 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:15.992686    4544 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05cbf75e-a21f-475f-afef-6e7c55cf0551-console-oauth-config\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:46:15.992709 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:15.992695    4544 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05cbf75e-a21f-475f-afef-6e7c55cf0551-console-config\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:46:15.992709 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:15.992703    4544 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05cbf75e-a21f-475f-afef-6e7c55cf0551-oauth-serving-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:46:16.440297 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:16.440261    4544 generic.go:356] "Generic (PLEG): container finished" podID="05cbf75e-a21f-475f-afef-6e7c55cf0551" containerID="1722daa2614ed284003bfb572360d8abe6c618a3ef969becba8962044e0e6d65" exitCode=0
Oct 30 12:46:16.440727 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:16.440320    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bf67cd67d-sm5rz" event={"ID":"05cbf75e-a21f-475f-afef-6e7c55cf0551","Type":"ContainerDied","Data":"1722daa2614ed284003bfb572360d8abe6c618a3ef969becba8962044e0e6d65"}
Oct 30 12:46:16.440727 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:16.440328    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bf67cd67d-sm5rz"
Oct 30 12:46:16.440727 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:16.440361    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bf67cd67d-sm5rz" event={"ID":"05cbf75e-a21f-475f-afef-6e7c55cf0551","Type":"ContainerDied","Data":"bc014583e7d4b114fcb4b61de3a744003feba3ae621ad8bc049f8f0a62f93d43"}
Oct 30 12:46:16.440727 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:16.440381    4544 scope.go:122] "RemoveContainer" containerID="1722daa2614ed284003bfb572360d8abe6c618a3ef969becba8962044e0e6d65"
Oct 30 12:46:16.448617 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:16.448580    4544 scope.go:122] "RemoveContainer" containerID="1722daa2614ed284003bfb572360d8abe6c618a3ef969becba8962044e0e6d65"
Oct 30 12:46:16.448846 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:46:16.448828    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1722daa2614ed284003bfb572360d8abe6c618a3ef969becba8962044e0e6d65\": container with ID starting with 1722daa2614ed284003bfb572360d8abe6c618a3ef969becba8962044e0e6d65 not found: ID does not exist" containerID="1722daa2614ed284003bfb572360d8abe6c618a3ef969becba8962044e0e6d65"
Oct 30 12:46:16.448896 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:16.448856    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1722daa2614ed284003bfb572360d8abe6c618a3ef969becba8962044e0e6d65"} err="failed to get container status \"1722daa2614ed284003bfb572360d8abe6c618a3ef969becba8962044e0e6d65\": rpc error: code = NotFound desc = could not find container \"1722daa2614ed284003bfb572360d8abe6c618a3ef969becba8962044e0e6d65\": container with ID starting with 1722daa2614ed284003bfb572360d8abe6c618a3ef969becba8962044e0e6d65 not found: ID does not exist"
Oct 30 12:46:16.490188 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:16.490166    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bf67cd67d-sm5rz"]
Oct 30 12:46:16.502232 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:16.502200    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-bf67cd67d-sm5rz"]
Oct 30 12:46:17.083947 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:17.083917    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05cbf75e-a21f-475f-afef-6e7c55cf0551" path="/var/lib/kubelet/pods/05cbf75e-a21f-475f-afef-6e7c55cf0551/volumes"
Oct 30 12:46:20.657071 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:20.657025    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:20.934969 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:20.934889    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:28.996882 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:28.996839    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5dfc766f68-gfrkb"]
Oct 30 12:46:28.997401 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:28.997240    4544 cpu_manager.go:414] "containerMap: removing container" podUID="184ffc49-7c79-41c2-b5b8-66e219e4f23e" containerName="insights-gathering"
Oct 30 12:46:28.997401 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:28.997260    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="184ffc49-7c79-41c2-b5b8-66e219e4f23e" containerName="insights-gathering"
Oct 30 12:46:28.997401 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:28.997291    4544 cpu_manager.go:414] "containerMap: removing container" podUID="05cbf75e-a21f-475f-afef-6e7c55cf0551" containerName="console"
Oct 30 12:46:28.997401 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:28.997299    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05cbf75e-a21f-475f-afef-6e7c55cf0551" containerName="console"
Oct 30 12:46:28.997401 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:28.997385    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="184ffc49-7c79-41c2-b5b8-66e219e4f23e" containerName="insights-gathering"
Oct 30 12:46:28.997401 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:28.997400    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="05cbf75e-a21f-475f-afef-6e7c55cf0551" containerName="console"
Oct 30 12:46:28.999983 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:28.999961    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dfc766f68-gfrkb"
Oct 30 12:46:29.009624 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:46:29.009580    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 12:46:29.036610 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:29.036576    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dfc766f68-gfrkb"]
Oct 30 12:46:29.089150 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:29.089114    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/22715ff5-9438-43ed-82e2-cffb9c93d107-oauth-serving-cert\") pod \"console-5dfc766f68-gfrkb\" (UID: \"22715ff5-9438-43ed-82e2-cffb9c93d107\") " pod="openshift-console/console-5dfc766f68-gfrkb"
Oct 30 12:46:29.089150 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:29.089155    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/22715ff5-9438-43ed-82e2-cffb9c93d107-console-oauth-config\") pod \"console-5dfc766f68-gfrkb\" (UID: \"22715ff5-9438-43ed-82e2-cffb9c93d107\") " pod="openshift-console/console-5dfc766f68-gfrkb"
Oct 30 12:46:29.089417 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:29.089186    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/22715ff5-9438-43ed-82e2-cffb9c93d107-service-ca\") pod \"console-5dfc766f68-gfrkb\" (UID: \"22715ff5-9438-43ed-82e2-cffb9c93d107\") " pod="openshift-console/console-5dfc766f68-gfrkb"
Oct 30 12:46:29.089417 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:29.089285    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22715ff5-9438-43ed-82e2-cffb9c93d107-tmp\") pod \"console-5dfc766f68-gfrkb\" (UID: \"22715ff5-9438-43ed-82e2-cffb9c93d107\") " pod="openshift-console/console-5dfc766f68-gfrkb"
Oct 30 12:46:29.089417 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:29.089404    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfmrs\" (UniqueName: \"kubernetes.io/projected/22715ff5-9438-43ed-82e2-cffb9c93d107-kube-api-access-sfmrs\") pod \"console-5dfc766f68-gfrkb\" (UID: \"22715ff5-9438-43ed-82e2-cffb9c93d107\") " pod="openshift-console/console-5dfc766f68-gfrkb"
Oct 30 12:46:29.089589 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:29.089432    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/22715ff5-9438-43ed-82e2-cffb9c93d107-console-config\") pod \"console-5dfc766f68-gfrkb\" (UID: \"22715ff5-9438-43ed-82e2-cffb9c93d107\") " pod="openshift-console/console-5dfc766f68-gfrkb"
Oct 30 12:46:29.089589 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:29.089464    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22715ff5-9438-43ed-82e2-cffb9c93d107-trusted-ca-bundle\") pod \"console-5dfc766f68-gfrkb\" (UID: \"22715ff5-9438-43ed-82e2-cffb9c93d107\") " pod="openshift-console/console-5dfc766f68-gfrkb"
Oct 30 12:46:29.089589 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:29.089532    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/22715ff5-9438-43ed-82e2-cffb9c93d107-console-serving-cert\") pod \"console-5dfc766f68-gfrkb\" (UID: \"22715ff5-9438-43ed-82e2-cffb9c93d107\") " pod="openshift-console/console-5dfc766f68-gfrkb"
Oct 30 12:46:29.161206 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:29.161162    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:29.190479 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:29.190440    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/22715ff5-9438-43ed-82e2-cffb9c93d107-oauth-serving-cert\") pod \"console-5dfc766f68-gfrkb\" (UID: \"22715ff5-9438-43ed-82e2-cffb9c93d107\") " pod="openshift-console/console-5dfc766f68-gfrkb"
Oct 30 12:46:29.190479 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:29.190479    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/22715ff5-9438-43ed-82e2-cffb9c93d107-console-oauth-config\") pod \"console-5dfc766f68-gfrkb\" (UID: \"22715ff5-9438-43ed-82e2-cffb9c93d107\") " pod="openshift-console/console-5dfc766f68-gfrkb"
Oct 30 12:46:29.190727 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:29.190519    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/22715ff5-9438-43ed-82e2-cffb9c93d107-service-ca\") pod \"console-5dfc766f68-gfrkb\" (UID: \"22715ff5-9438-43ed-82e2-cffb9c93d107\") " pod="openshift-console/console-5dfc766f68-gfrkb"
Oct 30 12:46:29.190727 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:29.190536    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22715ff5-9438-43ed-82e2-cffb9c93d107-tmp\") pod \"console-5dfc766f68-gfrkb\" (UID: \"22715ff5-9438-43ed-82e2-cffb9c93d107\") " pod="openshift-console/console-5dfc766f68-gfrkb"
Oct 30 12:46:29.190727 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:29.190694    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfmrs\" (UniqueName: \"kubernetes.io/projected/22715ff5-9438-43ed-82e2-cffb9c93d107-kube-api-access-sfmrs\") pod \"console-5dfc766f68-gfrkb\" (UID: \"22715ff5-9438-43ed-82e2-cffb9c93d107\") " pod="openshift-console/console-5dfc766f68-gfrkb"
Oct 30 12:46:29.190882 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:29.190784    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/22715ff5-9438-43ed-82e2-cffb9c93d107-console-config\") pod \"console-5dfc766f68-gfrkb\" (UID: \"22715ff5-9438-43ed-82e2-cffb9c93d107\") " pod="openshift-console/console-5dfc766f68-gfrkb"
Oct 30 12:46:29.190935 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:29.190912    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22715ff5-9438-43ed-82e2-cffb9c93d107-trusted-ca-bundle\") pod \"console-5dfc766f68-gfrkb\" (UID: \"22715ff5-9438-43ed-82e2-cffb9c93d107\") " pod="openshift-console/console-5dfc766f68-gfrkb"
Oct 30 12:46:29.190985 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:29.190956    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/22715ff5-9438-43ed-82e2-cffb9c93d107-console-serving-cert\") pod \"console-5dfc766f68-gfrkb\" (UID: \"22715ff5-9438-43ed-82e2-cffb9c93d107\") " pod="openshift-console/console-5dfc766f68-gfrkb"
Oct 30 12:46:29.191577 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:29.191544    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22715ff5-9438-43ed-82e2-cffb9c93d107-tmp\") pod \"console-5dfc766f68-gfrkb\" (UID: \"22715ff5-9438-43ed-82e2-cffb9c93d107\") " pod="openshift-console/console-5dfc766f68-gfrkb"
Oct 30 12:46:29.191701 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:29.191621    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/22715ff5-9438-43ed-82e2-cffb9c93d107-service-ca\") pod \"console-5dfc766f68-gfrkb\" (UID: \"22715ff5-9438-43ed-82e2-cffb9c93d107\") " pod="openshift-console/console-5dfc766f68-gfrkb"
Oct 30 12:46:29.192005 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:29.191912    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/22715ff5-9438-43ed-82e2-cffb9c93d107-oauth-serving-cert\") pod \"console-5dfc766f68-gfrkb\" (UID: \"22715ff5-9438-43ed-82e2-cffb9c93d107\") " pod="openshift-console/console-5dfc766f68-gfrkb"
Oct 30 12:46:29.192005 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:29.191934    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/22715ff5-9438-43ed-82e2-cffb9c93d107-console-config\") pod \"console-5dfc766f68-gfrkb\" (UID: \"22715ff5-9438-43ed-82e2-cffb9c93d107\") " pod="openshift-console/console-5dfc766f68-gfrkb"
Oct 30 12:46:29.196236 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:29.193040    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22715ff5-9438-43ed-82e2-cffb9c93d107-trusted-ca-bundle\") pod \"console-5dfc766f68-gfrkb\" (UID: \"22715ff5-9438-43ed-82e2-cffb9c93d107\") " pod="openshift-console/console-5dfc766f68-gfrkb"
Oct 30 12:46:29.197018 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:29.196995    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/22715ff5-9438-43ed-82e2-cffb9c93d107-console-serving-cert\") pod \"console-5dfc766f68-gfrkb\" (UID: \"22715ff5-9438-43ed-82e2-cffb9c93d107\") " pod="openshift-console/console-5dfc766f68-gfrkb"
Oct 30 12:46:29.197241 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:29.197192    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/22715ff5-9438-43ed-82e2-cffb9c93d107-console-oauth-config\") pod \"console-5dfc766f68-gfrkb\" (UID: \"22715ff5-9438-43ed-82e2-cffb9c93d107\") " pod="openshift-console/console-5dfc766f68-gfrkb"
Oct 30 12:46:29.219441 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:29.219408    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfmrs\" (UniqueName: \"kubernetes.io/projected/22715ff5-9438-43ed-82e2-cffb9c93d107-kube-api-access-sfmrs\") pod \"console-5dfc766f68-gfrkb\" (UID: \"22715ff5-9438-43ed-82e2-cffb9c93d107\") " pod="openshift-console/console-5dfc766f68-gfrkb"
Oct 30 12:46:29.280031 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:29.279936    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:29.313073 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:29.313038    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dfc766f68-gfrkb"
Oct 30 12:46:29.547986 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:29.546745    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:29.612240 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:29.612197    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dfc766f68-gfrkb"]
Oct 30 12:46:29.617103 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:46:29.617073    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22715ff5_9438_43ed_82e2_cffb9c93d107.slice/crio-f493c4f0f46d724f957e108b788ec9f0c5852cb7a1a453a959f673357975cbeb WatchSource:0}: Error finding container f493c4f0f46d724f957e108b788ec9f0c5852cb7a1a453a959f673357975cbeb: Status 404 returned error can't find the container with id f493c4f0f46d724f957e108b788ec9f0c5852cb7a1a453a959f673357975cbeb
Oct 30 12:46:29.670959 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:29.670930    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:30.492515 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:30.492476    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dfc766f68-gfrkb" event={"ID":"22715ff5-9438-43ed-82e2-cffb9c93d107","Type":"ContainerStarted","Data":"a25f1b7d5eda12181c3fb6548a9d362143f29b88e25407bd8712cd25259d0db4"}
Oct 30 12:46:30.492932 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:30.492533    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dfc766f68-gfrkb" event={"ID":"22715ff5-9438-43ed-82e2-cffb9c93d107","Type":"ContainerStarted","Data":"f493c4f0f46d724f957e108b788ec9f0c5852cb7a1a453a959f673357975cbeb"}
Oct 30 12:46:36.137984 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:36.137946    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9sprl"
Oct 30 12:46:36.179287 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:36.179202    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5dfc766f68-gfrkb" podStartSLOduration=8.179186263 podStartE2EDuration="8.179186263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:30.568668416 +0000 UTC m=+85.615802368" watchObservedRunningTime="2026-10-30 12:46:36.179186263 +0000 UTC m=+91.226320216"
Oct 30 12:46:36.275987 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:36.275942    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:37.926456 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:37.926424    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:39.314097 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:39.314044    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5dfc766f68-gfrkb"
Oct 30 12:46:39.314561 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:39.314117    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5dfc766f68-gfrkb"
Oct 30 12:46:39.318947 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:39.318922    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5dfc766f68-gfrkb"
Oct 30 12:46:39.532843 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:39.532815    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5dfc766f68-gfrkb"
Oct 30 12:46:39.628372 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:39.628334    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76bcb6b7cd-hw2zh"]
Oct 30 12:46:51.168617 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:51.168582    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:51.331489 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:46:51.331450    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:46:59.005002 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:46:59.004963    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 12:47:04.647901 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:04.647845    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-76bcb6b7cd-hw2zh" podUID="0283ec2a-eab1-483b-bf7f-4830ff7c4e28" containerName="console" containerID="cri-o://f98db6976acdfc68a94be76f68abb05993fd1ab81e0732ee5a4520bd73a2e213" gracePeriod=15
Oct 30 12:47:04.893763 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:04.893728    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76bcb6b7cd-hw2zh"
Oct 30 12:47:04.982047 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:04.981957    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-trusted-ca-bundle\") pod \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\" (UID: \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\") "
Oct 30 12:47:04.982047 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:04.982022    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-console-oauth-config\") pod \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\" (UID: \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\") "
Oct 30 12:47:04.982047 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:04.982049    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-service-ca\" (UniqueName: \"kubernetes.io/configmap/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-service-ca\") pod \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\" (UID: \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\") "
Oct 30 12:47:04.982339 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:04.982093    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-tmp\") pod \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\" (UID: \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\") "
Oct 30 12:47:04.982339 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:04.982139    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-console-serving-cert\") pod \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\" (UID: \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\") "
Oct 30 12:47:04.982339 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:04.982303    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-console-config\" (UniqueName: \"kubernetes.io/configmap/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-console-config\") pod \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\" (UID: \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\") "
Oct 30 12:47:04.982503 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:04.982349    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-oauth-serving-cert\") pod \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\" (UID: \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\") "
Oct 30 12:47:04.982503 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:04.982380    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-kube-api-access-c4h2v\" (UniqueName: \"kubernetes.io/projected/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-kube-api-access-c4h2v\") pod \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\" (UID: \"0283ec2a-eab1-483b-bf7f-4830ff7c4e28\") "
Oct 30 12:47:04.982600 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:04.982481    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-tmp" pod "0283ec2a-eab1-483b-bf7f-4830ff7c4e28" (UID: "0283ec2a-eab1-483b-bf7f-4830ff7c4e28"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:47:04.982600 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:04.982538    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-trusted-ca-bundle" pod "0283ec2a-eab1-483b-bf7f-4830ff7c4e28" (UID: "0283ec2a-eab1-483b-bf7f-4830ff7c4e28"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:47:04.982685 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:04.982547    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-service-ca" pod "0283ec2a-eab1-483b-bf7f-4830ff7c4e28" (UID: "0283ec2a-eab1-483b-bf7f-4830ff7c4e28"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:47:04.982727 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:04.982690    4544 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-tmp\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:47:04.982727 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:04.982712    4544 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-trusted-ca-bundle\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:47:04.982803 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:04.982733    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-console-config" pod "0283ec2a-eab1-483b-bf7f-4830ff7c4e28" (UID: "0283ec2a-eab1-483b-bf7f-4830ff7c4e28"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:47:04.982803 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:04.982742    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-oauth-serving-cert" pod "0283ec2a-eab1-483b-bf7f-4830ff7c4e28" (UID: "0283ec2a-eab1-483b-bf7f-4830ff7c4e28"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:47:04.984317 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:04.984269    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-console-oauth-config" pod "0283ec2a-eab1-483b-bf7f-4830ff7c4e28" (UID: "0283ec2a-eab1-483b-bf7f-4830ff7c4e28"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:47:04.984503 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:04.984486    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-console-serving-cert" pod "0283ec2a-eab1-483b-bf7f-4830ff7c4e28" (UID: "0283ec2a-eab1-483b-bf7f-4830ff7c4e28"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:47:04.984582 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:04.984567    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-kube-api-access-c4h2v" pod "0283ec2a-eab1-483b-bf7f-4830ff7c4e28" (UID: "0283ec2a-eab1-483b-bf7f-4830ff7c4e28"). InnerVolumeSpecName "kube-api-access-c4h2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:47:05.083020 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:05.082993    4544 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-console-oauth-config\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:47:05.083020 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:05.083020    4544 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:47:05.083274 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:05.083030    4544 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-console-serving-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:47:05.083274 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:05.083039    4544 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-console-config\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:47:05.083274 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:05.083048    4544 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-oauth-serving-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:47:05.083274 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:05.083057    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4h2v\" (UniqueName: \"kubernetes.io/projected/0283ec2a-eab1-483b-bf7f-4830ff7c4e28-kube-api-access-c4h2v\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:47:05.616632 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:05.616591    4544 generic.go:356] "Generic (PLEG): container finished" podID="0283ec2a-eab1-483b-bf7f-4830ff7c4e28" containerID="f98db6976acdfc68a94be76f68abb05993fd1ab81e0732ee5a4520bd73a2e213" exitCode=0
Oct 30 12:47:05.616844 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:05.616640    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76bcb6b7cd-hw2zh"
Oct 30 12:47:05.616844 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:05.616686    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76bcb6b7cd-hw2zh" event={"ID":"0283ec2a-eab1-483b-bf7f-4830ff7c4e28","Type":"ContainerDied","Data":"f98db6976acdfc68a94be76f68abb05993fd1ab81e0732ee5a4520bd73a2e213"}
Oct 30 12:47:05.616844 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:05.616743    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76bcb6b7cd-hw2zh" event={"ID":"0283ec2a-eab1-483b-bf7f-4830ff7c4e28","Type":"ContainerDied","Data":"a2ed1e0360ff141aa6aa994040e91c20479ad1890143e47386adffc14b29a04c"}
Oct 30 12:47:05.616844 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:05.616777    4544 scope.go:122] "RemoveContainer" containerID="f98db6976acdfc68a94be76f68abb05993fd1ab81e0732ee5a4520bd73a2e213"
Oct 30 12:47:05.624737 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:05.624714    4544 scope.go:122] "RemoveContainer" containerID="f98db6976acdfc68a94be76f68abb05993fd1ab81e0732ee5a4520bd73a2e213"
Oct 30 12:47:05.625046 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:47:05.625029    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f98db6976acdfc68a94be76f68abb05993fd1ab81e0732ee5a4520bd73a2e213\": container with ID starting with f98db6976acdfc68a94be76f68abb05993fd1ab81e0732ee5a4520bd73a2e213 not found: ID does not exist" containerID="f98db6976acdfc68a94be76f68abb05993fd1ab81e0732ee5a4520bd73a2e213"
Oct 30 12:47:05.625100 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:05.625059    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f98db6976acdfc68a94be76f68abb05993fd1ab81e0732ee5a4520bd73a2e213"} err="failed to get container status \"f98db6976acdfc68a94be76f68abb05993fd1ab81e0732ee5a4520bd73a2e213\": rpc error: code = NotFound desc = could not find container \"f98db6976acdfc68a94be76f68abb05993fd1ab81e0732ee5a4520bd73a2e213\": container with ID starting with f98db6976acdfc68a94be76f68abb05993fd1ab81e0732ee5a4520bd73a2e213 not found: ID does not exist"
Oct 30 12:47:05.667085 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:05.667046    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76bcb6b7cd-hw2zh"]
Oct 30 12:47:05.679128 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:05.679092    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76bcb6b7cd-hw2zh"]
Oct 30 12:47:07.082130 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:07.082093    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0283ec2a-eab1-483b-bf7f-4830ff7c4e28" path="/var/lib/kubelet/pods/0283ec2a-eab1-483b-bf7f-4830ff7c4e28/volumes"
Oct 30 12:47:16.543500 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:16.543467    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:16.654637 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:16.654600    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:29.004184 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:47:29.004098    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 12:47:29.333646 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:29.333620    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:29.444765 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:29.444734    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:37.720935 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:37.720901    4544 generic.go:356] "Generic (PLEG): container finished" podID="74bb378c-e8ce-4261-9ce4-4eea638acd5d" containerID="49db8b5fe3d513e430b6e60ce696f3a4c3b9459135ac817a74749fd6bc39d3c3" exitCode=0
Oct 30 12:47:37.721341 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:37.720946    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5djl4-d9vnz" event={"ID":"74bb378c-e8ce-4261-9ce4-4eea638acd5d","Type":"ContainerDied","Data":"49db8b5fe3d513e430b6e60ce696f3a4c3b9459135ac817a74749fd6bc39d3c3"}
Oct 30 12:47:38.856763 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:38.856733    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5djl4-d9vnz"
Oct 30 12:47:39.036143 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:39.036056    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74bb378c-e8ce-4261-9ce4-4eea638acd5d-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74bb378c-e8ce-4261-9ce4-4eea638acd5d-service-ca-bundle\") pod \"74bb378c-e8ce-4261-9ce4-4eea638acd5d\" (UID: \"74bb378c-e8ce-4261-9ce4-4eea638acd5d\") "
Oct 30 12:47:39.036143 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:39.036110    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74bb378c-e8ce-4261-9ce4-4eea638acd5d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74bb378c-e8ce-4261-9ce4-4eea638acd5d-trusted-ca-bundle\") pod \"74bb378c-e8ce-4261-9ce4-4eea638acd5d\" (UID: \"74bb378c-e8ce-4261-9ce4-4eea638acd5d\") "
Oct 30 12:47:39.036367 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:39.036155    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74bb378c-e8ce-4261-9ce4-4eea638acd5d-kube-api-access-v6lb8\" (UniqueName: \"kubernetes.io/projected/74bb378c-e8ce-4261-9ce4-4eea638acd5d-kube-api-access-v6lb8\") pod \"74bb378c-e8ce-4261-9ce4-4eea638acd5d\" (UID: \"74bb378c-e8ce-4261-9ce4-4eea638acd5d\") "
Oct 30 12:47:39.036367 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:39.036276    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74bb378c-e8ce-4261-9ce4-4eea638acd5d-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/74bb378c-e8ce-4261-9ce4-4eea638acd5d-archives-path\") pod \"74bb378c-e8ce-4261-9ce4-4eea638acd5d\" (UID: \"74bb378c-e8ce-4261-9ce4-4eea638acd5d\") "
Oct 30 12:47:39.036519 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:39.036460    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74bb378c-e8ce-4261-9ce4-4eea638acd5d-service-ca-bundle" pod "74bb378c-e8ce-4261-9ce4-4eea638acd5d" (UID: "74bb378c-e8ce-4261-9ce4-4eea638acd5d"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:47:39.036597 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:39.036527    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74bb378c-e8ce-4261-9ce4-4eea638acd5d-trusted-ca-bundle" pod "74bb378c-e8ce-4261-9ce4-4eea638acd5d" (UID: "74bb378c-e8ce-4261-9ce4-4eea638acd5d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:47:39.036655 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:39.036620    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74bb378c-e8ce-4261-9ce4-4eea638acd5d-archives-path" pod "74bb378c-e8ce-4261-9ce4-4eea638acd5d" (UID: "74bb378c-e8ce-4261-9ce4-4eea638acd5d"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:47:39.038264 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:39.038243    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74bb378c-e8ce-4261-9ce4-4eea638acd5d-kube-api-access-v6lb8" pod "74bb378c-e8ce-4261-9ce4-4eea638acd5d" (UID: "74bb378c-e8ce-4261-9ce4-4eea638acd5d"). InnerVolumeSpecName "kube-api-access-v6lb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:47:39.137441 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:39.137409    4544 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74bb378c-e8ce-4261-9ce4-4eea638acd5d-service-ca-bundle\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:47:39.137441 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:39.137435    4544 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74bb378c-e8ce-4261-9ce4-4eea638acd5d-trusted-ca-bundle\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:47:39.137441 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:39.137445    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6lb8\" (UniqueName: \"kubernetes.io/projected/74bb378c-e8ce-4261-9ce4-4eea638acd5d-kube-api-access-v6lb8\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:47:39.137668 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:39.137456    4544 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/74bb378c-e8ce-4261-9ce4-4eea638acd5d-archives-path\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:47:39.728963 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:39.728938    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5djl4-d9vnz"
Oct 30 12:47:39.729132 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:39.728937    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5djl4-d9vnz" event={"ID":"74bb378c-e8ce-4261-9ce4-4eea638acd5d","Type":"ContainerDied","Data":"e301d2780404e50db4af3ec28dd6771c03971d9afc3eb353201bc5e163b70c1d"}
Oct 30 12:47:39.729132 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:39.729052    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e301d2780404e50db4af3ec28dd6771c03971d9afc3eb353201bc5e163b70c1d"
Oct 30 12:47:40.206495 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:40.206466    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:40.317850 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:40.314590    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:41.980934 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:41.980901    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:42.092511 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:42.092476    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:43.970335 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:43.970299    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:51.241108 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.241073    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:51.711507 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.711474    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-69c8c76d5f-247nt"]
Oct 30 12:47:51.711804 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.711787    4544 cpu_manager.go:414] "containerMap: removing container" podUID="0283ec2a-eab1-483b-bf7f-4830ff7c4e28" containerName="console"
Oct 30 12:47:51.711881 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.711807    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0283ec2a-eab1-483b-bf7f-4830ff7c4e28" containerName="console"
Oct 30 12:47:51.711881 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.711823    4544 cpu_manager.go:414] "containerMap: removing container" podUID="74bb378c-e8ce-4261-9ce4-4eea638acd5d" containerName="insights-gathering"
Oct 30 12:47:51.711881 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.711830    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74bb378c-e8ce-4261-9ce4-4eea638acd5d" containerName="insights-gathering"
Oct 30 12:47:51.712023 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.711930    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="0283ec2a-eab1-483b-bf7f-4830ff7c4e28" containerName="console"
Oct 30 12:47:51.712023 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.711942    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="74bb378c-e8ce-4261-9ce4-4eea638acd5d" containerName="insights-gathering"
Oct 30 12:47:51.714827 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.714807    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69c8c76d5f-247nt"
Oct 30 12:47:51.731832 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.731799    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b44fb3f6-1691-4588-8a15-d0a0afe25dd1-console-oauth-config\") pod \"console-69c8c76d5f-247nt\" (UID: \"b44fb3f6-1691-4588-8a15-d0a0afe25dd1\") " pod="openshift-console/console-69c8c76d5f-247nt"
Oct 30 12:47:51.731948 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.731837    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b44fb3f6-1691-4588-8a15-d0a0afe25dd1-trusted-ca-bundle\") pod \"console-69c8c76d5f-247nt\" (UID: \"b44fb3f6-1691-4588-8a15-d0a0afe25dd1\") " pod="openshift-console/console-69c8c76d5f-247nt"
Oct 30 12:47:51.731948 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.731908    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b44fb3f6-1691-4588-8a15-d0a0afe25dd1-service-ca\") pod \"console-69c8c76d5f-247nt\" (UID: \"b44fb3f6-1691-4588-8a15-d0a0afe25dd1\") " pod="openshift-console/console-69c8c76d5f-247nt"
Oct 30 12:47:51.732045 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.731964    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97tzp\" (UniqueName: \"kubernetes.io/projected/b44fb3f6-1691-4588-8a15-d0a0afe25dd1-kube-api-access-97tzp\") pod \"console-69c8c76d5f-247nt\" (UID: \"b44fb3f6-1691-4588-8a15-d0a0afe25dd1\") " pod="openshift-console/console-69c8c76d5f-247nt"
Oct 30 12:47:51.732045 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.732006    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b44fb3f6-1691-4588-8a15-d0a0afe25dd1-oauth-serving-cert\") pod \"console-69c8c76d5f-247nt\" (UID: \"b44fb3f6-1691-4588-8a15-d0a0afe25dd1\") " pod="openshift-console/console-69c8c76d5f-247nt"
Oct 30 12:47:51.732130 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.732061    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b44fb3f6-1691-4588-8a15-d0a0afe25dd1-tmp\") pod \"console-69c8c76d5f-247nt\" (UID: \"b44fb3f6-1691-4588-8a15-d0a0afe25dd1\") " pod="openshift-console/console-69c8c76d5f-247nt"
Oct 30 12:47:51.732130 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.732089    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b44fb3f6-1691-4588-8a15-d0a0afe25dd1-console-config\") pod \"console-69c8c76d5f-247nt\" (UID: \"b44fb3f6-1691-4588-8a15-d0a0afe25dd1\") " pod="openshift-console/console-69c8c76d5f-247nt"
Oct 30 12:47:51.732130 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.732107    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b44fb3f6-1691-4588-8a15-d0a0afe25dd1-console-serving-cert\") pod \"console-69c8c76d5f-247nt\" (UID: \"b44fb3f6-1691-4588-8a15-d0a0afe25dd1\") " pod="openshift-console/console-69c8c76d5f-247nt"
Oct 30 12:47:51.740951 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.740930    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69c8c76d5f-247nt"]
Oct 30 12:47:51.832525 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.832480    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b44fb3f6-1691-4588-8a15-d0a0afe25dd1-console-config\") pod \"console-69c8c76d5f-247nt\" (UID: \"b44fb3f6-1691-4588-8a15-d0a0afe25dd1\") " pod="openshift-console/console-69c8c76d5f-247nt"
Oct 30 12:47:51.832525 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.832525    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b44fb3f6-1691-4588-8a15-d0a0afe25dd1-console-serving-cert\") pod \"console-69c8c76d5f-247nt\" (UID: \"b44fb3f6-1691-4588-8a15-d0a0afe25dd1\") " pod="openshift-console/console-69c8c76d5f-247nt"
Oct 30 12:47:51.832740 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.832555    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b44fb3f6-1691-4588-8a15-d0a0afe25dd1-console-oauth-config\") pod \"console-69c8c76d5f-247nt\" (UID: \"b44fb3f6-1691-4588-8a15-d0a0afe25dd1\") " pod="openshift-console/console-69c8c76d5f-247nt"
Oct 30 12:47:51.832740 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.832696    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b44fb3f6-1691-4588-8a15-d0a0afe25dd1-trusted-ca-bundle\") pod \"console-69c8c76d5f-247nt\" (UID: \"b44fb3f6-1691-4588-8a15-d0a0afe25dd1\") " pod="openshift-console/console-69c8c76d5f-247nt"
Oct 30 12:47:51.832841 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.832741    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b44fb3f6-1691-4588-8a15-d0a0afe25dd1-service-ca\") pod \"console-69c8c76d5f-247nt\" (UID: \"b44fb3f6-1691-4588-8a15-d0a0afe25dd1\") " pod="openshift-console/console-69c8c76d5f-247nt"
Oct 30 12:47:51.832841 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.832766    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97tzp\" (UniqueName: \"kubernetes.io/projected/b44fb3f6-1691-4588-8a15-d0a0afe25dd1-kube-api-access-97tzp\") pod \"console-69c8c76d5f-247nt\" (UID: \"b44fb3f6-1691-4588-8a15-d0a0afe25dd1\") " pod="openshift-console/console-69c8c76d5f-247nt"
Oct 30 12:47:51.832841 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.832819    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b44fb3f6-1691-4588-8a15-d0a0afe25dd1-oauth-serving-cert\") pod \"console-69c8c76d5f-247nt\" (UID: \"b44fb3f6-1691-4588-8a15-d0a0afe25dd1\") " pod="openshift-console/console-69c8c76d5f-247nt"
Oct 30 12:47:51.832998 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.832917    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b44fb3f6-1691-4588-8a15-d0a0afe25dd1-tmp\") pod \"console-69c8c76d5f-247nt\" (UID: \"b44fb3f6-1691-4588-8a15-d0a0afe25dd1\") " pod="openshift-console/console-69c8c76d5f-247nt"
Oct 30 12:47:51.833415 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.833363    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b44fb3f6-1691-4588-8a15-d0a0afe25dd1-tmp\") pod \"console-69c8c76d5f-247nt\" (UID: \"b44fb3f6-1691-4588-8a15-d0a0afe25dd1\") " pod="openshift-console/console-69c8c76d5f-247nt"
Oct 30 12:47:51.833549 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.833533    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b44fb3f6-1691-4588-8a15-d0a0afe25dd1-service-ca\") pod \"console-69c8c76d5f-247nt\" (UID: \"b44fb3f6-1691-4588-8a15-d0a0afe25dd1\") " pod="openshift-console/console-69c8c76d5f-247nt"
Oct 30 12:47:51.833615 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.833544    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b44fb3f6-1691-4588-8a15-d0a0afe25dd1-oauth-serving-cert\") pod \"console-69c8c76d5f-247nt\" (UID: \"b44fb3f6-1691-4588-8a15-d0a0afe25dd1\") " pod="openshift-console/console-69c8c76d5f-247nt"
Oct 30 12:47:51.833615 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.833546    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b44fb3f6-1691-4588-8a15-d0a0afe25dd1-console-config\") pod \"console-69c8c76d5f-247nt\" (UID: \"b44fb3f6-1691-4588-8a15-d0a0afe25dd1\") " pod="openshift-console/console-69c8c76d5f-247nt"
Oct 30 12:47:51.833726 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.833710    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b44fb3f6-1691-4588-8a15-d0a0afe25dd1-trusted-ca-bundle\") pod \"console-69c8c76d5f-247nt\" (UID: \"b44fb3f6-1691-4588-8a15-d0a0afe25dd1\") " pod="openshift-console/console-69c8c76d5f-247nt"
Oct 30 12:47:51.835048 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.835024    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b44fb3f6-1691-4588-8a15-d0a0afe25dd1-console-serving-cert\") pod \"console-69c8c76d5f-247nt\" (UID: \"b44fb3f6-1691-4588-8a15-d0a0afe25dd1\") " pod="openshift-console/console-69c8c76d5f-247nt"
Oct 30 12:47:51.835136 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.835121    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b44fb3f6-1691-4588-8a15-d0a0afe25dd1-console-oauth-config\") pod \"console-69c8c76d5f-247nt\" (UID: \"b44fb3f6-1691-4588-8a15-d0a0afe25dd1\") " pod="openshift-console/console-69c8c76d5f-247nt"
Oct 30 12:47:51.856284 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.856247    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97tzp\" (UniqueName: \"kubernetes.io/projected/b44fb3f6-1691-4588-8a15-d0a0afe25dd1-kube-api-access-97tzp\") pod \"console-69c8c76d5f-247nt\" (UID: \"b44fb3f6-1691-4588-8a15-d0a0afe25dd1\") " pod="openshift-console/console-69c8c76d5f-247nt"
Oct 30 12:47:51.857121 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.857094    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:51.966770 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:51.966680    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:52.027797 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:52.027761    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69c8c76d5f-247nt"
Oct 30 12:47:52.159408 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:52.159362    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:52.278601 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:52.278496    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:47:52.325992 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:52.325957    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69c8c76d5f-247nt"]
Oct 30 12:47:52.331010 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:47:52.330985    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb44fb3f6_1691_4588_8a15_d0a0afe25dd1.slice/crio-95c27dc2b7292a51dc91224d7291f30f1e862cf8d1fa39d8700b0305c4149e0e WatchSource:0}: Error finding container 95c27dc2b7292a51dc91224d7291f30f1e862cf8d1fa39d8700b0305c4149e0e: Status 404 returned error can't find the container with id 95c27dc2b7292a51dc91224d7291f30f1e862cf8d1fa39d8700b0305c4149e0e
Oct 30 12:47:52.772452 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:52.772411    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69c8c76d5f-247nt" event={"ID":"b44fb3f6-1691-4588-8a15-d0a0afe25dd1","Type":"ContainerStarted","Data":"f610e12fcc51dd8bb4a2a046a57ba506825831ab43ac97f99998be804984147f"}
Oct 30 12:47:52.772452 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:52.772460    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69c8c76d5f-247nt" event={"ID":"b44fb3f6-1691-4588-8a15-d0a0afe25dd1","Type":"ContainerStarted","Data":"95c27dc2b7292a51dc91224d7291f30f1e862cf8d1fa39d8700b0305c4149e0e"}
Oct 30 12:47:52.815302 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:47:52.815252    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-69c8c76d5f-247nt" podStartSLOduration=1.815216912 podStartE2EDuration="1.815216912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:47:52.81498356 +0000 UTC m=+167.862117512" watchObservedRunningTime="2026-10-30 12:47:52.815216912 +0000 UTC m=+167.862350859"
Oct 30 12:47:59.007993 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:47:59.007948    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 12:48:02.028327 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:02.028287    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69c8c76d5f-247nt"
Oct 30 12:48:02.028836 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:02.028349    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69c8c76d5f-247nt"
Oct 30 12:48:02.032878 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:02.032856    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69c8c76d5f-247nt"
Oct 30 12:48:02.809374 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:02.809347    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69c8c76d5f-247nt"
Oct 30 12:48:02.881985 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:02.881954    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5dfc766f68-gfrkb"]
Oct 30 12:48:12.593210 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:12.593178    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:12.705345 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:12.705313    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:27.901581 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:27.901517    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5dfc766f68-gfrkb" podUID="22715ff5-9438-43ed-82e2-cffb9c93d107" containerName="console" containerID="cri-o://a25f1b7d5eda12181c3fb6548a9d362143f29b88e25407bd8712cd25259d0db4" gracePeriod=15
Oct 30 12:48:28.140489 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:28.140467    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dfc766f68-gfrkb"
Oct 30 12:48:28.196859 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:28.196794    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22715ff5-9438-43ed-82e2-cffb9c93d107-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/22715ff5-9438-43ed-82e2-cffb9c93d107-oauth-serving-cert\") pod \"22715ff5-9438-43ed-82e2-cffb9c93d107\" (UID: \"22715ff5-9438-43ed-82e2-cffb9c93d107\") "
Oct 30 12:48:28.196859 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:28.196832    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22715ff5-9438-43ed-82e2-cffb9c93d107-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/22715ff5-9438-43ed-82e2-cffb9c93d107-console-serving-cert\") pod \"22715ff5-9438-43ed-82e2-cffb9c93d107\" (UID: \"22715ff5-9438-43ed-82e2-cffb9c93d107\") "
Oct 30 12:48:28.197032 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:28.196866    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22715ff5-9438-43ed-82e2-cffb9c93d107-kube-api-access-sfmrs\" (UniqueName: \"kubernetes.io/projected/22715ff5-9438-43ed-82e2-cffb9c93d107-kube-api-access-sfmrs\") pod \"22715ff5-9438-43ed-82e2-cffb9c93d107\" (UID: \"22715ff5-9438-43ed-82e2-cffb9c93d107\") "
Oct 30 12:48:28.197032 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:28.196893    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22715ff5-9438-43ed-82e2-cffb9c93d107-console-config\" (UniqueName: \"kubernetes.io/configmap/22715ff5-9438-43ed-82e2-cffb9c93d107-console-config\") pod \"22715ff5-9438-43ed-82e2-cffb9c93d107\" (UID: \"22715ff5-9438-43ed-82e2-cffb9c93d107\") "
Oct 30 12:48:28.197032 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:28.196947    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22715ff5-9438-43ed-82e2-cffb9c93d107-tmp\" (UniqueName: \"kubernetes.io/empty-dir/22715ff5-9438-43ed-82e2-cffb9c93d107-tmp\") pod \"22715ff5-9438-43ed-82e2-cffb9c93d107\" (UID: \"22715ff5-9438-43ed-82e2-cffb9c93d107\") "
Oct 30 12:48:28.197032 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:28.196989    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22715ff5-9438-43ed-82e2-cffb9c93d107-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22715ff5-9438-43ed-82e2-cffb9c93d107-trusted-ca-bundle\") pod \"22715ff5-9438-43ed-82e2-cffb9c93d107\" (UID: \"22715ff5-9438-43ed-82e2-cffb9c93d107\") "
Oct 30 12:48:28.197238 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:28.197041    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22715ff5-9438-43ed-82e2-cffb9c93d107-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/22715ff5-9438-43ed-82e2-cffb9c93d107-console-oauth-config\") pod \"22715ff5-9438-43ed-82e2-cffb9c93d107\" (UID: \"22715ff5-9438-43ed-82e2-cffb9c93d107\") "
Oct 30 12:48:28.197238 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:28.197069    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22715ff5-9438-43ed-82e2-cffb9c93d107-service-ca\" (UniqueName: \"kubernetes.io/configmap/22715ff5-9438-43ed-82e2-cffb9c93d107-service-ca\") pod \"22715ff5-9438-43ed-82e2-cffb9c93d107\" (UID: \"22715ff5-9438-43ed-82e2-cffb9c93d107\") "
Oct 30 12:48:28.197341 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:28.197323    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22715ff5-9438-43ed-82e2-cffb9c93d107-oauth-serving-cert" pod "22715ff5-9438-43ed-82e2-cffb9c93d107" (UID: "22715ff5-9438-43ed-82e2-cffb9c93d107"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:48:28.197522 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:28.197382    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22715ff5-9438-43ed-82e2-cffb9c93d107-console-config" pod "22715ff5-9438-43ed-82e2-cffb9c93d107" (UID: "22715ff5-9438-43ed-82e2-cffb9c93d107"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:48:28.197522 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:28.197457    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22715ff5-9438-43ed-82e2-cffb9c93d107-tmp" pod "22715ff5-9438-43ed-82e2-cffb9c93d107" (UID: "22715ff5-9438-43ed-82e2-cffb9c93d107"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:48:28.197677 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:28.197657    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22715ff5-9438-43ed-82e2-cffb9c93d107-trusted-ca-bundle" pod "22715ff5-9438-43ed-82e2-cffb9c93d107" (UID: "22715ff5-9438-43ed-82e2-cffb9c93d107"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:48:28.197764 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:28.197743    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22715ff5-9438-43ed-82e2-cffb9c93d107-service-ca" pod "22715ff5-9438-43ed-82e2-cffb9c93d107" (UID: "22715ff5-9438-43ed-82e2-cffb9c93d107"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:48:28.199237 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:28.199199    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22715ff5-9438-43ed-82e2-cffb9c93d107-console-oauth-config" pod "22715ff5-9438-43ed-82e2-cffb9c93d107" (UID: "22715ff5-9438-43ed-82e2-cffb9c93d107"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:48:28.199363 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:28.199346    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22715ff5-9438-43ed-82e2-cffb9c93d107-console-serving-cert" pod "22715ff5-9438-43ed-82e2-cffb9c93d107" (UID: "22715ff5-9438-43ed-82e2-cffb9c93d107"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:48:28.199577 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:28.199548    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22715ff5-9438-43ed-82e2-cffb9c93d107-kube-api-access-sfmrs" pod "22715ff5-9438-43ed-82e2-cffb9c93d107" (UID: "22715ff5-9438-43ed-82e2-cffb9c93d107"). InnerVolumeSpecName "kube-api-access-sfmrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:48:28.298099 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:28.298062    4544 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22715ff5-9438-43ed-82e2-cffb9c93d107-trusted-ca-bundle\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:48:28.298099 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:28.298098    4544 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/22715ff5-9438-43ed-82e2-cffb9c93d107-console-oauth-config\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:48:28.298099 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:28.298109    4544 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/22715ff5-9438-43ed-82e2-cffb9c93d107-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:48:28.298287 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:28.298119    4544 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/22715ff5-9438-43ed-82e2-cffb9c93d107-oauth-serving-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:48:28.298287 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:28.298128    4544 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/22715ff5-9438-43ed-82e2-cffb9c93d107-console-serving-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:48:28.298287 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:28.298137    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfmrs\" (UniqueName: \"kubernetes.io/projected/22715ff5-9438-43ed-82e2-cffb9c93d107-kube-api-access-sfmrs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:48:28.298287 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:28.298147    4544 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/22715ff5-9438-43ed-82e2-cffb9c93d107-console-config\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:48:28.298287 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:28.298155    4544 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22715ff5-9438-43ed-82e2-cffb9c93d107-tmp\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:48:28.895605 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:28.895571    4544 generic.go:356] "Generic (PLEG): container finished" podID="22715ff5-9438-43ed-82e2-cffb9c93d107" containerID="a25f1b7d5eda12181c3fb6548a9d362143f29b88e25407bd8712cd25259d0db4" exitCode=0
Oct 30 12:48:28.895774 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:28.895614    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dfc766f68-gfrkb" event={"ID":"22715ff5-9438-43ed-82e2-cffb9c93d107","Type":"ContainerDied","Data":"a25f1b7d5eda12181c3fb6548a9d362143f29b88e25407bd8712cd25259d0db4"}
Oct 30 12:48:28.895774 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:28.895625    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dfc766f68-gfrkb"
Oct 30 12:48:28.895774 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:28.895656    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dfc766f68-gfrkb" event={"ID":"22715ff5-9438-43ed-82e2-cffb9c93d107","Type":"ContainerDied","Data":"f493c4f0f46d724f957e108b788ec9f0c5852cb7a1a453a959f673357975cbeb"}
Oct 30 12:48:28.895774 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:28.895676    4544 scope.go:122] "RemoveContainer" containerID="a25f1b7d5eda12181c3fb6548a9d362143f29b88e25407bd8712cd25259d0db4"
Oct 30 12:48:28.904789 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:28.904596    4544 scope.go:122] "RemoveContainer" containerID="a25f1b7d5eda12181c3fb6548a9d362143f29b88e25407bd8712cd25259d0db4"
Oct 30 12:48:28.905026 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:48:28.904880    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a25f1b7d5eda12181c3fb6548a9d362143f29b88e25407bd8712cd25259d0db4\": container with ID starting with a25f1b7d5eda12181c3fb6548a9d362143f29b88e25407bd8712cd25259d0db4 not found: ID does not exist" containerID="a25f1b7d5eda12181c3fb6548a9d362143f29b88e25407bd8712cd25259d0db4"
Oct 30 12:48:28.905026 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:28.904904    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a25f1b7d5eda12181c3fb6548a9d362143f29b88e25407bd8712cd25259d0db4"} err="failed to get container status \"a25f1b7d5eda12181c3fb6548a9d362143f29b88e25407bd8712cd25259d0db4\": rpc error: code = NotFound desc = could not find container \"a25f1b7d5eda12181c3fb6548a9d362143f29b88e25407bd8712cd25259d0db4\": container with ID starting with a25f1b7d5eda12181c3fb6548a9d362143f29b88e25407bd8712cd25259d0db4 not found: ID does not exist"
Oct 30 12:48:28.948280 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:28.948255    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5dfc766f68-gfrkb"]
Oct 30 12:48:28.955794 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:28.955773    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5dfc766f68-gfrkb"]
Oct 30 12:48:29.003635 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:48:29.003600    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 12:48:29.081967 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:29.081935    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22715ff5-9438-43ed-82e2-cffb9c93d107" path="/var/lib/kubelet/pods/22715ff5-9438-43ed-82e2-cffb9c93d107/volumes"
Oct 30 12:48:51.360628 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:48:51.360525    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:48:59.004251 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:48:59.004196    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 12:49:29.004173 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:49:29.004133    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 12:49:51.552797 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:49:51.552760    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:53.983522 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:49:53.983490    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:54.124106 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:49:54.124067    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:49:59.005253 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:49:59.005199    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 12:50:17.026244 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:17.026145    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:17.571582 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:17.571549    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:20.678166 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:20.678127    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:20.846025 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:20.845994    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:22.452115 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:22.452082    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:22.644042 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:22.644011    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:22.809394 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:22.809324    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-6nbzr"]
Oct 30 12:50:22.809636 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:22.809620    4544 cpu_manager.go:414] "containerMap: removing container" podUID="22715ff5-9438-43ed-82e2-cffb9c93d107" containerName="console"
Oct 30 12:50:22.809710 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:22.809638    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22715ff5-9438-43ed-82e2-cffb9c93d107" containerName="console"
Oct 30 12:50:22.809761 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:22.809730    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="22715ff5-9438-43ed-82e2-cffb9c93d107" containerName="console"
Oct 30 12:50:22.812602 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:22.812584    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6nbzr"
Oct 30 12:50:22.890471 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:22.890442    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 12:50:22.902513 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:22.902477    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-8rq4h\""
Oct 30 12:50:22.903369 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:22.903344    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-6nbzr"]
Oct 30 12:50:22.921728 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:22.921704    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmlft\" (UniqueName: \"kubernetes.io/projected/8c41830a-e18f-4b80-a625-d013bc6f6f5a-kube-api-access-bmlft\") pod \"cert-manager-webhook-5dbbbf75cc-6nbzr\" (UID: \"8c41830a-e18f-4b80-a625-d013bc6f6f5a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6nbzr"
Oct 30 12:50:22.921806 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:22.921745    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c41830a-e18f-4b80-a625-d013bc6f6f5a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-6nbzr\" (UID: \"8c41830a-e18f-4b80-a625-d013bc6f6f5a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6nbzr"
Oct 30 12:50:22.949478 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:22.949453    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:22.981897 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:22.981871    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 12:50:23.022759 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:23.022721    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmlft\" (UniqueName: \"kubernetes.io/projected/8c41830a-e18f-4b80-a625-d013bc6f6f5a-kube-api-access-bmlft\") pod \"cert-manager-webhook-5dbbbf75cc-6nbzr\" (UID: \"8c41830a-e18f-4b80-a625-d013bc6f6f5a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6nbzr"
Oct 30 12:50:23.022862 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:23.022778    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c41830a-e18f-4b80-a625-d013bc6f6f5a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-6nbzr\" (UID: \"8c41830a-e18f-4b80-a625-d013bc6f6f5a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6nbzr"
Oct 30 12:50:23.062326 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:23.062266    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c41830a-e18f-4b80-a625-d013bc6f6f5a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-6nbzr\" (UID: \"8c41830a-e18f-4b80-a625-d013bc6f6f5a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6nbzr"
Oct 30 12:50:23.062495 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:23.062473    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmlft\" (UniqueName: \"kubernetes.io/projected/8c41830a-e18f-4b80-a625-d013bc6f6f5a-kube-api-access-bmlft\") pod \"cert-manager-webhook-5dbbbf75cc-6nbzr\" (UID: \"8c41830a-e18f-4b80-a625-d013bc6f6f5a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6nbzr"
Oct 30 12:50:23.135392 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:23.135355    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6nbzr"
Oct 30 12:50:23.467268 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:23.467215    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-6nbzr"]
Oct 30 12:50:23.471870 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:50:23.471829    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c41830a_e18f_4b80_a625_d013bc6f6f5a.slice/crio-27ac2bd4c3affd374c3eeb0a6bf3a1d00be979f87d269ffc9dcf2738007f03f8 WatchSource:0}: Error finding container 27ac2bd4c3affd374c3eeb0a6bf3a1d00be979f87d269ffc9dcf2738007f03f8: Status 404 returned error can't find the container with id 27ac2bd4c3affd374c3eeb0a6bf3a1d00be979f87d269ffc9dcf2738007f03f8
Oct 30 12:50:23.473788 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:23.473770    4544 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 12:50:24.269710 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:24.269672    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6nbzr" event={"ID":"8c41830a-e18f-4b80-a625-d013bc6f6f5a","Type":"ContainerStarted","Data":"27ac2bd4c3affd374c3eeb0a6bf3a1d00be979f87d269ffc9dcf2738007f03f8"}
Oct 30 12:50:24.722676 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:24.722644    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:25.116606 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:25.116565    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:26.278406 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:26.278376    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6nbzr" event={"ID":"8c41830a-e18f-4b80-a625-d013bc6f6f5a","Type":"ContainerStarted","Data":"01c5d490c0498eee8a80a8c3d38b4adf57968691bc97d722bbf69dfc8e015401"}
Oct 30 12:50:26.278726 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:26.278432    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6nbzr"
Oct 30 12:50:26.329047 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:26.329000    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6nbzr" podStartSLOduration=1.5931606280000001 podStartE2EDuration="4.328986567s" totalImagesPullingTime="2.735825939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:50:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:50:26.328373963 +0000 UTC m=+321.375507915" watchObservedRunningTime="2026-10-30 12:50:26.328986567 +0000 UTC m=+321.376120520"
Oct 30 12:50:27.049492 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:27.049463    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:27.188576 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:27.188526    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:29.005252 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:50:29.005203    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 12:50:32.282877 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:32.282846    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6nbzr"
Oct 30 12:50:32.534357 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:32.534280    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:34.966142 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:34.966105    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:35.082377 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:35.082345    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:36.263256 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:36.263211    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-gkssn"]
Oct 30 12:50:36.266440 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:36.266420    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-gkssn"
Oct 30 12:50:36.294868 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:36.294833    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:50:36.294868 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:36.294834    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-6n6ht\""
Oct 30 12:50:36.294868 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:36.294847    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 30 12:50:36.295842 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:36.295819    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-gkssn"]
Oct 30 12:50:36.314807 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:36.314785    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a143f92-d905-46a8-9916-73f7477b2c86-tmp\") pod \"openshift-lws-operator-8465994787-gkssn\" (UID: \"8a143f92-d905-46a8-9916-73f7477b2c86\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-gkssn"
Oct 30 12:50:36.314897 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:36.314811    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqjrn\" (UniqueName: \"kubernetes.io/projected/8a143f92-d905-46a8-9916-73f7477b2c86-kube-api-access-mqjrn\") pod \"openshift-lws-operator-8465994787-gkssn\" (UID: \"8a143f92-d905-46a8-9916-73f7477b2c86\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-gkssn"
Oct 30 12:50:36.415459 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:36.415430    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a143f92-d905-46a8-9916-73f7477b2c86-tmp\") pod \"openshift-lws-operator-8465994787-gkssn\" (UID: \"8a143f92-d905-46a8-9916-73f7477b2c86\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-gkssn"
Oct 30 12:50:36.415590 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:36.415463    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqjrn\" (UniqueName: \"kubernetes.io/projected/8a143f92-d905-46a8-9916-73f7477b2c86-kube-api-access-mqjrn\") pod \"openshift-lws-operator-8465994787-gkssn\" (UID: \"8a143f92-d905-46a8-9916-73f7477b2c86\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-gkssn"
Oct 30 12:50:36.415811 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:36.415791    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a143f92-d905-46a8-9916-73f7477b2c86-tmp\") pod \"openshift-lws-operator-8465994787-gkssn\" (UID: \"8a143f92-d905-46a8-9916-73f7477b2c86\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-gkssn"
Oct 30 12:50:36.450434 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:36.450410    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqjrn\" (UniqueName: \"kubernetes.io/projected/8a143f92-d905-46a8-9916-73f7477b2c86-kube-api-access-mqjrn\") pod \"openshift-lws-operator-8465994787-gkssn\" (UID: \"8a143f92-d905-46a8-9916-73f7477b2c86\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-gkssn"
Oct 30 12:50:36.577471 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:36.577447    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-gkssn"
Oct 30 12:50:36.874191 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:36.874122    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-gkssn"]
Oct 30 12:50:36.877749 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:50:36.877715    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a143f92_d905_46a8_9916_73f7477b2c86.slice/crio-bc08cecb80ada90ec7590c12df6678edf524ada30e428c1955a068400004677a WatchSource:0}: Error finding container bc08cecb80ada90ec7590c12df6678edf524ada30e428c1955a068400004677a: Status 404 returned error can't find the container with id bc08cecb80ada90ec7590c12df6678edf524ada30e428c1955a068400004677a
Oct 30 12:50:37.313995 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:37.313909    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-gkssn" event={"ID":"8a143f92-d905-46a8-9916-73f7477b2c86","Type":"ContainerStarted","Data":"bc08cecb80ada90ec7590c12df6678edf524ada30e428c1955a068400004677a"}
Oct 30 12:50:39.322535 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:39.322500    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-gkssn" event={"ID":"8a143f92-d905-46a8-9916-73f7477b2c86","Type":"ContainerStarted","Data":"c5ed695e17fa4729dc449b841ab66a8bf607bd1c9a934105fff58106023693b0"}
Oct 30 12:50:39.365608 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:39.365551    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-8465994787-gkssn" podStartSLOduration=1.1103136 podStartE2EDuration="3.365533823s" totalImagesPullingTime="2.255220223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:50:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:50:39.364426643 +0000 UTC m=+334.411560599" watchObservedRunningTime="2026-10-30 12:50:39.365533823 +0000 UTC m=+334.412667778"
Oct 30 12:50:40.557408 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:40.557378    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:40.690501 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:40.690467    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:42.280865 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:42.280822    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:51.673463 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:51.673418    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:53.404628 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:53.404593    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:53.574402 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:53.574366    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:54.228096 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:54.228058    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:54.727056 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:54.727015    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:54.925895 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:54.925859    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c4494f88b-7l28c"]
Oct 30 12:50:54.929201 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:54.929184    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-7l28c"
Oct 30 12:50:54.968070 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:54.968040    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c4494f88b-7l28c"]
Oct 30 12:50:55.014684 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:55.014596    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 12:50:55.014817 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:55.014770    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 12:50:55.020206 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:55.020188    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 12:50:55.022304 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:55.022285    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 30 12:50:55.022418 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:55.022360    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-fmghg\""
Oct 30 12:50:55.022478 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:55.022445    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 12:50:55.022729 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:55.022709    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:55.066617 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:55.066582    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/36d7a70c-cbe6-481a-a8ba-7e2a8511f59d-metrics-cert\") pod \"opendatahub-operator-controller-manager-7c4494f88b-7l28c\" (UID: \"36d7a70c-cbe6-481a-a8ba-7e2a8511f59d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-7l28c"
Oct 30 12:50:55.066754 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:55.066645    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/36d7a70c-cbe6-481a-a8ba-7e2a8511f59d-manifests\") pod \"opendatahub-operator-controller-manager-7c4494f88b-7l28c\" (UID: \"36d7a70c-cbe6-481a-a8ba-7e2a8511f59d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-7l28c"
Oct 30 12:50:55.066754 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:55.066701    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36d7a70c-cbe6-481a-a8ba-7e2a8511f59d-tmp\") pod \"opendatahub-operator-controller-manager-7c4494f88b-7l28c\" (UID: \"36d7a70c-cbe6-481a-a8ba-7e2a8511f59d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-7l28c"
Oct 30 12:50:55.066754 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:55.066737    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s555f\" (UniqueName: \"kubernetes.io/projected/36d7a70c-cbe6-481a-a8ba-7e2a8511f59d-kube-api-access-s555f\") pod \"opendatahub-operator-controller-manager-7c4494f88b-7l28c\" (UID: \"36d7a70c-cbe6-481a-a8ba-7e2a8511f59d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-7l28c"
Oct 30 12:50:55.066919 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:55.066808    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36d7a70c-cbe6-481a-a8ba-7e2a8511f59d-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c4494f88b-7l28c\" (UID: \"36d7a70c-cbe6-481a-a8ba-7e2a8511f59d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-7l28c"
Oct 30 12:50:55.066919 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:55.066896    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/36d7a70c-cbe6-481a-a8ba-7e2a8511f59d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c4494f88b-7l28c\" (UID: \"36d7a70c-cbe6-481a-a8ba-7e2a8511f59d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-7l28c"
Oct 30 12:50:55.167437 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:55.167402    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/36d7a70c-cbe6-481a-a8ba-7e2a8511f59d-metrics-cert\") pod \"opendatahub-operator-controller-manager-7c4494f88b-7l28c\" (UID: \"36d7a70c-cbe6-481a-a8ba-7e2a8511f59d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-7l28c"
Oct 30 12:50:55.167634 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:55.167464    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/36d7a70c-cbe6-481a-a8ba-7e2a8511f59d-manifests\") pod \"opendatahub-operator-controller-manager-7c4494f88b-7l28c\" (UID: \"36d7a70c-cbe6-481a-a8ba-7e2a8511f59d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-7l28c"
Oct 30 12:50:55.167634 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:55.167492    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36d7a70c-cbe6-481a-a8ba-7e2a8511f59d-tmp\") pod \"opendatahub-operator-controller-manager-7c4494f88b-7l28c\" (UID: \"36d7a70c-cbe6-481a-a8ba-7e2a8511f59d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-7l28c"
Oct 30 12:50:55.167634 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:55.167516    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s555f\" (UniqueName: \"kubernetes.io/projected/36d7a70c-cbe6-481a-a8ba-7e2a8511f59d-kube-api-access-s555f\") pod \"opendatahub-operator-controller-manager-7c4494f88b-7l28c\" (UID: \"36d7a70c-cbe6-481a-a8ba-7e2a8511f59d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-7l28c"
Oct 30 12:50:55.167634 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:55.167540    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36d7a70c-cbe6-481a-a8ba-7e2a8511f59d-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c4494f88b-7l28c\" (UID: \"36d7a70c-cbe6-481a-a8ba-7e2a8511f59d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-7l28c"
Oct 30 12:50:55.167634 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:55.167581    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/36d7a70c-cbe6-481a-a8ba-7e2a8511f59d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c4494f88b-7l28c\" (UID: \"36d7a70c-cbe6-481a-a8ba-7e2a8511f59d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-7l28c"
Oct 30 12:50:55.168537 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:55.168460    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/36d7a70c-cbe6-481a-a8ba-7e2a8511f59d-manifests\") pod \"opendatahub-operator-controller-manager-7c4494f88b-7l28c\" (UID: \"36d7a70c-cbe6-481a-a8ba-7e2a8511f59d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-7l28c"
Oct 30 12:50:55.174417 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:55.170755    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/36d7a70c-cbe6-481a-a8ba-7e2a8511f59d-metrics-cert\") pod \"opendatahub-operator-controller-manager-7c4494f88b-7l28c\" (UID: \"36d7a70c-cbe6-481a-a8ba-7e2a8511f59d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-7l28c"
Oct 30 12:50:55.174417 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:55.171267    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36d7a70c-cbe6-481a-a8ba-7e2a8511f59d-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c4494f88b-7l28c\" (UID: \"36d7a70c-cbe6-481a-a8ba-7e2a8511f59d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-7l28c"
Oct 30 12:50:55.174417 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:55.171315    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36d7a70c-cbe6-481a-a8ba-7e2a8511f59d-tmp\") pod \"opendatahub-operator-controller-manager-7c4494f88b-7l28c\" (UID: \"36d7a70c-cbe6-481a-a8ba-7e2a8511f59d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-7l28c"
Oct 30 12:50:55.175069 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:55.175049    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/36d7a70c-cbe6-481a-a8ba-7e2a8511f59d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c4494f88b-7l28c\" (UID: \"36d7a70c-cbe6-481a-a8ba-7e2a8511f59d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-7l28c"
Oct 30 12:50:55.208415 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:55.208386    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s555f\" (UniqueName: \"kubernetes.io/projected/36d7a70c-cbe6-481a-a8ba-7e2a8511f59d-kube-api-access-s555f\") pod \"opendatahub-operator-controller-manager-7c4494f88b-7l28c\" (UID: \"36d7a70c-cbe6-481a-a8ba-7e2a8511f59d\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-7l28c"
Oct 30 12:50:55.214664 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:55.214641    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:55.243900 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:55.243878    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-7l28c"
Oct 30 12:50:55.389415 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:55.389383    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:55.526051 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:55.526019    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:55.601649 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:55.601619    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c4494f88b-7l28c"]
Oct 30 12:50:55.617911 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:50:55.617884    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36d7a70c_cbe6_481a_a8ba_7e2a8511f59d.slice/crio-afdaf823afd71840fb1f676cb7501bd7ac6cc6f1268bea0a843bf1f4d1def560 WatchSource:0}: Error finding container afdaf823afd71840fb1f676cb7501bd7ac6cc6f1268bea0a843bf1f4d1def560: Status 404 returned error can't find the container with id afdaf823afd71840fb1f676cb7501bd7ac6cc6f1268bea0a843bf1f4d1def560
Oct 30 12:50:56.386279 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:56.386238    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-7l28c" event={"ID":"36d7a70c-cbe6-481a-a8ba-7e2a8511f59d","Type":"ContainerStarted","Data":"afdaf823afd71840fb1f676cb7501bd7ac6cc6f1268bea0a843bf1f4d1def560"}
Oct 30 12:50:59.004945 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:50:59.004911    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 12:50:59.398674 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:59.398638    4544 generic.go:356] "Generic (PLEG): container finished" podID="36d7a70c-cbe6-481a-a8ba-7e2a8511f59d" containerID="407fce8e0170aa6344e01fdeb73e50b880b8df4c39a2c9d2426d2fd4fbf87273" exitCode=0
Oct 30 12:50:59.398903 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:59.398692    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-7l28c" event={"ID":"36d7a70c-cbe6-481a-a8ba-7e2a8511f59d","Type":"ContainerDied","Data":"407fce8e0170aa6344e01fdeb73e50b880b8df4c39a2c9d2426d2fd4fbf87273"}
Oct 30 12:50:59.448699 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:59.448642    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:50:59.613360 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:50:59.613321    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:00.404420 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:00.404386    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-7l28c" event={"ID":"36d7a70c-cbe6-481a-a8ba-7e2a8511f59d","Type":"ContainerStarted","Data":"574432c780f939e09ab32e4b439d2e652d814e87d2267db931105752fdc20d2e"}
Oct 30 12:51:00.404933 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:00.404499    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-7l28c"
Oct 30 12:51:00.522780 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:00.522729    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-7l28c" podStartSLOduration=2.8280940980000002 podStartE2EDuration="6.522714624s" totalImagesPullingTime="3.260041566s" totalInitContainerRuntime="434.57896ms" isStatefulPod=false podCreationTimestamp="2026-10-30 12:50:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:00.509860589 +0000 UTC m=+355.556994542" watchObservedRunningTime="2026-10-30 12:51:00.522714624 +0000 UTC m=+355.569848577"
Oct 30 12:51:01.116544 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:01.116504    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:04.004462 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:04.004427    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:04.119871 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:04.119842    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:09.649950 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:09.649907    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:09.801438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:09.801390    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:11.410971 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:11.410939    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-7l28c"
Oct 30 12:51:12.154735 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:12.154700    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:12.368688 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:12.368658    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:15.223710 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:15.223673    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:15.402049 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:15.402012    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:18.787280 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:18.787202    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:19.048440 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:19.048363    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-observability-584f99c788-rvsjg"]
Oct 30 12:51:19.051963 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:19.051939    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-rvsjg"
Oct 30 12:51:19.084130 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:19.084050    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-webhook-cert\""
Oct 30 12:51:19.084130 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:19.084090    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-dockercfg-cx545\""
Oct 30 12:51:19.086245 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:19.086207    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-rvsjg"]
Oct 30 12:51:19.118276 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:19.118242    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:19.158343 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:19.158309    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnqc9\" (UniqueName: \"kubernetes.io/projected/50683686-80a0-4425-8052-104dcf94a8aa-kube-api-access-pnqc9\") pod \"odh-observability-584f99c788-rvsjg\" (UID: \"50683686-80a0-4425-8052-104dcf94a8aa\") " pod="opendatahub/odh-observability-584f99c788-rvsjg"
Oct 30 12:51:19.158498 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:19.158350    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/50683686-80a0-4425-8052-104dcf94a8aa-webhook-certs\") pod \"odh-observability-584f99c788-rvsjg\" (UID: \"50683686-80a0-4425-8052-104dcf94a8aa\") " pod="opendatahub/odh-observability-584f99c788-rvsjg"
Oct 30 12:51:19.259704 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:19.259670    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnqc9\" (UniqueName: \"kubernetes.io/projected/50683686-80a0-4425-8052-104dcf94a8aa-kube-api-access-pnqc9\") pod \"odh-observability-584f99c788-rvsjg\" (UID: \"50683686-80a0-4425-8052-104dcf94a8aa\") " pod="opendatahub/odh-observability-584f99c788-rvsjg"
Oct 30 12:51:19.259903 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:19.259710    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/50683686-80a0-4425-8052-104dcf94a8aa-webhook-certs\") pod \"odh-observability-584f99c788-rvsjg\" (UID: \"50683686-80a0-4425-8052-104dcf94a8aa\") " pod="opendatahub/odh-observability-584f99c788-rvsjg"
Oct 30 12:51:19.259903 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:51:19.259832    4544 secret.go:189] Couldn't get secret opendatahub/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found
Oct 30 12:51:19.260023 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:51:19.259907    4544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50683686-80a0-4425-8052-104dcf94a8aa-webhook-certs podName:50683686-80a0-4425-8052-104dcf94a8aa nodeName:}" failed. No retries permitted until 2026-10-30 12:51:19.759887599 +0000 UTC m=+374.807021543 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/50683686-80a0-4425-8052-104dcf94a8aa-webhook-certs") pod "odh-observability-584f99c788-rvsjg" (UID: "50683686-80a0-4425-8052-104dcf94a8aa") : secret "odh-observability-webhook-cert" not found
Oct 30 12:51:19.283380 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:19.283349    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnqc9\" (UniqueName: \"kubernetes.io/projected/50683686-80a0-4425-8052-104dcf94a8aa-kube-api-access-pnqc9\") pod \"odh-observability-584f99c788-rvsjg\" (UID: \"50683686-80a0-4425-8052-104dcf94a8aa\") " pod="opendatahub/odh-observability-584f99c788-rvsjg"
Oct 30 12:51:19.508470 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:19.508435    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:19.763682 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:19.763601    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/50683686-80a0-4425-8052-104dcf94a8aa-webhook-certs\") pod \"odh-observability-584f99c788-rvsjg\" (UID: \"50683686-80a0-4425-8052-104dcf94a8aa\") " pod="opendatahub/odh-observability-584f99c788-rvsjg"
Oct 30 12:51:19.766103 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:19.766080    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/50683686-80a0-4425-8052-104dcf94a8aa-webhook-certs\") pod \"odh-observability-584f99c788-rvsjg\" (UID: \"50683686-80a0-4425-8052-104dcf94a8aa\") " pod="opendatahub/odh-observability-584f99c788-rvsjg"
Oct 30 12:51:19.964111 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:19.964068    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-rvsjg"
Oct 30 12:51:20.265157 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:20.265087    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-rvsjg"]
Oct 30 12:51:20.269826 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:51:20.269798    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50683686_80a0_4425_8052_104dcf94a8aa.slice/crio-830dc1899f93f840d53b387d34d5641500e7edfc013fe2423b11ac1d8580c785 WatchSource:0}: Error finding container 830dc1899f93f840d53b387d34d5641500e7edfc013fe2423b11ac1d8580c785: Status 404 returned error can't find the container with id 830dc1899f93f840d53b387d34d5641500e7edfc013fe2423b11ac1d8580c785
Oct 30 12:51:20.479187 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:20.479153    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-rvsjg" event={"ID":"50683686-80a0-4425-8052-104dcf94a8aa","Type":"ContainerStarted","Data":"830dc1899f93f840d53b387d34d5641500e7edfc013fe2423b11ac1d8580c785"}
Oct 30 12:51:23.494867 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:23.494832    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-rvsjg" event={"ID":"50683686-80a0-4425-8052-104dcf94a8aa","Type":"ContainerStarted","Data":"7c75f7c93a875f4cef80ccf89436c6d0b3a858882995e785f624155c84677c4b"}
Oct 30 12:51:23.495259 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:23.494964    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-observability-584f99c788-rvsjg"
Oct 30 12:51:23.538800 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:23.538749    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-observability-584f99c788-rvsjg" podStartSLOduration=2.378529167 podStartE2EDuration="4.538732411s" totalImagesPullingTime="2.160203244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:23.538039243 +0000 UTC m=+378.585173207" watchObservedRunningTime="2026-10-30 12:51:23.538732411 +0000 UTC m=+378.585866364"
Oct 30 12:51:26.911101 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:26.911053    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-cbd4cb4f6-fl8vc"]
Oct 30 12:51:26.914786 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:26.914753    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-cbd4cb4f6-fl8vc"
Oct 30 12:51:26.922682 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:26.922642    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:26.929998 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:26.929970    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 12:51:26.930153 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:26.930015    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 12:51:26.941922 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:26.941889    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 12:51:26.941922 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:26.941889    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 12:51:26.942155 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:26.942074    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xgj4z\""
Oct 30 12:51:26.943245 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:26.943204    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-cbd4cb4f6-fl8vc"]
Oct 30 12:51:27.017548 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:27.017507    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj7b4\" (UniqueName: \"kubernetes.io/projected/3eb6d5b4-6fbb-4ae8-9359-833f745cf6c2-kube-api-access-zj7b4\") pod \"kube-auth-proxy-cbd4cb4f6-fl8vc\" (UID: \"3eb6d5b4-6fbb-4ae8-9359-833f745cf6c2\") " pod="openshift-ingress/kube-auth-proxy-cbd4cb4f6-fl8vc"
Oct 30 12:51:27.017548 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:27.017549    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3eb6d5b4-6fbb-4ae8-9359-833f745cf6c2-tmp\") pod \"kube-auth-proxy-cbd4cb4f6-fl8vc\" (UID: \"3eb6d5b4-6fbb-4ae8-9359-833f745cf6c2\") " pod="openshift-ingress/kube-auth-proxy-cbd4cb4f6-fl8vc"
Oct 30 12:51:27.017816 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:27.017723    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb6d5b4-6fbb-4ae8-9359-833f745cf6c2-tls-certs\") pod \"kube-auth-proxy-cbd4cb4f6-fl8vc\" (UID: \"3eb6d5b4-6fbb-4ae8-9359-833f745cf6c2\") " pod="openshift-ingress/kube-auth-proxy-cbd4cb4f6-fl8vc"
Oct 30 12:51:27.056589 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:27.056558    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:27.118626 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:27.118584    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj7b4\" (UniqueName: \"kubernetes.io/projected/3eb6d5b4-6fbb-4ae8-9359-833f745cf6c2-kube-api-access-zj7b4\") pod \"kube-auth-proxy-cbd4cb4f6-fl8vc\" (UID: \"3eb6d5b4-6fbb-4ae8-9359-833f745cf6c2\") " pod="openshift-ingress/kube-auth-proxy-cbd4cb4f6-fl8vc"
Oct 30 12:51:27.118825 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:27.118631    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3eb6d5b4-6fbb-4ae8-9359-833f745cf6c2-tmp\") pod \"kube-auth-proxy-cbd4cb4f6-fl8vc\" (UID: \"3eb6d5b4-6fbb-4ae8-9359-833f745cf6c2\") " pod="openshift-ingress/kube-auth-proxy-cbd4cb4f6-fl8vc"
Oct 30 12:51:27.118825 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:27.118708    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb6d5b4-6fbb-4ae8-9359-833f745cf6c2-tls-certs\") pod \"kube-auth-proxy-cbd4cb4f6-fl8vc\" (UID: \"3eb6d5b4-6fbb-4ae8-9359-833f745cf6c2\") " pod="openshift-ingress/kube-auth-proxy-cbd4cb4f6-fl8vc"
Oct 30 12:51:27.121029 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:27.121000    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3eb6d5b4-6fbb-4ae8-9359-833f745cf6c2-tmp\") pod \"kube-auth-proxy-cbd4cb4f6-fl8vc\" (UID: \"3eb6d5b4-6fbb-4ae8-9359-833f745cf6c2\") " pod="openshift-ingress/kube-auth-proxy-cbd4cb4f6-fl8vc"
Oct 30 12:51:27.121159 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:27.121114    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb6d5b4-6fbb-4ae8-9359-833f745cf6c2-tls-certs\") pod \"kube-auth-proxy-cbd4cb4f6-fl8vc\" (UID: \"3eb6d5b4-6fbb-4ae8-9359-833f745cf6c2\") " pod="openshift-ingress/kube-auth-proxy-cbd4cb4f6-fl8vc"
Oct 30 12:51:27.149844 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:27.149808    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj7b4\" (UniqueName: \"kubernetes.io/projected/3eb6d5b4-6fbb-4ae8-9359-833f745cf6c2-kube-api-access-zj7b4\") pod \"kube-auth-proxy-cbd4cb4f6-fl8vc\" (UID: \"3eb6d5b4-6fbb-4ae8-9359-833f745cf6c2\") " pod="openshift-ingress/kube-auth-proxy-cbd4cb4f6-fl8vc"
Oct 30 12:51:27.229618 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:27.229541    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-cbd4cb4f6-fl8vc"
Oct 30 12:51:27.526703 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:27.526627    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-cbd4cb4f6-fl8vc"]
Oct 30 12:51:27.531604 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:51:27.531580    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb6d5b4_6fbb_4ae8_9359_833f745cf6c2.slice/crio-8fe267ffa9075c0a3999a5b22f852208ab517f06212089f1109d0b0a658573b7 WatchSource:0}: Error finding container 8fe267ffa9075c0a3999a5b22f852208ab517f06212089f1109d0b0a658573b7: Status 404 returned error can't find the container with id 8fe267ffa9075c0a3999a5b22f852208ab517f06212089f1109d0b0a658573b7
Oct 30 12:51:28.526277 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:28.526230    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-cbd4cb4f6-fl8vc" event={"ID":"3eb6d5b4-6fbb-4ae8-9359-833f745cf6c2","Type":"ContainerStarted","Data":"8fe267ffa9075c0a3999a5b22f852208ab517f06212089f1109d0b0a658573b7"}
Oct 30 12:51:29.011689 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:51:29.011648    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 12:51:29.083924 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:29.083889    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5"]
Oct 30 12:51:29.092385 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:29.092356    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5"
Oct 30 12:51:29.112147 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:29.112112    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 12:51:29.129320 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:29.129271    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5"]
Oct 30 12:51:29.130516 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:29.130481    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 12:51:29.130648 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:29.130551    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 12:51:29.130648 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:29.130618    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 12:51:29.132258 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:29.132215    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jj6x8\""
Oct 30 12:51:29.135556 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:29.135533    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:29.136590 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:29.136571    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prbjp\" (UniqueName: \"kubernetes.io/projected/d81a90a1-b544-43df-b61a-4547d69832f3-kube-api-access-prbjp\") pod \"istiod-openshift-gateway-7876bfcb74-bvss5\" (UID: \"d81a90a1-b544-43df-b61a-4547d69832f3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5"
Oct 30 12:51:29.136676 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:29.136601    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d81a90a1-b544-43df-b61a-4547d69832f3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-bvss5\" (UID: \"d81a90a1-b544-43df-b61a-4547d69832f3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5"
Oct 30 12:51:29.136676 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:29.136623    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d81a90a1-b544-43df-b61a-4547d69832f3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-bvss5\" (UID: \"d81a90a1-b544-43df-b61a-4547d69832f3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5"
Oct 30 12:51:29.136676 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:29.136641    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81a90a1-b544-43df-b61a-4547d69832f3-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-bvss5\" (UID: \"d81a90a1-b544-43df-b61a-4547d69832f3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5"
Oct 30 12:51:29.136792 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:29.136701    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d81a90a1-b544-43df-b61a-4547d69832f3-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-bvss5\" (UID: \"d81a90a1-b544-43df-b61a-4547d69832f3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5"
Oct 30 12:51:29.136830 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:29.136811    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d81a90a1-b544-43df-b61a-4547d69832f3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-bvss5\" (UID: \"d81a90a1-b544-43df-b61a-4547d69832f3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5"
Oct 30 12:51:29.136869 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:29.136859    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81a90a1-b544-43df-b61a-4547d69832f3-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-bvss5\" (UID: \"d81a90a1-b544-43df-b61a-4547d69832f3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5"
Oct 30 12:51:29.237521 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:29.237485    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81a90a1-b544-43df-b61a-4547d69832f3-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-bvss5\" (UID: \"d81a90a1-b544-43df-b61a-4547d69832f3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5"
Oct 30 12:51:29.237721 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:29.237541    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prbjp\" (UniqueName: \"kubernetes.io/projected/d81a90a1-b544-43df-b61a-4547d69832f3-kube-api-access-prbjp\") pod \"istiod-openshift-gateway-7876bfcb74-bvss5\" (UID: \"d81a90a1-b544-43df-b61a-4547d69832f3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5"
Oct 30 12:51:29.237721 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:29.237582    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d81a90a1-b544-43df-b61a-4547d69832f3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-bvss5\" (UID: \"d81a90a1-b544-43df-b61a-4547d69832f3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5"
Oct 30 12:51:29.237721 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:29.237615    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d81a90a1-b544-43df-b61a-4547d69832f3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-bvss5\" (UID: \"d81a90a1-b544-43df-b61a-4547d69832f3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5"
Oct 30 12:51:29.237721 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:29.237645    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81a90a1-b544-43df-b61a-4547d69832f3-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-bvss5\" (UID: \"d81a90a1-b544-43df-b61a-4547d69832f3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5"
Oct 30 12:51:29.237721 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:29.237673    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d81a90a1-b544-43df-b61a-4547d69832f3-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-bvss5\" (UID: \"d81a90a1-b544-43df-b61a-4547d69832f3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5"
Oct 30 12:51:29.237721 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:29.237707    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d81a90a1-b544-43df-b61a-4547d69832f3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-bvss5\" (UID: \"d81a90a1-b544-43df-b61a-4547d69832f3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5"
Oct 30 12:51:29.238589 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:29.238563    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d81a90a1-b544-43df-b61a-4547d69832f3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-bvss5\" (UID: \"d81a90a1-b544-43df-b61a-4547d69832f3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5"
Oct 30 12:51:29.240560 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:29.240533    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81a90a1-b544-43df-b61a-4547d69832f3-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-bvss5\" (UID: \"d81a90a1-b544-43df-b61a-4547d69832f3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5"
Oct 30 12:51:29.240704 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:29.240566    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d81a90a1-b544-43df-b61a-4547d69832f3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-bvss5\" (UID: \"d81a90a1-b544-43df-b61a-4547d69832f3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5"
Oct 30 12:51:29.240704 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:29.240597    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d81a90a1-b544-43df-b61a-4547d69832f3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-bvss5\" (UID: \"d81a90a1-b544-43df-b61a-4547d69832f3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5"
Oct 30 12:51:29.241269 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:29.241245    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d81a90a1-b544-43df-b61a-4547d69832f3-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-bvss5\" (UID: \"d81a90a1-b544-43df-b61a-4547d69832f3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5"
Oct 30 12:51:29.274711 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:29.274626    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prbjp\" (UniqueName: \"kubernetes.io/projected/d81a90a1-b544-43df-b61a-4547d69832f3-kube-api-access-prbjp\") pod \"istiod-openshift-gateway-7876bfcb74-bvss5\" (UID: \"d81a90a1-b544-43df-b61a-4547d69832f3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5"
Oct 30 12:51:29.292179 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:29.292148    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81a90a1-b544-43df-b61a-4547d69832f3-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-bvss5\" (UID: \"d81a90a1-b544-43df-b61a-4547d69832f3\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5"
Oct 30 12:51:29.406803 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:29.406763    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5"
Oct 30 12:51:29.491430 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:29.491394    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:29.744654 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:29.744615    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5"]
Oct 30 12:51:29.748870 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:51:29.748838    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd81a90a1_b544_43df_b61a_4547d69832f3.slice/crio-302b24cd8fe2f893199119d016b5ff1514c421960005f537321647a9e2413671 WatchSource:0}: Error finding container 302b24cd8fe2f893199119d016b5ff1514c421960005f537321647a9e2413671: Status 404 returned error can't find the container with id 302b24cd8fe2f893199119d016b5ff1514c421960005f537321647a9e2413671
Oct 30 12:51:30.537736 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:30.537690    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-cbd4cb4f6-fl8vc" event={"ID":"3eb6d5b4-6fbb-4ae8-9359-833f745cf6c2","Type":"ContainerStarted","Data":"9e15c3a2de27d0ece6423f67843fb59c474463699100b9f6ce74285e5818056d"}
Oct 30 12:51:30.539303 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:30.539268    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5" event={"ID":"d81a90a1-b544-43df-b61a-4547d69832f3","Type":"ContainerStarted","Data":"302b24cd8fe2f893199119d016b5ff1514c421960005f537321647a9e2413671"}
Oct 30 12:51:30.589198 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:30.589137    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-cbd4cb4f6-fl8vc" podStartSLOduration=2.134033158 podStartE2EDuration="4.589114402s" totalImagesPullingTime="2.455081244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:30.588441411 +0000 UTC m=+385.635575365" watchObservedRunningTime="2026-10-30 12:51:30.589114402 +0000 UTC m=+385.636248359"
Oct 30 12:51:30.793474 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:30.793387    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:31.785688 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:31.785654    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:32.003353 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:32.003311    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:32.260778 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:32.260738    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:51:32.260919 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:32.260822    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:51:32.549180 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:32.549085    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5" event={"ID":"d81a90a1-b544-43df-b61a-4547d69832f3","Type":"ContainerStarted","Data":"854b83e113691877f2700ca17283d90ce45ba8444de11aa15b0ef83ec3f90787"}
Oct 30 12:51:32.549363 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:32.549250    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5"
Oct 30 12:51:32.618388 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:32.618323    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5" podStartSLOduration=1.109519011 podStartE2EDuration="3.618303224s" totalImagesPullingTime="2.508784213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:32.605429587 +0000 UTC m=+387.652563537" watchObservedRunningTime="2026-10-30 12:51:32.618303224 +0000 UTC m=+387.665437180"
Oct 30 12:51:33.070204 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:33.070158    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:33.249663 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:33.249625    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:33.555189 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:33.555157    4544 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-bvss5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:51:33.555382 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:33.555209    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5" podUID="d81a90a1-b544-43df-b61a-4547d69832f3" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:51:34.096728 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:34.096670    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:34.227468 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:34.227439    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:34.501491 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:34.501410    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-observability-584f99c788-rvsjg"
Oct 30 12:51:34.703362 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:34.703324    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:36.554754 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:36.554721    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5"
Oct 30 12:51:51.762259 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:51:51.762140    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:59.006373 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:51:59.006332    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 12:52:19.289922 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:19.289881    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dlbtt"]
Oct 30 12:52:19.293703 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:19.293679    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dlbtt"
Oct 30 12:52:19.368285 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:19.368247    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dlbtt"]
Oct 30 12:52:19.467413 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:19.467374    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1e8d89e6-3301-45d8-8e9d-4629f6bad974-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dlbtt\" (UID: \"1e8d89e6-3301-45d8-8e9d-4629f6bad974\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dlbtt"
Oct 30 12:52:19.467619 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:19.467432    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qscb8\" (UniqueName: \"kubernetes.io/projected/1e8d89e6-3301-45d8-8e9d-4629f6bad974-kube-api-access-qscb8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dlbtt\" (UID: \"1e8d89e6-3301-45d8-8e9d-4629f6bad974\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dlbtt"
Oct 30 12:52:19.467619 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:19.467557    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e8d89e6-3301-45d8-8e9d-4629f6bad974-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dlbtt\" (UID: \"1e8d89e6-3301-45d8-8e9d-4629f6bad974\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dlbtt"
Oct 30 12:52:19.467702 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:19.467622    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1e8d89e6-3301-45d8-8e9d-4629f6bad974-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dlbtt\" (UID: \"1e8d89e6-3301-45d8-8e9d-4629f6bad974\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dlbtt"
Oct 30 12:52:19.467741 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:19.467711    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e8d89e6-3301-45d8-8e9d-4629f6bad974-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dlbtt\" (UID: \"1e8d89e6-3301-45d8-8e9d-4629f6bad974\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dlbtt"
Oct 30 12:52:19.467741 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:19.467735    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1e8d89e6-3301-45d8-8e9d-4629f6bad974-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dlbtt\" (UID: \"1e8d89e6-3301-45d8-8e9d-4629f6bad974\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dlbtt"
Oct 30 12:52:19.467802 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:19.467758    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1e8d89e6-3301-45d8-8e9d-4629f6bad974-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dlbtt\" (UID: \"1e8d89e6-3301-45d8-8e9d-4629f6bad974\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dlbtt"
Oct 30 12:52:19.568364 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:19.568261    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e8d89e6-3301-45d8-8e9d-4629f6bad974-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dlbtt\" (UID: \"1e8d89e6-3301-45d8-8e9d-4629f6bad974\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dlbtt"
Oct 30 12:52:19.568364 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:19.568308    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1e8d89e6-3301-45d8-8e9d-4629f6bad974-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dlbtt\" (UID: \"1e8d89e6-3301-45d8-8e9d-4629f6bad974\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dlbtt"
Oct 30 12:52:19.568364 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:19.568335    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1e8d89e6-3301-45d8-8e9d-4629f6bad974-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dlbtt\" (UID: \"1e8d89e6-3301-45d8-8e9d-4629f6bad974\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dlbtt"
Oct 30 12:52:19.568650 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:19.568370    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1e8d89e6-3301-45d8-8e9d-4629f6bad974-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dlbtt\" (UID: \"1e8d89e6-3301-45d8-8e9d-4629f6bad974\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dlbtt"
Oct 30 12:52:19.568650 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:19.568426    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qscb8\" (UniqueName: \"kubernetes.io/projected/1e8d89e6-3301-45d8-8e9d-4629f6bad974-kube-api-access-qscb8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dlbtt\" (UID: \"1e8d89e6-3301-45d8-8e9d-4629f6bad974\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dlbtt"
Oct 30 12:52:19.568650 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:19.568620    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e8d89e6-3301-45d8-8e9d-4629f6bad974-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dlbtt\" (UID: \"1e8d89e6-3301-45d8-8e9d-4629f6bad974\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dlbtt"
Oct 30 12:52:19.568796 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:19.568667    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1e8d89e6-3301-45d8-8e9d-4629f6bad974-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dlbtt\" (UID: \"1e8d89e6-3301-45d8-8e9d-4629f6bad974\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dlbtt"
Oct 30 12:52:19.569269 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:19.569247    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1e8d89e6-3301-45d8-8e9d-4629f6bad974-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dlbtt\" (UID: \"1e8d89e6-3301-45d8-8e9d-4629f6bad974\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dlbtt"
Oct 30 12:52:19.571016 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:19.570988    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1e8d89e6-3301-45d8-8e9d-4629f6bad974-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dlbtt\" (UID: \"1e8d89e6-3301-45d8-8e9d-4629f6bad974\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dlbtt"
Oct 30 12:52:19.571139 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:19.571059    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e8d89e6-3301-45d8-8e9d-4629f6bad974-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dlbtt\" (UID: \"1e8d89e6-3301-45d8-8e9d-4629f6bad974\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dlbtt"
Oct 30 12:52:19.571274 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:19.571257    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1e8d89e6-3301-45d8-8e9d-4629f6bad974-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dlbtt\" (UID: \"1e8d89e6-3301-45d8-8e9d-4629f6bad974\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dlbtt"
Oct 30 12:52:19.571342 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:19.571281    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1e8d89e6-3301-45d8-8e9d-4629f6bad974-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dlbtt\" (UID: \"1e8d89e6-3301-45d8-8e9d-4629f6bad974\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dlbtt"
Oct 30 12:52:19.595498 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:19.595466    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e8d89e6-3301-45d8-8e9d-4629f6bad974-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dlbtt\" (UID: \"1e8d89e6-3301-45d8-8e9d-4629f6bad974\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dlbtt"
Oct 30 12:52:19.595767 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:19.595743    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qscb8\" (UniqueName: \"kubernetes.io/projected/1e8d89e6-3301-45d8-8e9d-4629f6bad974-kube-api-access-qscb8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dlbtt\" (UID: \"1e8d89e6-3301-45d8-8e9d-4629f6bad974\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dlbtt"
Oct 30 12:52:19.605372 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:19.605342    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dlbtt"
Oct 30 12:52:19.928031 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:19.927995    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dlbtt"]
Oct 30 12:52:19.932808 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:52:19.932775    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e8d89e6_3301_45d8_8e9d_4629f6bad974.slice/crio-6379ab8c7924b87b5f971104f1c5ae5e8002b33563964de8129f957f2b49b5a1 WatchSource:0}: Error finding container 6379ab8c7924b87b5f971104f1c5ae5e8002b33563964de8129f957f2b49b5a1: Status 404 returned error can't find the container with id 6379ab8c7924b87b5f971104f1c5ae5e8002b33563964de8129f957f2b49b5a1
Oct 30 12:52:19.934987 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:19.934946    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:52:19.935119 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:19.935017    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:52:20.444657 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:20.444623    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:20.594488 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:20.594452    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:20.724372 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:20.724284    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:20.740622 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:20.740561    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dlbtt" event={"ID":"1e8d89e6-3301-45d8-8e9d-4629f6bad974","Type":"ContainerStarted","Data":"ec42282f635fc5ccad397da76c64f68c15e4e61c147f149c9211bc56eb5f3aee"}
Oct 30 12:52:20.741564 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:20.740641    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dlbtt" event={"ID":"1e8d89e6-3301-45d8-8e9d-4629f6bad974","Type":"ContainerStarted","Data":"6379ab8c7924b87b5f971104f1c5ae5e8002b33563964de8129f957f2b49b5a1"}
Oct 30 12:52:20.741564 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:20.740948    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dlbtt"
Oct 30 12:52:20.742832 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:20.742803    4544 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-dlbtt container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:52:20.742974 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:20.742857    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dlbtt" podUID="1e8d89e6-3301-45d8-8e9d-4629f6bad974" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:52:20.804127 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:20.804070    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dlbtt" podStartSLOduration=1.804050251 podStartE2EDuration="1.804050251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:20.79851258 +0000 UTC m=+435.845646534" watchObservedRunningTime="2026-10-30 12:52:20.804050251 +0000 UTC m=+435.851184204"
Oct 30 12:52:20.862174 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:20.862132    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:20.979184 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:20.979076    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:21.301241 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:21.301082    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:21.745394 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:21.745359    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dlbtt"
Oct 30 12:52:21.838698 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:21.838580    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5"]
Oct 30 12:52:21.838915 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:21.838799    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5" podUID="d81a90a1-b544-43df-b61a-4547d69832f3" containerName="discovery" containerID="cri-o://854b83e113691877f2700ca17283d90ce45ba8444de11aa15b0ef83ec3f90787" gracePeriod=30
Oct 30 12:52:22.089307 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:22.089265    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5"
Oct 30 12:52:22.190015 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:22.189978    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d81a90a1-b544-43df-b61a-4547d69832f3-kube-api-access-prbjp\" (UniqueName: \"kubernetes.io/projected/d81a90a1-b544-43df-b61a-4547d69832f3-kube-api-access-prbjp\") pod \"d81a90a1-b544-43df-b61a-4547d69832f3\" (UID: \"d81a90a1-b544-43df-b61a-4547d69832f3\") "
Oct 30 12:52:22.190191 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:22.190023    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d81a90a1-b544-43df-b61a-4547d69832f3-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d81a90a1-b544-43df-b61a-4547d69832f3-istio-kubeconfig\") pod \"d81a90a1-b544-43df-b61a-4547d69832f3\" (UID: \"d81a90a1-b544-43df-b61a-4547d69832f3\") "
Oct 30 12:52:22.190191 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:22.190047    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d81a90a1-b544-43df-b61a-4547d69832f3-cacerts\" (UniqueName: \"kubernetes.io/secret/d81a90a1-b544-43df-b61a-4547d69832f3-cacerts\") pod \"d81a90a1-b544-43df-b61a-4547d69832f3\" (UID: \"d81a90a1-b544-43df-b61a-4547d69832f3\") "
Oct 30 12:52:22.190191 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:22.190074    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d81a90a1-b544-43df-b61a-4547d69832f3-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d81a90a1-b544-43df-b61a-4547d69832f3-istio-csr-dns-cert\") pod \"d81a90a1-b544-43df-b61a-4547d69832f3\" (UID: \"d81a90a1-b544-43df-b61a-4547d69832f3\") "
Oct 30 12:52:22.190397 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:22.190178    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d81a90a1-b544-43df-b61a-4547d69832f3-istio-token\" (UniqueName: \"kubernetes.io/projected/d81a90a1-b544-43df-b61a-4547d69832f3-istio-token\") pod \"d81a90a1-b544-43df-b61a-4547d69832f3\" (UID: \"d81a90a1-b544-43df-b61a-4547d69832f3\") "
Oct 30 12:52:22.190397 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:22.190290    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d81a90a1-b544-43df-b61a-4547d69832f3-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d81a90a1-b544-43df-b61a-4547d69832f3-istio-csr-ca-configmap\") pod \"d81a90a1-b544-43df-b61a-4547d69832f3\" (UID: \"d81a90a1-b544-43df-b61a-4547d69832f3\") "
Oct 30 12:52:22.190397 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:22.190337    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d81a90a1-b544-43df-b61a-4547d69832f3-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81a90a1-b544-43df-b61a-4547d69832f3-local-certs\") pod \"d81a90a1-b544-43df-b61a-4547d69832f3\" (UID: \"d81a90a1-b544-43df-b61a-4547d69832f3\") "
Oct 30 12:52:22.190754 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:22.190703    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d81a90a1-b544-43df-b61a-4547d69832f3-istio-csr-ca-configmap" pod "d81a90a1-b544-43df-b61a-4547d69832f3" (UID: "d81a90a1-b544-43df-b61a-4547d69832f3"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:52:22.193035 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:22.192977    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81a90a1-b544-43df-b61a-4547d69832f3-local-certs" pod "d81a90a1-b544-43df-b61a-4547d69832f3" (UID: "d81a90a1-b544-43df-b61a-4547d69832f3"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:52:22.193151 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:22.193067    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d81a90a1-b544-43df-b61a-4547d69832f3-istio-token" pod "d81a90a1-b544-43df-b61a-4547d69832f3" (UID: "d81a90a1-b544-43df-b61a-4547d69832f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:52:22.193151 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:22.193086    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d81a90a1-b544-43df-b61a-4547d69832f3-cacerts" pod "d81a90a1-b544-43df-b61a-4547d69832f3" (UID: "d81a90a1-b544-43df-b61a-4547d69832f3"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:52:22.193151 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:22.193098    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d81a90a1-b544-43df-b61a-4547d69832f3-istio-kubeconfig" pod "d81a90a1-b544-43df-b61a-4547d69832f3" (UID: "d81a90a1-b544-43df-b61a-4547d69832f3"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:52:22.193151 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:22.193123    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d81a90a1-b544-43df-b61a-4547d69832f3-istio-csr-dns-cert" pod "d81a90a1-b544-43df-b61a-4547d69832f3" (UID: "d81a90a1-b544-43df-b61a-4547d69832f3"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:52:22.193372 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:22.193161    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d81a90a1-b544-43df-b61a-4547d69832f3-kube-api-access-prbjp" pod "d81a90a1-b544-43df-b61a-4547d69832f3" (UID: "d81a90a1-b544-43df-b61a-4547d69832f3"). InnerVolumeSpecName "kube-api-access-prbjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:52:22.291174 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:22.291084    4544 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d81a90a1-b544-43df-b61a-4547d69832f3-istio-csr-ca-configmap\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:52:22.291174 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:22.291118    4544 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81a90a1-b544-43df-b61a-4547d69832f3-local-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:52:22.291174 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:22.291128    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prbjp\" (UniqueName: \"kubernetes.io/projected/d81a90a1-b544-43df-b61a-4547d69832f3-kube-api-access-prbjp\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:52:22.291174 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:22.291137    4544 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d81a90a1-b544-43df-b61a-4547d69832f3-istio-kubeconfig\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:52:22.291174 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:22.291146    4544 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d81a90a1-b544-43df-b61a-4547d69832f3-cacerts\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:52:22.291174 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:22.291155    4544 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d81a90a1-b544-43df-b61a-4547d69832f3-istio-csr-dns-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:52:22.291174 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:22.291162    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81a90a1-b544-43df-b61a-4547d69832f3-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:52:22.749732 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:22.749698    4544 generic.go:356] "Generic (PLEG): container finished" podID="d81a90a1-b544-43df-b61a-4547d69832f3" containerID="854b83e113691877f2700ca17283d90ce45ba8444de11aa15b0ef83ec3f90787" exitCode=0
Oct 30 12:52:22.750167 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:22.749754    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5"
Oct 30 12:52:22.750167 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:22.749780    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5" event={"ID":"d81a90a1-b544-43df-b61a-4547d69832f3","Type":"ContainerDied","Data":"854b83e113691877f2700ca17283d90ce45ba8444de11aa15b0ef83ec3f90787"}
Oct 30 12:52:22.750167 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:22.749838    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5" event={"ID":"d81a90a1-b544-43df-b61a-4547d69832f3","Type":"ContainerDied","Data":"302b24cd8fe2f893199119d016b5ff1514c421960005f537321647a9e2413671"}
Oct 30 12:52:22.750167 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:22.749862    4544 scope.go:122] "RemoveContainer" containerID="854b83e113691877f2700ca17283d90ce45ba8444de11aa15b0ef83ec3f90787"
Oct 30 12:52:22.759150 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:22.759132    4544 scope.go:122] "RemoveContainer" containerID="854b83e113691877f2700ca17283d90ce45ba8444de11aa15b0ef83ec3f90787"
Oct 30 12:52:22.759616 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:52:22.759594    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"854b83e113691877f2700ca17283d90ce45ba8444de11aa15b0ef83ec3f90787\": container with ID starting with 854b83e113691877f2700ca17283d90ce45ba8444de11aa15b0ef83ec3f90787 not found: ID does not exist" containerID="854b83e113691877f2700ca17283d90ce45ba8444de11aa15b0ef83ec3f90787"
Oct 30 12:52:22.759715 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:22.759625    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"854b83e113691877f2700ca17283d90ce45ba8444de11aa15b0ef83ec3f90787"} err="failed to get container status \"854b83e113691877f2700ca17283d90ce45ba8444de11aa15b0ef83ec3f90787\": rpc error: code = NotFound desc = could not find container \"854b83e113691877f2700ca17283d90ce45ba8444de11aa15b0ef83ec3f90787\": container with ID starting with 854b83e113691877f2700ca17283d90ce45ba8444de11aa15b0ef83ec3f90787 not found: ID does not exist"
Oct 30 12:52:22.805126 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:22.805078    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5"]
Oct 30 12:52:22.817002 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:22.816972    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5"]
Oct 30 12:52:23.082810 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:23.082775    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d81a90a1-b544-43df-b61a-4547d69832f3" path="/var/lib/kubelet/pods/d81a90a1-b544-43df-b61a-4547d69832f3/volumes"
Oct 30 12:52:23.233979 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:23.233945    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:23.342090 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:23.342014    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:23.455063 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:23.455027    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:23.578032 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:23.577996    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:23.692620 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:23.692524    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:29.006047 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:52:29.006008    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 12:52:51.897171 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:52:51.897135    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:52:59.006232 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:52:59.006179    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 12:53:05.636438 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:05.636402    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:08.669208 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:08.669168    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:08.794420 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:08.794380    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:09.802637 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:09.802600    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:09.912617 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:09.912585    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:20.369919 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:20.369837    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:22.401997 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:22.401960    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-n6qxn"]
Oct 30 12:53:22.402481 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:22.402284    4544 cpu_manager.go:414] "containerMap: removing container" podUID="d81a90a1-b544-43df-b61a-4547d69832f3" containerName="discovery"
Oct 30 12:53:22.402481 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:22.402296    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d81a90a1-b544-43df-b61a-4547d69832f3" containerName="discovery"
Oct 30 12:53:22.402481 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:22.402349    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="d81a90a1-b544-43df-b61a-4547d69832f3" containerName="discovery"
Oct 30 12:53:22.405238 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:22.405207    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-n6qxn"
Oct 30 12:53:22.438093 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:22.438061    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 30 12:53:22.438093 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:22.438066    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 12:53:22.438093 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:22.438075    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-zhblc\""
Oct 30 12:53:22.438403 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:22.438148    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 12:53:22.439660 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:22.439634    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-n6qxn"]
Oct 30 12:53:22.491440 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:22.491404    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkc7c\" (UniqueName: \"kubernetes.io/projected/a7c27cba-095a-446a-bf7c-acab18ce8bca-kube-api-access-hkc7c\") pod \"dns-operator-controller-manager-65bfd79598-n6qxn\" (UID: \"a7c27cba-095a-446a-bf7c-acab18ce8bca\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-n6qxn"
Oct 30 12:53:22.591947 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:22.591916    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkc7c\" (UniqueName: \"kubernetes.io/projected/a7c27cba-095a-446a-bf7c-acab18ce8bca-kube-api-access-hkc7c\") pod \"dns-operator-controller-manager-65bfd79598-n6qxn\" (UID: \"a7c27cba-095a-446a-bf7c-acab18ce8bca\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-n6qxn"
Oct 30 12:53:22.615236 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:22.615197    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkc7c\" (UniqueName: \"kubernetes.io/projected/a7c27cba-095a-446a-bf7c-acab18ce8bca-kube-api-access-hkc7c\") pod \"dns-operator-controller-manager-65bfd79598-n6qxn\" (UID: \"a7c27cba-095a-446a-bf7c-acab18ce8bca\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-n6qxn"
Oct 30 12:53:22.717536 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:22.717452    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-n6qxn"
Oct 30 12:53:22.765410 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:22.765373    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:23.015208 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:23.015097    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-n6qxn"]
Oct 30 12:53:23.021123 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:53:23.021093    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7c27cba_095a_446a_bf7c_acab18ce8bca.slice/crio-355b9350fb919ded5dc5c7317d44da9cfc5c06ee4db9658d5a69125a0ab0fe1e WatchSource:0}: Error finding container 355b9350fb919ded5dc5c7317d44da9cfc5c06ee4db9658d5a69125a0ab0fe1e: Status 404 returned error can't find the container with id 355b9350fb919ded5dc5c7317d44da9cfc5c06ee4db9658d5a69125a0ab0fe1e
Oct 30 12:53:23.170916 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:23.170874    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:24.005200 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:24.005124    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-n6qxn" event={"ID":"a7c27cba-095a-446a-bf7c-acab18ce8bca","Type":"ContainerStarted","Data":"355b9350fb919ded5dc5c7317d44da9cfc5c06ee4db9658d5a69125a0ab0fe1e"}
Oct 30 12:53:24.649924 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:24.649886    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:24.787972 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:24.787936    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:24.914677 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:24.914602    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-nlk4n"]
Oct 30 12:53:24.924956 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:24.924925    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-nlk4n"
Oct 30 12:53:24.942173 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:24.942150    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-nlk4n"]
Oct 30 12:53:24.952031 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:24.952002    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-q5m7f\""
Oct 30 12:53:25.010080 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:25.009935    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h85wx\" (UniqueName: \"kubernetes.io/projected/8714d903-8aa8-4b34-ab60-943c82d814c3-kube-api-access-h85wx\") pod \"authorino-operator-68d6dd76dc-nlk4n\" (UID: \"8714d903-8aa8-4b34-ab60-943c82d814c3\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-nlk4n"
Oct 30 12:53:25.111487 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:25.111451    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h85wx\" (UniqueName: \"kubernetes.io/projected/8714d903-8aa8-4b34-ab60-943c82d814c3-kube-api-access-h85wx\") pod \"authorino-operator-68d6dd76dc-nlk4n\" (UID: \"8714d903-8aa8-4b34-ab60-943c82d814c3\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-nlk4n"
Oct 30 12:53:25.137239 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:25.137202    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h85wx\" (UniqueName: \"kubernetes.io/projected/8714d903-8aa8-4b34-ab60-943c82d814c3-kube-api-access-h85wx\") pod \"authorino-operator-68d6dd76dc-nlk4n\" (UID: \"8714d903-8aa8-4b34-ab60-943c82d814c3\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-nlk4n"
Oct 30 12:53:25.240891 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:25.240811    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-nlk4n"
Oct 30 12:53:25.554001 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:25.553973    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-nlk4n"]
Oct 30 12:53:25.562358 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:53:25.562324    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8714d903_8aa8_4b34_ab60_943c82d814c3.slice/crio-2ea6da59efa05f3047567a492817c4a6c02e0b8abf2bc0726bb1aa8bb6f14479 WatchSource:0}: Error finding container 2ea6da59efa05f3047567a492817c4a6c02e0b8abf2bc0726bb1aa8bb6f14479: Status 404 returned error can't find the container with id 2ea6da59efa05f3047567a492817c4a6c02e0b8abf2bc0726bb1aa8bb6f14479
Oct 30 12:53:25.833938 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:25.833907    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:26.015038 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:26.015001    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-nlk4n" event={"ID":"8714d903-8aa8-4b34-ab60-943c82d814c3","Type":"ContainerStarted","Data":"2ea6da59efa05f3047567a492817c4a6c02e0b8abf2bc0726bb1aa8bb6f14479"}
Oct 30 12:53:26.016666 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:26.016642    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-n6qxn" event={"ID":"a7c27cba-095a-446a-bf7c-acab18ce8bca","Type":"ContainerStarted","Data":"9d2bde37f639984ebf2d92a9dece58bd8ec395eb1218e27b5b80603faac684f6"}
Oct 30 12:53:26.016790 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:26.016778    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-n6qxn"
Oct 30 12:53:26.039450 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:26.039415    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:26.064854 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:26.064811    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-n6qxn" podStartSLOduration=1.554101703 podStartE2EDuration="4.064798926s" totalImagesPullingTime="2.510697223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:26.063966539 +0000 UTC m=+501.111100491" watchObservedRunningTime="2026-10-30 12:53:26.064798926 +0000 UTC m=+501.111932878"
Oct 30 12:53:27.415258 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:27.415206    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-7bqsq"]
Oct 30 12:53:27.419001 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:27.418978    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7bqsq"
Oct 30 12:53:27.429871 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:27.429848    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbdwj\" (UniqueName: \"kubernetes.io/projected/79f27b5e-4f24-4970-87fb-d6d819190912-kube-api-access-fbdwj\") pod \"limitador-operator-controller-manager-856dd584c5-7bqsq\" (UID: \"79f27b5e-4f24-4970-87fb-d6d819190912\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7bqsq"
Oct 30 12:53:27.432831 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:27.432805    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:27.447032 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:27.446995    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-6t7xh\""
Oct 30 12:53:27.450001 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:27.449980    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-7bqsq"]
Oct 30 12:53:27.530362 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:27.530320    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbdwj\" (UniqueName: \"kubernetes.io/projected/79f27b5e-4f24-4970-87fb-d6d819190912-kube-api-access-fbdwj\") pod \"limitador-operator-controller-manager-856dd584c5-7bqsq\" (UID: \"79f27b5e-4f24-4970-87fb-d6d819190912\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7bqsq"
Oct 30 12:53:27.560296 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:27.560271    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbdwj\" (UniqueName: \"kubernetes.io/projected/79f27b5e-4f24-4970-87fb-d6d819190912-kube-api-access-fbdwj\") pod \"limitador-operator-controller-manager-856dd584c5-7bqsq\" (UID: \"79f27b5e-4f24-4970-87fb-d6d819190912\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7bqsq"
Oct 30 12:53:27.735901 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:27.735826    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7bqsq"
Oct 30 12:53:27.992197 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:27.992126    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:28.026621 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:28.026579    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-nlk4n" event={"ID":"8714d903-8aa8-4b34-ab60-943c82d814c3","Type":"ContainerStarted","Data":"62687a51a87a8a8b283ac5ab918e22c7de9a538cad633b64cc4dce3300331451"}
Oct 30 12:53:28.026805 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:28.026680    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-nlk4n"
Oct 30 12:53:28.032267 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:28.032237    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-7bqsq"]
Oct 30 12:53:28.036908 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:53:28.036886    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f27b5e_4f24_4970_87fb_d6d819190912.slice/crio-2fffb51c9faeab0cffd1f498f5e2212166382d891ff5f172a7569d8c7d3267b2 WatchSource:0}: Error finding container 2fffb51c9faeab0cffd1f498f5e2212166382d891ff5f172a7569d8c7d3267b2: Status 404 returned error can't find the container with id 2fffb51c9faeab0cffd1f498f5e2212166382d891ff5f172a7569d8c7d3267b2
Oct 30 12:53:28.072475 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:28.072422    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d6dd76dc-nlk4n" podStartSLOduration=1.8348363490000001 podStartE2EDuration="4.072406897s" totalImagesPullingTime="2.237570548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:28.071642779 +0000 UTC m=+503.118776733" watchObservedRunningTime="2026-10-30 12:53:28.072406897 +0000 UTC m=+503.119540850"
Oct 30 12:53:28.235138 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:28.235103    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:28.365311 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:28.365276    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:28.484068 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:28.484037    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:28.779706 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:28.779615    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:28.925708 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:28.925673    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:29.004430 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:53:29.004391    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 12:53:29.033138 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:29.033049    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7bqsq" event={"ID":"79f27b5e-4f24-4970-87fb-d6d819190912","Type":"ContainerStarted","Data":"2fffb51c9faeab0cffd1f498f5e2212166382d891ff5f172a7569d8c7d3267b2"}
Oct 30 12:53:33.873808 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:33.873773    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:33.992192 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:33.992160    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:34.055187 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:34.055150    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7bqsq" event={"ID":"79f27b5e-4f24-4970-87fb-d6d819190912","Type":"ContainerStarted","Data":"506adf94309d286ecc77c4ac93e30778b6c8314e42bfad4e6b031b5d1a455ef9"}
Oct 30 12:53:34.055419 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:34.055296    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7bqsq"
Oct 30 12:53:34.099336 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:34.099275    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7bqsq" podStartSLOduration=1.8684739879999999 podStartE2EDuration="7.099260404s" totalImagesPullingTime="5.230786416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:34.096071596 +0000 UTC m=+509.143205549" watchObservedRunningTime="2026-10-30 12:53:34.099260404 +0000 UTC m=+509.146394358"
Oct 30 12:53:37.023109 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:37.023076    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-n6qxn"
Oct 30 12:53:38.489790 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:38.489740    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:38.602201 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:38.602169    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:39.035766 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:39.035737    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-nlk4n"
Oct 30 12:53:39.836273 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:39.836241    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:39.956795 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:39.956755    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:41.471919 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:41.471883    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:41.592604 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:41.592568    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:45.061627 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:45.061596    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7bqsq"
Oct 30 12:53:46.336297 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:46.336263    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:46.457667 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:46.457633    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:51.357004 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:51.356913    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-7bqsq"]
Oct 30 12:53:51.358150 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:51.357182    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7bqsq" podUID="79f27b5e-4f24-4970-87fb-d6d819190912" containerName="manager" containerID="cri-o://506adf94309d286ecc77c4ac93e30778b6c8314e42bfad4e6b031b5d1a455ef9" gracePeriod=2
Oct 30 12:53:51.368664 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:51.368629    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-7bqsq"]
Oct 30 12:53:51.370620 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:53:51.370588    4544 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-7bqsq\" is forbidden: User \"system:node:ip-10-0-143-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-38.ec2.internal' and this object" podUID="79f27b5e-4f24-4970-87fb-d6d819190912" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7bqsq"
Oct 30 12:53:51.384145 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:51.384110    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-689fp"]
Oct 30 12:53:51.384512 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:51.384494    4544 cpu_manager.go:414] "containerMap: removing container" podUID="79f27b5e-4f24-4970-87fb-d6d819190912" containerName="manager"
Oct 30 12:53:51.384592 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:51.384514    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79f27b5e-4f24-4970-87fb-d6d819190912" containerName="manager"
Oct 30 12:53:51.384647 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:51.384600    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="79f27b5e-4f24-4970-87fb-d6d819190912" containerName="manager"
Oct 30 12:53:51.387865 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:51.387840    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-689fp"
Oct 30 12:53:51.401080 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:53:51.401039    4544 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-7bqsq\" is forbidden: User \"system:node:ip-10-0-143-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-38.ec2.internal' and this object" podUID="79f27b5e-4f24-4970-87fb-d6d819190912" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7bqsq"
Oct 30 12:53:51.421203 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:51.421168    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-689fp"]
Oct 30 12:53:51.430960 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:51.430928    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znvhv\" (UniqueName: \"kubernetes.io/projected/ebca0355-cba2-4a9d-8a7c-81036e479ca8-kube-api-access-znvhv\") pod \"limitador-operator-controller-manager-856dd584c5-689fp\" (UID: \"ebca0355-cba2-4a9d-8a7c-81036e479ca8\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-689fp"
Oct 30 12:53:51.531732 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:51.531685    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znvhv\" (UniqueName: \"kubernetes.io/projected/ebca0355-cba2-4a9d-8a7c-81036e479ca8-kube-api-access-znvhv\") pod \"limitador-operator-controller-manager-856dd584c5-689fp\" (UID: \"ebca0355-cba2-4a9d-8a7c-81036e479ca8\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-689fp"
Oct 30 12:53:51.555001 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:51.554964    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znvhv\" (UniqueName: \"kubernetes.io/projected/ebca0355-cba2-4a9d-8a7c-81036e479ca8-kube-api-access-znvhv\") pod \"limitador-operator-controller-manager-856dd584c5-689fp\" (UID: \"ebca0355-cba2-4a9d-8a7c-81036e479ca8\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-689fp"
Oct 30 12:53:51.612159 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:51.612086    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7bqsq"
Oct 30 12:53:51.625442 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:53:51.625399    4544 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-7bqsq\" is forbidden: User \"system:node:ip-10-0-143-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-38.ec2.internal' and this object" podUID="79f27b5e-4f24-4970-87fb-d6d819190912" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7bqsq"
Oct 30 12:53:51.733046 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:51.733014    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79f27b5e-4f24-4970-87fb-d6d819190912-kube-api-access-fbdwj\" (UniqueName: \"kubernetes.io/projected/79f27b5e-4f24-4970-87fb-d6d819190912-kube-api-access-fbdwj\") pod \"79f27b5e-4f24-4970-87fb-d6d819190912\" (UID: \"79f27b5e-4f24-4970-87fb-d6d819190912\") "
Oct 30 12:53:51.735166 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:51.735137    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f27b5e-4f24-4970-87fb-d6d819190912-kube-api-access-fbdwj" pod "79f27b5e-4f24-4970-87fb-d6d819190912" (UID: "79f27b5e-4f24-4970-87fb-d6d819190912"). InnerVolumeSpecName "kube-api-access-fbdwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:53:51.754444 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:51.754409    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-689fp"
Oct 30 12:53:51.833753 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:51.833711    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbdwj\" (UniqueName: \"kubernetes.io/projected/79f27b5e-4f24-4970-87fb-d6d819190912-kube-api-access-fbdwj\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:53:52.036890 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:52.036814    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:52.085040 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:52.084994    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-689fp"]
Oct 30 12:53:52.089956 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:53:52.089928    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebca0355_cba2_4a9d_8a7c_81036e479ca8.slice/crio-553ee245c6e0d3050f08bb5ee642c77c10307f09afea238c6b52e9b0f0a35954 WatchSource:0}: Error finding container 553ee245c6e0d3050f08bb5ee642c77c10307f09afea238c6b52e9b0f0a35954: Status 404 returned error can't find the container with id 553ee245c6e0d3050f08bb5ee642c77c10307f09afea238c6b52e9b0f0a35954
Oct 30 12:53:52.129633 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:52.129592    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-689fp" event={"ID":"ebca0355-cba2-4a9d-8a7c-81036e479ca8","Type":"ContainerStarted","Data":"553ee245c6e0d3050f08bb5ee642c77c10307f09afea238c6b52e9b0f0a35954"}
Oct 30 12:53:52.131196 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:52.131171    4544 generic.go:356] "Generic (PLEG): container finished" podID="79f27b5e-4f24-4970-87fb-d6d819190912" containerID="506adf94309d286ecc77c4ac93e30778b6c8314e42bfad4e6b031b5d1a455ef9" exitCode=0
Oct 30 12:53:52.131344 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:52.131211    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7bqsq"
Oct 30 12:53:52.131344 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:52.131275    4544 scope.go:122] "RemoveContainer" containerID="506adf94309d286ecc77c4ac93e30778b6c8314e42bfad4e6b031b5d1a455ef9"
Oct 30 12:53:52.140006 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:52.139987    4544 scope.go:122] "RemoveContainer" containerID="506adf94309d286ecc77c4ac93e30778b6c8314e42bfad4e6b031b5d1a455ef9"
Oct 30 12:53:52.140334 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:53:52.140314    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"506adf94309d286ecc77c4ac93e30778b6c8314e42bfad4e6b031b5d1a455ef9\": container with ID starting with 506adf94309d286ecc77c4ac93e30778b6c8314e42bfad4e6b031b5d1a455ef9 not found: ID does not exist" containerID="506adf94309d286ecc77c4ac93e30778b6c8314e42bfad4e6b031b5d1a455ef9"
Oct 30 12:53:52.140407 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:52.140342    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"506adf94309d286ecc77c4ac93e30778b6c8314e42bfad4e6b031b5d1a455ef9"} err="failed to get container status \"506adf94309d286ecc77c4ac93e30778b6c8314e42bfad4e6b031b5d1a455ef9\": rpc error: code = NotFound desc = could not find container \"506adf94309d286ecc77c4ac93e30778b6c8314e42bfad4e6b031b5d1a455ef9\": container with ID starting with 506adf94309d286ecc77c4ac93e30778b6c8314e42bfad4e6b031b5d1a455ef9 not found: ID does not exist"
Oct 30 12:53:52.144471 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:53:52.144439    4544 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-7bqsq\" is forbidden: User \"system:node:ip-10-0-143-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-38.ec2.internal' and this object" podUID="79f27b5e-4f24-4970-87fb-d6d819190912" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7bqsq"
Oct 30 12:53:52.157638 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:53:52.157598    4544 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-7bqsq\" is forbidden: User \"system:node:ip-10-0-143-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-38.ec2.internal' and this object" podUID="79f27b5e-4f24-4970-87fb-d6d819190912" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7bqsq"
Oct 30 12:53:52.959548 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:52.959511    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:53.084551 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:53.084520    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79f27b5e-4f24-4970-87fb-d6d819190912" path="/var/lib/kubelet/pods/79f27b5e-4f24-4970-87fb-d6d819190912/volumes"
Oct 30 12:53:53.137352 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:53.137318    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-689fp" event={"ID":"ebca0355-cba2-4a9d-8a7c-81036e479ca8","Type":"ContainerStarted","Data":"e33e0ec9cc483cdffe101031fbf48b09cabf80a1873a6bad659d03437d0e1634"}
Oct 30 12:53:53.137526 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:53.137403    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-689fp"
Oct 30 12:53:53.159309 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:53.159275    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:53.186151 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:53.186095    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-689fp" podStartSLOduration=2.1860813869999998 podStartE2EDuration="2.186081387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:53.184843455 +0000 UTC m=+528.231977408" watchObservedRunningTime="2026-10-30 12:53:53.186081387 +0000 UTC m=+528.233215339"
Oct 30 12:53:58.341619 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:58.341585    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:58.451872 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:58.451792    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:58.940633 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:58.940597    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:53:59.007495 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:53:59.007457    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 12:53:59.082546 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:53:59.082487    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:02.358945 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:02.358907    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:02.492109 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:02.492076    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:04.144316 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:04.144282    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-689fp"
Oct 30 12:54:06.364374 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:06.364340    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:06.490840 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:06.490802    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:06.605646 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:06.605613    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:06.751772 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:06.751668    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:06.864666 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:06.864629    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:07.141299 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:07.141253    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:07.254193 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:07.254154    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:14.241248 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:14.241195    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:14.357781 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:14.357737    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:23.310491 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:23.310454    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:23.468765 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:23.468729    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:29.005178 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:54:29.005067    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 12:54:39.510890 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:39.510848    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-7ctvk"]
Oct 30 12:54:39.514504 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:39.514476    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-7ctvk"
Oct 30 12:54:39.524878 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:39.524840    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:39.533692 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:39.533662    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 12:54:39.542425 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:39.542395    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-s2c7q\""
Oct 30 12:54:39.543943 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:39.543919    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-7ctvk"]
Oct 30 12:54:39.631993 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:39.631959    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4610c929-1146-41b3-8af3-a1a137c31b5f-config-file\") pod \"limitador-limitador-74dfcfb894-7ctvk\" (UID: \"4610c929-1146-41b3-8af3-a1a137c31b5f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-7ctvk"
Oct 30 12:54:39.632188 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:39.632019    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fgqc\" (UniqueName: \"kubernetes.io/projected/4610c929-1146-41b3-8af3-a1a137c31b5f-kube-api-access-4fgqc\") pod \"limitador-limitador-74dfcfb894-7ctvk\" (UID: \"4610c929-1146-41b3-8af3-a1a137c31b5f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-7ctvk"
Oct 30 12:54:39.635717 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:39.635683    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:39.648627 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:39.648593    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-7ctvk"]
Oct 30 12:54:39.733262 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:39.733205    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4610c929-1146-41b3-8af3-a1a137c31b5f-config-file\") pod \"limitador-limitador-74dfcfb894-7ctvk\" (UID: \"4610c929-1146-41b3-8af3-a1a137c31b5f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-7ctvk"
Oct 30 12:54:39.733493 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:39.733294    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fgqc\" (UniqueName: \"kubernetes.io/projected/4610c929-1146-41b3-8af3-a1a137c31b5f-kube-api-access-4fgqc\") pod \"limitador-limitador-74dfcfb894-7ctvk\" (UID: \"4610c929-1146-41b3-8af3-a1a137c31b5f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-7ctvk"
Oct 30 12:54:39.734187 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:39.734163    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4610c929-1146-41b3-8af3-a1a137c31b5f-config-file\") pod \"limitador-limitador-74dfcfb894-7ctvk\" (UID: \"4610c929-1146-41b3-8af3-a1a137c31b5f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-7ctvk"
Oct 30 12:54:39.748455 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:39.748418    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:39.756180 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:39.756142    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fgqc\" (UniqueName: \"kubernetes.io/projected/4610c929-1146-41b3-8af3-a1a137c31b5f-kube-api-access-4fgqc\") pod \"limitador-limitador-74dfcfb894-7ctvk\" (UID: \"4610c929-1146-41b3-8af3-a1a137c31b5f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-7ctvk"
Oct 30 12:54:39.828202 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:39.828161    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-7ctvk"
Oct 30 12:54:40.127766 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:40.127678    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-7ctvk"]
Oct 30 12:54:40.133136 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:54:40.133110    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4610c929_1146_41b3_8af3_a1a137c31b5f.slice/crio-172d1f823c5c33bc8363f0939e4dc580a9cf8972b81b02edd441b0925ba9be0c WatchSource:0}: Error finding container 172d1f823c5c33bc8363f0939e4dc580a9cf8972b81b02edd441b0925ba9be0c: Status 404 returned error can't find the container with id 172d1f823c5c33bc8363f0939e4dc580a9cf8972b81b02edd441b0925ba9be0c
Oct 30 12:54:40.332778 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:40.332739    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-7ctvk" event={"ID":"4610c929-1146-41b3-8af3-a1a137c31b5f","Type":"ContainerStarted","Data":"172d1f823c5c33bc8363f0939e4dc580a9cf8972b81b02edd441b0925ba9be0c"}
Oct 30 12:54:40.448788 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:40.448703    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:40.585479 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:40.585426    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:41.081282 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:41.081255    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:41.193033 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:41.193002    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:41.338315 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:41.338272    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-7ctvk" event={"ID":"4610c929-1146-41b3-8af3-a1a137c31b5f","Type":"ContainerStarted","Data":"13addfc24607c1de0a0fe0fbcf57dfdc629780e313eca1a197066eb3081a2d3e"}
Oct 30 12:54:41.338513 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:41.338385    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-7ctvk"
Oct 30 12:54:41.378982 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:41.378925    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-7ctvk" podStartSLOduration=1.471604799 podStartE2EDuration="2.378910994s" totalImagesPullingTime="907.306195ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:41.378198387 +0000 UTC m=+576.425332361" watchObservedRunningTime="2026-10-30 12:54:41.378910994 +0000 UTC m=+576.426044947"
Oct 30 12:54:52.131237 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:52.131135    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:52.344113 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:52.344081    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-7ctvk"
Oct 30 12:54:54.155962 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:54.155922    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-7ctvk"]
Oct 30 12:54:54.156819 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:54.156157    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-7ctvk" podUID="4610c929-1146-41b3-8af3-a1a137c31b5f" containerName="limitador" containerID="cri-o://13addfc24607c1de0a0fe0fbcf57dfdc629780e313eca1a197066eb3081a2d3e" gracePeriod=30
Oct 30 12:54:54.699154 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:54.699131    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-7ctvk"
Oct 30 12:54:54.760087 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:54.760002    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4610c929-1146-41b3-8af3-a1a137c31b5f-kube-api-access-4fgqc\" (UniqueName: \"kubernetes.io/projected/4610c929-1146-41b3-8af3-a1a137c31b5f-kube-api-access-4fgqc\") pod \"4610c929-1146-41b3-8af3-a1a137c31b5f\" (UID: \"4610c929-1146-41b3-8af3-a1a137c31b5f\") "
Oct 30 12:54:54.760275 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:54.760195    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4610c929-1146-41b3-8af3-a1a137c31b5f-config-file\" (UniqueName: \"kubernetes.io/configmap/4610c929-1146-41b3-8af3-a1a137c31b5f-config-file\") pod \"4610c929-1146-41b3-8af3-a1a137c31b5f\" (UID: \"4610c929-1146-41b3-8af3-a1a137c31b5f\") "
Oct 30 12:54:54.760507 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:54.760487    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4610c929-1146-41b3-8af3-a1a137c31b5f-config-file" pod "4610c929-1146-41b3-8af3-a1a137c31b5f" (UID: "4610c929-1146-41b3-8af3-a1a137c31b5f"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:54:54.762123 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:54.762097    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4610c929-1146-41b3-8af3-a1a137c31b5f-kube-api-access-4fgqc" pod "4610c929-1146-41b3-8af3-a1a137c31b5f" (UID: "4610c929-1146-41b3-8af3-a1a137c31b5f"). InnerVolumeSpecName "kube-api-access-4fgqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:54:54.861695 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:54.861657    4544 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4610c929-1146-41b3-8af3-a1a137c31b5f-config-file\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:54:54.861695 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:54.861691    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fgqc\" (UniqueName: \"kubernetes.io/projected/4610c929-1146-41b3-8af3-a1a137c31b5f-kube-api-access-4fgqc\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:54:55.399250 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:55.399143    4544 generic.go:356] "Generic (PLEG): container finished" podID="4610c929-1146-41b3-8af3-a1a137c31b5f" containerID="13addfc24607c1de0a0fe0fbcf57dfdc629780e313eca1a197066eb3081a2d3e" exitCode=0
Oct 30 12:54:55.399250 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:55.399192    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-7ctvk"
Oct 30 12:54:55.399737 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:55.399191    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-7ctvk" event={"ID":"4610c929-1146-41b3-8af3-a1a137c31b5f","Type":"ContainerDied","Data":"13addfc24607c1de0a0fe0fbcf57dfdc629780e313eca1a197066eb3081a2d3e"}
Oct 30 12:54:55.399737 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:55.399326    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-7ctvk" event={"ID":"4610c929-1146-41b3-8af3-a1a137c31b5f","Type":"ContainerDied","Data":"172d1f823c5c33bc8363f0939e4dc580a9cf8972b81b02edd441b0925ba9be0c"}
Oct 30 12:54:55.399737 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:55.399357    4544 scope.go:122] "RemoveContainer" containerID="13addfc24607c1de0a0fe0fbcf57dfdc629780e313eca1a197066eb3081a2d3e"
Oct 30 12:54:55.407517 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:55.407497    4544 scope.go:122] "RemoveContainer" containerID="13addfc24607c1de0a0fe0fbcf57dfdc629780e313eca1a197066eb3081a2d3e"
Oct 30 12:54:55.407867 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:54:55.407846    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13addfc24607c1de0a0fe0fbcf57dfdc629780e313eca1a197066eb3081a2d3e\": container with ID starting with 13addfc24607c1de0a0fe0fbcf57dfdc629780e313eca1a197066eb3081a2d3e not found: ID does not exist" containerID="13addfc24607c1de0a0fe0fbcf57dfdc629780e313eca1a197066eb3081a2d3e"
Oct 30 12:54:55.407916 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:55.407878    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13addfc24607c1de0a0fe0fbcf57dfdc629780e313eca1a197066eb3081a2d3e"} err="failed to get container status \"13addfc24607c1de0a0fe0fbcf57dfdc629780e313eca1a197066eb3081a2d3e\": rpc error: code = NotFound desc = could not find container \"13addfc24607c1de0a0fe0fbcf57dfdc629780e313eca1a197066eb3081a2d3e\": container with ID starting with 13addfc24607c1de0a0fe0fbcf57dfdc629780e313eca1a197066eb3081a2d3e not found: ID does not exist"
Oct 30 12:54:55.448458 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:55.448428    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-7ctvk"]
Oct 30 12:54:55.460074 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:55.460039    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-7ctvk"]
Oct 30 12:54:57.084022 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:57.083983    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4610c929-1146-41b3-8af3-a1a137c31b5f" path="/var/lib/kubelet/pods/4610c929-1146-41b3-8af3-a1a137c31b5f/volumes"
Oct 30 12:54:57.730357 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:57.730328    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:58.449080 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:58.448993    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-4l4tc"]
Oct 30 12:54:58.449501 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:58.449331    4544 cpu_manager.go:414] "containerMap: removing container" podUID="4610c929-1146-41b3-8af3-a1a137c31b5f" containerName="limitador"
Oct 30 12:54:58.449501 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:58.449342    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4610c929-1146-41b3-8af3-a1a137c31b5f" containerName="limitador"
Oct 30 12:54:58.449501 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:58.449392    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="4610c929-1146-41b3-8af3-a1a137c31b5f" containerName="limitador"
Oct 30 12:54:58.454625 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:58.454597    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4l4tc"
Oct 30 12:54:58.474352 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:58.474263    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-4l4tc"]
Oct 30 12:54:58.481391 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:58.481358    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:54:58.481391 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:58.481363    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 30 12:54:58.481633 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:58.481433    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-2pl9r\""
Oct 30 12:54:58.481633 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:58.481486    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:54:58.595578 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:58.595536    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6m79\" (UniqueName: \"kubernetes.io/projected/7472d739-738d-481d-9845-ff8959593b6a-kube-api-access-m6m79\") pod \"postgres-7cdbf6dc44-4l4tc\" (UID: \"7472d739-738d-481d-9845-ff8959593b6a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4l4tc"
Oct 30 12:54:58.595755 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:58.595606    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7472d739-738d-481d-9845-ff8959593b6a-data\") pod \"postgres-7cdbf6dc44-4l4tc\" (UID: \"7472d739-738d-481d-9845-ff8959593b6a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4l4tc"
Oct 30 12:54:58.696236 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:58.696188    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7472d739-738d-481d-9845-ff8959593b6a-data\") pod \"postgres-7cdbf6dc44-4l4tc\" (UID: \"7472d739-738d-481d-9845-ff8959593b6a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4l4tc"
Oct 30 12:54:58.696420 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:58.696303    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6m79\" (UniqueName: \"kubernetes.io/projected/7472d739-738d-481d-9845-ff8959593b6a-kube-api-access-m6m79\") pod \"postgres-7cdbf6dc44-4l4tc\" (UID: \"7472d739-738d-481d-9845-ff8959593b6a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4l4tc"
Oct 30 12:54:58.696696 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:58.696671    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7472d739-738d-481d-9845-ff8959593b6a-data\") pod \"postgres-7cdbf6dc44-4l4tc\" (UID: \"7472d739-738d-481d-9845-ff8959593b6a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4l4tc"
Oct 30 12:54:58.719831 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:58.719759    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6m79\" (UniqueName: \"kubernetes.io/projected/7472d739-738d-481d-9845-ff8959593b6a-kube-api-access-m6m79\") pod \"postgres-7cdbf6dc44-4l4tc\" (UID: \"7472d739-738d-481d-9845-ff8959593b6a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4l4tc"
Oct 30 12:54:58.767399 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:58.767354    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4l4tc"
Oct 30 12:54:58.839447 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:58.839399    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:54:59.006733 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:54:59.006648    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 12:54:59.064291 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:59.064257    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-4l4tc"]
Oct 30 12:54:59.069261 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:54:59.069208    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7472d739_738d_481d_9845_ff8959593b6a.slice/crio-602a6160b82218355e4f1e54c5805f5344b2c01f994bf7c5cd78652e73b94b2c WatchSource:0}: Error finding container 602a6160b82218355e4f1e54c5805f5344b2c01f994bf7c5cd78652e73b94b2c: Status 404 returned error can't find the container with id 602a6160b82218355e4f1e54c5805f5344b2c01f994bf7c5cd78652e73b94b2c
Oct 30 12:54:59.418428 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:54:59.418389    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4l4tc" event={"ID":"7472d739-738d-481d-9845-ff8959593b6a","Type":"ContainerStarted","Data":"602a6160b82218355e4f1e54c5805f5344b2c01f994bf7c5cd78652e73b94b2c"}
Oct 30 12:55:04.451358 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:04.451319    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4l4tc" event={"ID":"7472d739-738d-481d-9845-ff8959593b6a","Type":"ContainerStarted","Data":"a6f5adc0765200753a9e8d3b51452bdcef5295b5d842a204150199f1fbad7a33"}
Oct 30 12:55:04.451780 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:04.451421    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4l4tc"
Oct 30 12:55:04.493100 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:04.493037    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4l4tc" podStartSLOduration=1.879963158 podStartE2EDuration="6.493020503s" totalImagesPullingTime="4.613057345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:04.491829213 +0000 UTC m=+599.538963210" watchObservedRunningTime="2026-10-30 12:55:04.493020503 +0000 UTC m=+599.540154459"
Oct 30 12:55:10.485613 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:10.485577    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4l4tc"
Oct 30 12:55:10.990643 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:10.990607    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:11.388061 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:11.388018    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:24.398417 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:24.398374    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:24.448873 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:24.448832    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-577f596567-5lv67"]
Oct 30 12:55:24.452705 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:24.452677    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-5lv67"
Oct 30 12:55:24.467159 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:24.467127    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 12:55:24.467159 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:24.467155    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 12:55:24.478789 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:24.478746    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 12:55:24.478789 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:24.478781    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2z8s8\""
Oct 30 12:55:24.480521 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:24.480481    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-5lv67"]
Oct 30 12:55:24.517082 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:24.517047    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:24.609456 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:24.609402    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f6v7\" (UniqueName: \"kubernetes.io/projected/16c82a23-b021-46c3-807d-24af470a19b3-kube-api-access-6f6v7\") pod \"maas-controller-577f596567-5lv67\" (UID: \"16c82a23-b021-46c3-807d-24af470a19b3\") " pod="opendatahub/maas-controller-577f596567-5lv67"
Oct 30 12:55:24.609658 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:24.609581    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16c82a23-b021-46c3-807d-24af470a19b3-metrics-certs\") pod \"maas-controller-577f596567-5lv67\" (UID: \"16c82a23-b021-46c3-807d-24af470a19b3\") " pod="opendatahub/maas-controller-577f596567-5lv67"
Oct 30 12:55:24.609730 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:24.609661    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/16c82a23-b021-46c3-807d-24af470a19b3-webhook-cert\") pod \"maas-controller-577f596567-5lv67\" (UID: \"16c82a23-b021-46c3-807d-24af470a19b3\") " pod="opendatahub/maas-controller-577f596567-5lv67"
Oct 30 12:55:24.634871 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:24.634832    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:24.711093 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:24.710995    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f6v7\" (UniqueName: \"kubernetes.io/projected/16c82a23-b021-46c3-807d-24af470a19b3-kube-api-access-6f6v7\") pod \"maas-controller-577f596567-5lv67\" (UID: \"16c82a23-b021-46c3-807d-24af470a19b3\") " pod="opendatahub/maas-controller-577f596567-5lv67"
Oct 30 12:55:24.711093 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:24.711086    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16c82a23-b021-46c3-807d-24af470a19b3-metrics-certs\") pod \"maas-controller-577f596567-5lv67\" (UID: \"16c82a23-b021-46c3-807d-24af470a19b3\") " pod="opendatahub/maas-controller-577f596567-5lv67"
Oct 30 12:55:24.711384 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:24.711120    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/16c82a23-b021-46c3-807d-24af470a19b3-webhook-cert\") pod \"maas-controller-577f596567-5lv67\" (UID: \"16c82a23-b021-46c3-807d-24af470a19b3\") " pod="opendatahub/maas-controller-577f596567-5lv67"
Oct 30 12:55:24.713911 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:24.713881    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16c82a23-b021-46c3-807d-24af470a19b3-metrics-certs\") pod \"maas-controller-577f596567-5lv67\" (UID: \"16c82a23-b021-46c3-807d-24af470a19b3\") " pod="opendatahub/maas-controller-577f596567-5lv67"
Oct 30 12:55:24.714030 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:24.713881    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/16c82a23-b021-46c3-807d-24af470a19b3-webhook-cert\") pod \"maas-controller-577f596567-5lv67\" (UID: \"16c82a23-b021-46c3-807d-24af470a19b3\") " pod="opendatahub/maas-controller-577f596567-5lv67"
Oct 30 12:55:24.734393 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:24.734359    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f6v7\" (UniqueName: \"kubernetes.io/projected/16c82a23-b021-46c3-807d-24af470a19b3-kube-api-access-6f6v7\") pod \"maas-controller-577f596567-5lv67\" (UID: \"16c82a23-b021-46c3-807d-24af470a19b3\") " pod="opendatahub/maas-controller-577f596567-5lv67"
Oct 30 12:55:24.748845 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:24.748812    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:24.765388 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:24.765350    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-5lv67"
Oct 30 12:55:24.829650 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:24.829381    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-75d8bff59-xxtf4"]
Oct 30 12:55:24.835381 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:24.835355    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75d8bff59-xxtf4"
Oct 30 12:55:24.855958 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:24.855923    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75d8bff59-xxtf4"]
Oct 30 12:55:25.013661 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:25.013564    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qjtv\" (UniqueName: \"kubernetes.io/projected/f9ef7c6e-9978-477c-be85-5816cf030b20-kube-api-access-9qjtv\") pod \"maas-controller-75d8bff59-xxtf4\" (UID: \"f9ef7c6e-9978-477c-be85-5816cf030b20\") " pod="opendatahub/maas-controller-75d8bff59-xxtf4"
Oct 30 12:55:25.013661 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:25.013621    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9ef7c6e-9978-477c-be85-5816cf030b20-metrics-certs\") pod \"maas-controller-75d8bff59-xxtf4\" (UID: \"f9ef7c6e-9978-477c-be85-5816cf030b20\") " pod="opendatahub/maas-controller-75d8bff59-xxtf4"
Oct 30 12:55:25.013857 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:25.013731    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f9ef7c6e-9978-477c-be85-5816cf030b20-webhook-cert\") pod \"maas-controller-75d8bff59-xxtf4\" (UID: \"f9ef7c6e-9978-477c-be85-5816cf030b20\") " pod="opendatahub/maas-controller-75d8bff59-xxtf4"
Oct 30 12:55:25.064355 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:25.064315    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-5lv67"]
Oct 30 12:55:25.069372 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:55:25.069345    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16c82a23_b021_46c3_807d_24af470a19b3.slice/crio-b34dce2bbf83afd6354732024f00c8996718e7c21b7c5e37e6c891e3b0517291 WatchSource:0}: Error finding container b34dce2bbf83afd6354732024f00c8996718e7c21b7c5e37e6c891e3b0517291: Status 404 returned error can't find the container with id b34dce2bbf83afd6354732024f00c8996718e7c21b7c5e37e6c891e3b0517291
Oct 30 12:55:25.070740 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:25.070723    4544 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 12:55:25.115076 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:25.115038    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qjtv\" (UniqueName: \"kubernetes.io/projected/f9ef7c6e-9978-477c-be85-5816cf030b20-kube-api-access-9qjtv\") pod \"maas-controller-75d8bff59-xxtf4\" (UID: \"f9ef7c6e-9978-477c-be85-5816cf030b20\") " pod="opendatahub/maas-controller-75d8bff59-xxtf4"
Oct 30 12:55:25.115282 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:25.115092    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9ef7c6e-9978-477c-be85-5816cf030b20-metrics-certs\") pod \"maas-controller-75d8bff59-xxtf4\" (UID: \"f9ef7c6e-9978-477c-be85-5816cf030b20\") " pod="opendatahub/maas-controller-75d8bff59-xxtf4"
Oct 30 12:55:25.115282 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:25.115135    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f9ef7c6e-9978-477c-be85-5816cf030b20-webhook-cert\") pod \"maas-controller-75d8bff59-xxtf4\" (UID: \"f9ef7c6e-9978-477c-be85-5816cf030b20\") " pod="opendatahub/maas-controller-75d8bff59-xxtf4"
Oct 30 12:55:25.117610 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:25.117579    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9ef7c6e-9978-477c-be85-5816cf030b20-metrics-certs\") pod \"maas-controller-75d8bff59-xxtf4\" (UID: \"f9ef7c6e-9978-477c-be85-5816cf030b20\") " pod="opendatahub/maas-controller-75d8bff59-xxtf4"
Oct 30 12:55:25.117610 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:25.117599    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f9ef7c6e-9978-477c-be85-5816cf030b20-webhook-cert\") pod \"maas-controller-75d8bff59-xxtf4\" (UID: \"f9ef7c6e-9978-477c-be85-5816cf030b20\") " pod="opendatahub/maas-controller-75d8bff59-xxtf4"
Oct 30 12:55:25.138467 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:25.138432    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qjtv\" (UniqueName: \"kubernetes.io/projected/f9ef7c6e-9978-477c-be85-5816cf030b20-kube-api-access-9qjtv\") pod \"maas-controller-75d8bff59-xxtf4\" (UID: \"f9ef7c6e-9978-477c-be85-5816cf030b20\") " pod="opendatahub/maas-controller-75d8bff59-xxtf4"
Oct 30 12:55:25.149276 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:25.149254    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75d8bff59-xxtf4"
Oct 30 12:55:25.448513 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:25.448471    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75d8bff59-xxtf4"]
Oct 30 12:55:25.453354 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:55:25.453321    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9ef7c6e_9978_477c_be85_5816cf030b20.slice/crio-d669be218a32550c76c557b654261a8ee55282801c6567fb6524b2ceb9e9e6fe WatchSource:0}: Error finding container d669be218a32550c76c557b654261a8ee55282801c6567fb6524b2ceb9e9e6fe: Status 404 returned error can't find the container with id d669be218a32550c76c557b654261a8ee55282801c6567fb6524b2ceb9e9e6fe
Oct 30 12:55:25.542505 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:25.542459    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-5lv67" event={"ID":"16c82a23-b021-46c3-807d-24af470a19b3","Type":"ContainerStarted","Data":"b34dce2bbf83afd6354732024f00c8996718e7c21b7c5e37e6c891e3b0517291"}
Oct 30 12:55:25.543443 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:25.543420    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75d8bff59-xxtf4" event={"ID":"f9ef7c6e-9978-477c-be85-5816cf030b20","Type":"ContainerStarted","Data":"d669be218a32550c76c557b654261a8ee55282801c6567fb6524b2ceb9e9e6fe"}
Oct 30 12:55:28.094166 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:28.094138    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:28.222199 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:28.222154    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:28.563560 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:28.563467    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-5lv67" event={"ID":"16c82a23-b021-46c3-807d-24af470a19b3","Type":"ContainerStarted","Data":"36a27c0278ee599198de8a540733fb8c367404b043f3732951618f517a2c1733"}
Oct 30 12:55:28.563748 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:28.563590    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-577f596567-5lv67"
Oct 30 12:55:28.565024 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:28.565001    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75d8bff59-xxtf4" event={"ID":"f9ef7c6e-9978-477c-be85-5816cf030b20","Type":"ContainerStarted","Data":"027309718f22a19a12793b4cc725756604a6144b25790a21332a3075bb524e07"}
Oct 30 12:55:28.565143 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:28.565118    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-75d8bff59-xxtf4"
Oct 30 12:55:28.634002 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:28.633949    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-577f596567-5lv67" podStartSLOduration=1.889121381 podStartE2EDuration="4.63393454s" totalImagesPullingTime="2.744813159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:28.633232435 +0000 UTC m=+623.680366379" watchObservedRunningTime="2026-10-30 12:55:28.63393454 +0000 UTC m=+623.681068492"
Oct 30 12:55:28.700643 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:28.700591    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-75d8bff59-xxtf4" podStartSLOduration=2.107449063 podStartE2EDuration="4.700574986s" totalImagesPullingTime="2.593125923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:28.70054778 +0000 UTC m=+623.747681733" watchObservedRunningTime="2026-10-30 12:55:28.700574986 +0000 UTC m=+623.747708939"
Oct 30 12:55:29.005230 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:55:29.005175    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 12:55:31.575128 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:31.575077    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-ssc54"]
Oct 30 12:55:31.578987 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:31.578950    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-ssc54"
Oct 30 12:55:31.605407 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:31.605377    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 30 12:55:31.605698 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:31.605435    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 30 12:55:31.605698 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:31.605515    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-hzztd\""
Oct 30 12:55:31.605698 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:31.605520    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 30 12:55:31.608464 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:31.608435    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-ssc54"]
Oct 30 12:55:31.669443 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:31.669412    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:31.671488 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:31.671468    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/509fe526-3fb6-4afa-807c-25028c1160b7-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-ssc54\" (UID: \"509fe526-3fb6-4afa-807c-25028c1160b7\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-ssc54"
Oct 30 12:55:31.671547 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:31.671503    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/509fe526-3fb6-4afa-807c-25028c1160b7-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-ssc54\" (UID: \"509fe526-3fb6-4afa-807c-25028c1160b7\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-ssc54"
Oct 30 12:55:31.671639 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:31.671623    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/509fe526-3fb6-4afa-807c-25028c1160b7-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-ssc54\" (UID: \"509fe526-3fb6-4afa-807c-25028c1160b7\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-ssc54"
Oct 30 12:55:31.671673 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:31.671658    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpjjt\" (UniqueName: \"kubernetes.io/projected/509fe526-3fb6-4afa-807c-25028c1160b7-kube-api-access-jpjjt\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-ssc54\" (UID: \"509fe526-3fb6-4afa-807c-25028c1160b7\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-ssc54"
Oct 30 12:55:31.772870 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:31.772837    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/509fe526-3fb6-4afa-807c-25028c1160b7-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-ssc54\" (UID: \"509fe526-3fb6-4afa-807c-25028c1160b7\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-ssc54"
Oct 30 12:55:31.773069 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:31.772882    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpjjt\" (UniqueName: \"kubernetes.io/projected/509fe526-3fb6-4afa-807c-25028c1160b7-kube-api-access-jpjjt\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-ssc54\" (UID: \"509fe526-3fb6-4afa-807c-25028c1160b7\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-ssc54"
Oct 30 12:55:31.773069 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:31.772911    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/509fe526-3fb6-4afa-807c-25028c1160b7-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-ssc54\" (UID: \"509fe526-3fb6-4afa-807c-25028c1160b7\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-ssc54"
Oct 30 12:55:31.773069 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:31.772960    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/509fe526-3fb6-4afa-807c-25028c1160b7-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-ssc54\" (UID: \"509fe526-3fb6-4afa-807c-25028c1160b7\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-ssc54"
Oct 30 12:55:31.773069 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:55:31.773007    4544 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found
Oct 30 12:55:31.773069 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:55:31.773060    4544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/509fe526-3fb6-4afa-807c-25028c1160b7-tls-cert podName:509fe526-3fb6-4afa-807c-25028c1160b7 nodeName:}" failed. No retries permitted until 2026-10-30 12:55:32.273041616 +0000 UTC m=+627.320175565 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/509fe526-3fb6-4afa-807c-25028c1160b7-tls-cert") pod "usage-logs-tenancy-proxy-7d94f55f4b-ssc54" (UID: "509fe526-3fb6-4afa-807c-25028c1160b7") : secret "usage-logs-tenancy-proxy-tls" not found
Oct 30 12:55:31.773648 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:31.773628    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/509fe526-3fb6-4afa-807c-25028c1160b7-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-ssc54\" (UID: \"509fe526-3fb6-4afa-807c-25028c1160b7\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-ssc54"
Oct 30 12:55:31.773805 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:31.773787    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/509fe526-3fb6-4afa-807c-25028c1160b7-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-ssc54\" (UID: \"509fe526-3fb6-4afa-807c-25028c1160b7\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-ssc54"
Oct 30 12:55:31.799145 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:31.799108    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpjjt\" (UniqueName: \"kubernetes.io/projected/509fe526-3fb6-4afa-807c-25028c1160b7-kube-api-access-jpjjt\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-ssc54\" (UID: \"509fe526-3fb6-4afa-807c-25028c1160b7\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-ssc54"
Oct 30 12:55:31.878469 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:31.878383    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:32.276432 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:32.276334    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/509fe526-3fb6-4afa-807c-25028c1160b7-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-ssc54\" (UID: \"509fe526-3fb6-4afa-807c-25028c1160b7\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-ssc54"
Oct 30 12:55:32.278938 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:32.278917    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/509fe526-3fb6-4afa-807c-25028c1160b7-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-ssc54\" (UID: \"509fe526-3fb6-4afa-807c-25028c1160b7\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-ssc54"
Oct 30 12:55:32.493025 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:32.492988    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-ssc54"
Oct 30 12:55:32.790842 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:32.790801    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-ssc54"]
Oct 30 12:55:32.795909 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:55:32.795875    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod509fe526_3fb6_4afa_807c_25028c1160b7.slice/crio-e49982ecc89415352dbf9a83eee4ef22b4c882e92a1e8189b8834cfc57cf8c29 WatchSource:0}: Error finding container e49982ecc89415352dbf9a83eee4ef22b4c882e92a1e8189b8834cfc57cf8c29: Status 404 returned error can't find the container with id e49982ecc89415352dbf9a83eee4ef22b4c882e92a1e8189b8834cfc57cf8c29
Oct 30 12:55:33.591816 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:33.591784    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-ssc54" event={"ID":"509fe526-3fb6-4afa-807c-25028c1160b7","Type":"ContainerStarted","Data":"e49982ecc89415352dbf9a83eee4ef22b4c882e92a1e8189b8834cfc57cf8c29"}
Oct 30 12:55:39.576667 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:39.576624    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-577f596567-5lv67"
Oct 30 12:55:39.577173 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:39.576721    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-75d8bff59-xxtf4"
Oct 30 12:55:39.691246 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:39.690696    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-5lv67"]
Oct 30 12:55:39.691246 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:39.690917    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-577f596567-5lv67" podUID="16c82a23-b021-46c3-807d-24af470a19b3" containerName="manager" containerID="cri-o://36a27c0278ee599198de8a540733fb8c367404b043f3732951618f517a2c1733" gracePeriod=30
Oct 30 12:55:40.057645 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:40.057607    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-5lv67"
Oct 30 12:55:40.151499 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:40.151418    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16c82a23-b021-46c3-807d-24af470a19b3-kube-api-access-6f6v7\" (UniqueName: \"kubernetes.io/projected/16c82a23-b021-46c3-807d-24af470a19b3-kube-api-access-6f6v7\") pod \"16c82a23-b021-46c3-807d-24af470a19b3\" (UID: \"16c82a23-b021-46c3-807d-24af470a19b3\") "
Oct 30 12:55:40.151499 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:40.151467    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16c82a23-b021-46c3-807d-24af470a19b3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/16c82a23-b021-46c3-807d-24af470a19b3-metrics-certs\") pod \"16c82a23-b021-46c3-807d-24af470a19b3\" (UID: \"16c82a23-b021-46c3-807d-24af470a19b3\") "
Oct 30 12:55:40.151733 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:40.151537    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16c82a23-b021-46c3-807d-24af470a19b3-webhook-cert\" (UniqueName: \"kubernetes.io/secret/16c82a23-b021-46c3-807d-24af470a19b3-webhook-cert\") pod \"16c82a23-b021-46c3-807d-24af470a19b3\" (UID: \"16c82a23-b021-46c3-807d-24af470a19b3\") "
Oct 30 12:55:40.154133 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:40.154103    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16c82a23-b021-46c3-807d-24af470a19b3-kube-api-access-6f6v7" pod "16c82a23-b021-46c3-807d-24af470a19b3" (UID: "16c82a23-b021-46c3-807d-24af470a19b3"). InnerVolumeSpecName "kube-api-access-6f6v7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:55:40.154276 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:40.154109    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16c82a23-b021-46c3-807d-24af470a19b3-metrics-certs" pod "16c82a23-b021-46c3-807d-24af470a19b3" (UID: "16c82a23-b021-46c3-807d-24af470a19b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:55:40.154276 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:40.154202    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16c82a23-b021-46c3-807d-24af470a19b3-webhook-cert" pod "16c82a23-b021-46c3-807d-24af470a19b3" (UID: "16c82a23-b021-46c3-807d-24af470a19b3"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:55:40.252673 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:40.252629    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f6v7\" (UniqueName: \"kubernetes.io/projected/16c82a23-b021-46c3-807d-24af470a19b3-kube-api-access-6f6v7\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:55:40.252673 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:40.252673    4544 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16c82a23-b021-46c3-807d-24af470a19b3-metrics-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:55:40.252892 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:40.252691    4544 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/16c82a23-b021-46c3-807d-24af470a19b3-webhook-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:55:40.632405 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:40.632353    4544 generic.go:356] "Generic (PLEG): container finished" podID="16c82a23-b021-46c3-807d-24af470a19b3" containerID="36a27c0278ee599198de8a540733fb8c367404b043f3732951618f517a2c1733" exitCode=0
Oct 30 12:55:40.632882 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:40.632407    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-5lv67"
Oct 30 12:55:40.632882 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:40.632426    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-5lv67" event={"ID":"16c82a23-b021-46c3-807d-24af470a19b3","Type":"ContainerDied","Data":"36a27c0278ee599198de8a540733fb8c367404b043f3732951618f517a2c1733"}
Oct 30 12:55:40.632882 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:40.632484    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-5lv67" event={"ID":"16c82a23-b021-46c3-807d-24af470a19b3","Type":"ContainerDied","Data":"b34dce2bbf83afd6354732024f00c8996718e7c21b7c5e37e6c891e3b0517291"}
Oct 30 12:55:40.632882 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:40.632519    4544 scope.go:122] "RemoveContainer" containerID="36a27c0278ee599198de8a540733fb8c367404b043f3732951618f517a2c1733"
Oct 30 12:55:40.643433 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:40.643408    4544 scope.go:122] "RemoveContainer" containerID="36a27c0278ee599198de8a540733fb8c367404b043f3732951618f517a2c1733"
Oct 30 12:55:40.643751 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:55:40.643727    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36a27c0278ee599198de8a540733fb8c367404b043f3732951618f517a2c1733\": container with ID starting with 36a27c0278ee599198de8a540733fb8c367404b043f3732951618f517a2c1733 not found: ID does not exist" containerID="36a27c0278ee599198de8a540733fb8c367404b043f3732951618f517a2c1733"
Oct 30 12:55:40.643840 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:40.643765    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36a27c0278ee599198de8a540733fb8c367404b043f3732951618f517a2c1733"} err="failed to get container status \"36a27c0278ee599198de8a540733fb8c367404b043f3732951618f517a2c1733\": rpc error: code = NotFound desc = could not find container \"36a27c0278ee599198de8a540733fb8c367404b043f3732951618f517a2c1733\": container with ID starting with 36a27c0278ee599198de8a540733fb8c367404b043f3732951618f517a2c1733 not found: ID does not exist"
Oct 30 12:55:40.682584 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:40.682547    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-5lv67"]
Oct 30 12:55:40.694290 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:40.694253    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-577f596567-5lv67"]
Oct 30 12:55:41.084768 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:41.084731    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16c82a23-b021-46c3-807d-24af470a19b3" path="/var/lib/kubelet/pods/16c82a23-b021-46c3-807d-24af470a19b3/volumes"
Oct 30 12:55:43.648532 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:43.648491    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-ssc54" event={"ID":"509fe526-3fb6-4afa-807c-25028c1160b7","Type":"ContainerStarted","Data":"9a0c9fdd11f4987253e52eff588a4ab47e1cb4f380345346591128bae12df07f"}
Oct 30 12:55:43.648935 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:43.648589    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-ssc54"
Oct 30 12:55:43.689860 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:43.689798    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-ssc54" podStartSLOduration=2.791806233 podStartE2EDuration="12.6897826s" totalImagesPullingTime="9.897976367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:43.688956433 +0000 UTC m=+638.736090388" watchObservedRunningTime="2026-10-30 12:55:43.6897826 +0000 UTC m=+638.736916554"
Oct 30 12:55:52.260322 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:52.260284    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:54.657529 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:54.657498    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-ssc54"
Oct 30 12:55:59.005527 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:55:59.005488    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 12:55:59.201465 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:59.201432    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:55:59.315349 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:55:59.315261    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:00.569437 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:00.569404    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:00.689099 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:00.689065    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:00.798721 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:00.798683    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:08.957562 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:08.957528    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"]
Oct 30 12:56:08.958067 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:08.957857    4544 cpu_manager.go:414] "containerMap: removing container" podUID="16c82a23-b021-46c3-807d-24af470a19b3" containerName="manager"
Oct 30 12:56:08.958067 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:08.957872    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16c82a23-b021-46c3-807d-24af470a19b3" containerName="manager"
Oct 30 12:56:08.958067 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:08.957981    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="16c82a23-b021-46c3-807d-24af470a19b3" containerName="manager"
Oct 30 12:56:09.075331 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:09.075291    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"]
Oct 30 12:56:09.075507 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:09.075427    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"
Oct 30 12:56:09.096859 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:09.096837    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4s79t\""
Oct 30 12:56:09.209142 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:09.209054    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16d5364-17da-4fef-993d-9ac8a77f56a5-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7grvt\" (UID: \"c16d5364-17da-4fef-993d-9ac8a77f56a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"
Oct 30 12:56:09.209142 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:09.209101    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c16d5364-17da-4fef-993d-9ac8a77f56a5-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-7grvt\" (UID: \"c16d5364-17da-4fef-993d-9ac8a77f56a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"
Oct 30 12:56:09.209142 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:09.209122    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c16d5364-17da-4fef-993d-9ac8a77f56a5-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-7grvt\" (UID: \"c16d5364-17da-4fef-993d-9ac8a77f56a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"
Oct 30 12:56:09.209414 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:09.209282    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16d5364-17da-4fef-993d-9ac8a77f56a5-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7grvt\" (UID: \"c16d5364-17da-4fef-993d-9ac8a77f56a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"
Oct 30 12:56:09.209414 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:09.209319    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c16d5364-17da-4fef-993d-9ac8a77f56a5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-7grvt\" (UID: \"c16d5364-17da-4fef-993d-9ac8a77f56a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"
Oct 30 12:56:09.209484 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:09.209427    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c16d5364-17da-4fef-993d-9ac8a77f56a5-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-7grvt\" (UID: \"c16d5364-17da-4fef-993d-9ac8a77f56a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"
Oct 30 12:56:09.209484 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:09.209451    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-625qk\" (UniqueName: \"kubernetes.io/projected/c16d5364-17da-4fef-993d-9ac8a77f56a5-kube-api-access-625qk\") pod \"maas-default-gateway-openshift-default-58c6585447-7grvt\" (UID: \"c16d5364-17da-4fef-993d-9ac8a77f56a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"
Oct 30 12:56:09.209547 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:09.209517    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16d5364-17da-4fef-993d-9ac8a77f56a5-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-7grvt\" (UID: \"c16d5364-17da-4fef-993d-9ac8a77f56a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"
Oct 30 12:56:09.209593 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:09.209580    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c16d5364-17da-4fef-993d-9ac8a77f56a5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-7grvt\" (UID: \"c16d5364-17da-4fef-993d-9ac8a77f56a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"
Oct 30 12:56:09.310465 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:09.310421    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c16d5364-17da-4fef-993d-9ac8a77f56a5-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-7grvt\" (UID: \"c16d5364-17da-4fef-993d-9ac8a77f56a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"
Oct 30 12:56:09.310465 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:09.310470    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c16d5364-17da-4fef-993d-9ac8a77f56a5-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-7grvt\" (UID: \"c16d5364-17da-4fef-993d-9ac8a77f56a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"
Oct 30 12:56:09.310727 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:09.310510    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16d5364-17da-4fef-993d-9ac8a77f56a5-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7grvt\" (UID: \"c16d5364-17da-4fef-993d-9ac8a77f56a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"
Oct 30 12:56:09.310727 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:09.310543    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c16d5364-17da-4fef-993d-9ac8a77f56a5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-7grvt\" (UID: \"c16d5364-17da-4fef-993d-9ac8a77f56a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"
Oct 30 12:56:09.310727 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:09.310595    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c16d5364-17da-4fef-993d-9ac8a77f56a5-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-7grvt\" (UID: \"c16d5364-17da-4fef-993d-9ac8a77f56a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"
Oct 30 12:56:09.310727 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:09.310622    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-625qk\" (UniqueName: \"kubernetes.io/projected/c16d5364-17da-4fef-993d-9ac8a77f56a5-kube-api-access-625qk\") pod \"maas-default-gateway-openshift-default-58c6585447-7grvt\" (UID: \"c16d5364-17da-4fef-993d-9ac8a77f56a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"
Oct 30 12:56:09.310727 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:09.310653    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16d5364-17da-4fef-993d-9ac8a77f56a5-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-7grvt\" (UID: \"c16d5364-17da-4fef-993d-9ac8a77f56a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"
Oct 30 12:56:09.310950 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:09.310728    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c16d5364-17da-4fef-993d-9ac8a77f56a5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-7grvt\" (UID: \"c16d5364-17da-4fef-993d-9ac8a77f56a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"
Oct 30 12:56:09.310950 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:09.310779    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16d5364-17da-4fef-993d-9ac8a77f56a5-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7grvt\" (UID: \"c16d5364-17da-4fef-993d-9ac8a77f56a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"
Oct 30 12:56:09.311552 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:09.311524    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16d5364-17da-4fef-993d-9ac8a77f56a5-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-7grvt\" (UID: \"c16d5364-17da-4fef-993d-9ac8a77f56a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"
Oct 30 12:56:09.311713 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:09.311536    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c16d5364-17da-4fef-993d-9ac8a77f56a5-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-7grvt\" (UID: \"c16d5364-17da-4fef-993d-9ac8a77f56a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"
Oct 30 12:56:09.311813 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:09.311565    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16d5364-17da-4fef-993d-9ac8a77f56a5-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7grvt\" (UID: \"c16d5364-17da-4fef-993d-9ac8a77f56a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"
Oct 30 12:56:09.311894 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:09.311840    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16d5364-17da-4fef-993d-9ac8a77f56a5-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7grvt\" (UID: \"c16d5364-17da-4fef-993d-9ac8a77f56a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"
Oct 30 12:56:09.318435 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:09.312155    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c16d5364-17da-4fef-993d-9ac8a77f56a5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-7grvt\" (UID: \"c16d5364-17da-4fef-993d-9ac8a77f56a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"
Oct 30 12:56:09.318435 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:09.314201    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c16d5364-17da-4fef-993d-9ac8a77f56a5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-7grvt\" (UID: \"c16d5364-17da-4fef-993d-9ac8a77f56a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"
Oct 30 12:56:09.318699 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:09.318673    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c16d5364-17da-4fef-993d-9ac8a77f56a5-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-7grvt\" (UID: \"c16d5364-17da-4fef-993d-9ac8a77f56a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"
Oct 30 12:56:09.333652 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:09.333626    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c16d5364-17da-4fef-993d-9ac8a77f56a5-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-7grvt\" (UID: \"c16d5364-17da-4fef-993d-9ac8a77f56a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"
Oct 30 12:56:09.333858 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:09.333837    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-625qk\" (UniqueName: \"kubernetes.io/projected/c16d5364-17da-4fef-993d-9ac8a77f56a5-kube-api-access-625qk\") pod \"maas-default-gateway-openshift-default-58c6585447-7grvt\" (UID: \"c16d5364-17da-4fef-993d-9ac8a77f56a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"
Oct 30 12:56:09.409443 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:09.409406    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"
Oct 30 12:56:09.746212 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:09.746124    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"]
Oct 30 12:56:09.751319 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:56:09.751287    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc16d5364_17da_4fef_993d_9ac8a77f56a5.slice/crio-aec4d2566fdfeffb5c7e28f8087ba0e271317c66d542b0d80d0392ef1266ea4f WatchSource:0}: Error finding container aec4d2566fdfeffb5c7e28f8087ba0e271317c66d542b0d80d0392ef1266ea4f: Status 404 returned error can't find the container with id aec4d2566fdfeffb5c7e28f8087ba0e271317c66d542b0d80d0392ef1266ea4f
Oct 30 12:56:09.761132 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:09.761094    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt" event={"ID":"c16d5364-17da-4fef-993d-9ac8a77f56a5","Type":"ContainerStarted","Data":"aec4d2566fdfeffb5c7e28f8087ba0e271317c66d542b0d80d0392ef1266ea4f"}
Oct 30 12:56:12.266609 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:12.266576    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:56:12.266890 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:12.266647    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:56:12.266890 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:12.266679    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:56:12.777281 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:12.777243    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt" event={"ID":"c16d5364-17da-4fef-993d-9ac8a77f56a5","Type":"ContainerStarted","Data":"c8b0a2740859fec9a7c6002151fd928782d644dacf2d44ef4b82e1cd80b3532d"}
Oct 30 12:56:12.875447 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:12.875390    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt" podStartSLOduration=2.363029844 podStartE2EDuration="4.875374447s" totalImagesPullingTime="2.512344603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:12.862755079 +0000 UTC m=+667.909889047" watchObservedRunningTime="2026-10-30 12:56:12.875374447 +0000 UTC m=+667.922508400"
Oct 30 12:56:13.410311 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:13.410260    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"
Oct 30 12:56:13.411711 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:13.411683    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7grvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 12:56:13.411829 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:13.411734    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt" podUID="c16d5364-17da-4fef-993d-9ac8a77f56a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 12:56:14.410360 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:14.410317    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7grvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 12:56:14.410763 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:14.410375    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt" podUID="c16d5364-17da-4fef-993d-9ac8a77f56a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 12:56:15.410125 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:15.410081    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7grvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 12:56:15.410342 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:15.410137    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt" podUID="c16d5364-17da-4fef-993d-9ac8a77f56a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 12:56:16.410682 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:16.410641    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7grvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 12:56:16.411100 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:16.410698    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt" podUID="c16d5364-17da-4fef-993d-9ac8a77f56a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 12:56:17.410918 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:17.410820    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7grvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 12:56:17.411365 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:17.410916    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt" podUID="c16d5364-17da-4fef-993d-9ac8a77f56a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 12:56:18.414140 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:18.414108    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"
Oct 30 12:56:18.414566 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:18.414545    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"
Oct 30 12:56:18.415245 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:18.415200    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt"
Oct 30 12:56:29.006899 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:56:29.006859    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 12:56:46.549036 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:46.548993    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:46.669068 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:46.669037    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:52.368678 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:56:52.368638    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:56:59.004378 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:56:59.004333    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 12:57:06.384798 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:06.384751    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6"]
Oct 30 12:57:06.389185 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:06.389158    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6"
Oct 30 12:57:06.403981 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:06.403953    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 12:57:06.415457 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:06.415419    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 30 12:57:06.418172 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:06.418144    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-58g6n\""
Oct 30 12:57:06.420752 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:06.420722    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6"]
Oct 30 12:57:06.466299 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:06.466265    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:06.493567 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:06.493522    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a23bc66-d828-423f-a2a1-e803aef031d1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6\" (UID: \"4a23bc66-d828-423f-a2a1-e803aef031d1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6"
Oct 30 12:57:06.493567 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:06.493567    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfqn5\" (UniqueName: \"kubernetes.io/projected/4a23bc66-d828-423f-a2a1-e803aef031d1-kube-api-access-rfqn5\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6\" (UID: \"4a23bc66-d828-423f-a2a1-e803aef031d1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6"
Oct 30 12:57:06.493785 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:06.493644    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a23bc66-d828-423f-a2a1-e803aef031d1-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6\" (UID: \"4a23bc66-d828-423f-a2a1-e803aef031d1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6"
Oct 30 12:57:06.493785 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:06.493757    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a23bc66-d828-423f-a2a1-e803aef031d1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6\" (UID: \"4a23bc66-d828-423f-a2a1-e803aef031d1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6"
Oct 30 12:57:06.494001 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:06.493786    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a23bc66-d828-423f-a2a1-e803aef031d1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6\" (UID: \"4a23bc66-d828-423f-a2a1-e803aef031d1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6"
Oct 30 12:57:06.494375 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:06.494347    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a23bc66-d828-423f-a2a1-e803aef031d1-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6\" (UID: \"4a23bc66-d828-423f-a2a1-e803aef031d1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6"
Oct 30 12:57:06.512613 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:06.512573    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 12:57:06.595804 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:06.595760    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a23bc66-d828-423f-a2a1-e803aef031d1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6\" (UID: \"4a23bc66-d828-423f-a2a1-e803aef031d1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6"
Oct 30 12:57:06.595804 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:06.595806    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a23bc66-d828-423f-a2a1-e803aef031d1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6\" (UID: \"4a23bc66-d828-423f-a2a1-e803aef031d1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6"
Oct 30 12:57:06.596064 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:06.595825    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a23bc66-d828-423f-a2a1-e803aef031d1-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6\" (UID: \"4a23bc66-d828-423f-a2a1-e803aef031d1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6"
Oct 30 12:57:06.596064 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:06.595861    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a23bc66-d828-423f-a2a1-e803aef031d1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6\" (UID: \"4a23bc66-d828-423f-a2a1-e803aef031d1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6"
Oct 30 12:57:06.596064 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:06.595992    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfqn5\" (UniqueName: \"kubernetes.io/projected/4a23bc66-d828-423f-a2a1-e803aef031d1-kube-api-access-rfqn5\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6\" (UID: \"4a23bc66-d828-423f-a2a1-e803aef031d1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6"
Oct 30 12:57:06.596064 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:06.596060    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a23bc66-d828-423f-a2a1-e803aef031d1-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6\" (UID: \"4a23bc66-d828-423f-a2a1-e803aef031d1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6"
Oct 30 12:57:06.596274 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:06.596255    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a23bc66-d828-423f-a2a1-e803aef031d1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6\" (UID: \"4a23bc66-d828-423f-a2a1-e803aef031d1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6"
Oct 30 12:57:06.596340 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:06.596317    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a23bc66-d828-423f-a2a1-e803aef031d1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6\" (UID: \"4a23bc66-d828-423f-a2a1-e803aef031d1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6"
Oct 30 12:57:06.596464 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:06.596443    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a23bc66-d828-423f-a2a1-e803aef031d1-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6\" (UID: \"4a23bc66-d828-423f-a2a1-e803aef031d1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6"
Oct 30 12:57:06.598471 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:06.598442    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a23bc66-d828-423f-a2a1-e803aef031d1-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6\" (UID: \"4a23bc66-d828-423f-a2a1-e803aef031d1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6"
Oct 30 12:57:06.598642 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:06.598625    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a23bc66-d828-423f-a2a1-e803aef031d1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6\" (UID: \"4a23bc66-d828-423f-a2a1-e803aef031d1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6"
Oct 30 12:57:06.627123 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:06.627090    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfqn5\" (UniqueName: \"kubernetes.io/projected/4a23bc66-d828-423f-a2a1-e803aef031d1-kube-api-access-rfqn5\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6\" (UID: \"4a23bc66-d828-423f-a2a1-e803aef031d1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6"
Oct 30 12:57:06.705919 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:06.705813    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6"
Oct 30 12:57:07.015041 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:07.014870    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6"]
Oct 30 12:57:07.019514 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:57:07.019473    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a23bc66_d828_423f_a2a1_e803aef031d1.slice/crio-5f73a45e3ded14b2b14c22c50d6babf2f57c5912649ac3d46dddfc371a2a64b3 WatchSource:0}: Error finding container 5f73a45e3ded14b2b14c22c50d6babf2f57c5912649ac3d46dddfc371a2a64b3: Status 404 returned error can't find the container with id 5f73a45e3ded14b2b14c22c50d6babf2f57c5912649ac3d46dddfc371a2a64b3
Oct 30 12:57:08.014309 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:08.014264    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6" event={"ID":"4a23bc66-d828-423f-a2a1-e803aef031d1","Type":"ContainerStarted","Data":"5f73a45e3ded14b2b14c22c50d6babf2f57c5912649ac3d46dddfc371a2a64b3"}
Oct 30 12:57:08.250760 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:08.250707    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-bd6w2"]
Oct 30 12:57:08.254943 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:08.254883    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bd6w2"
Oct 30 12:57:08.269662 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:08.269575    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 30 12:57:08.283069 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:08.283034    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-bd6w2"]
Oct 30 12:57:08.315632 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:08.315599    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:08.412946 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:08.412901    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2chk4\" (UniqueName: \"kubernetes.io/projected/c0251641-4f60-4eae-8970-dbc5f652936b-kube-api-access-2chk4\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bd6w2\" (UID: \"c0251641-4f60-4eae-8970-dbc5f652936b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bd6w2"
Oct 30 12:57:08.412946 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:08.412950    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0251641-4f60-4eae-8970-dbc5f652936b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bd6w2\" (UID: \"c0251641-4f60-4eae-8970-dbc5f652936b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bd6w2"
Oct 30 12:57:08.413183 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:08.413103    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0251641-4f60-4eae-8970-dbc5f652936b-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bd6w2\" (UID: \"c0251641-4f60-4eae-8970-dbc5f652936b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bd6w2"
Oct 30 12:57:08.413183 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:08.413153    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0251641-4f60-4eae-8970-dbc5f652936b-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bd6w2\" (UID: \"c0251641-4f60-4eae-8970-dbc5f652936b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bd6w2"
Oct 30 12:57:08.413277 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:08.413263    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0251641-4f60-4eae-8970-dbc5f652936b-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bd6w2\" (UID: \"c0251641-4f60-4eae-8970-dbc5f652936b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bd6w2"
Oct 30 12:57:08.413312 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:08.413285    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0251641-4f60-4eae-8970-dbc5f652936b-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bd6w2\" (UID: \"c0251641-4f60-4eae-8970-dbc5f652936b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bd6w2"
Oct 30 12:57:08.514293 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:08.514253    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0251641-4f60-4eae-8970-dbc5f652936b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bd6w2\" (UID: \"c0251641-4f60-4eae-8970-dbc5f652936b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bd6w2"
Oct 30 12:57:08.514485 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:08.514320    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0251641-4f60-4eae-8970-dbc5f652936b-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bd6w2\" (UID: \"c0251641-4f60-4eae-8970-dbc5f652936b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bd6w2"
Oct 30 12:57:08.514485 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:08.514348    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0251641-4f60-4eae-8970-dbc5f652936b-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bd6w2\" (UID: \"c0251641-4f60-4eae-8970-dbc5f652936b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bd6w2"
Oct 30 12:57:08.514485 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:08.514370    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0251641-4f60-4eae-8970-dbc5f652936b-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bd6w2\" (UID: \"c0251641-4f60-4eae-8970-dbc5f652936b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bd6w2"
Oct 30 12:57:08.514485 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:08.514385    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0251641-4f60-4eae-8970-dbc5f652936b-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bd6w2\" (UID: \"c0251641-4f60-4eae-8970-dbc5f652936b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bd6w2"
Oct 30 12:57:08.514485 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:08.514408    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2chk4\" (UniqueName: \"kubernetes.io/projected/c0251641-4f60-4eae-8970-dbc5f652936b-kube-api-access-2chk4\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bd6w2\" (UID: \"c0251641-4f60-4eae-8970-dbc5f652936b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bd6w2"
Oct 30 12:57:08.514768 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:08.514741    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0251641-4f60-4eae-8970-dbc5f652936b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bd6w2\" (UID: \"c0251641-4f60-4eae-8970-dbc5f652936b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bd6w2"
Oct 30 12:57:08.514829 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:08.514804    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0251641-4f60-4eae-8970-dbc5f652936b-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bd6w2\" (UID: \"c0251641-4f60-4eae-8970-dbc5f652936b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bd6w2"
Oct 30 12:57:08.514829 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:08.514817    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0251641-4f60-4eae-8970-dbc5f652936b-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bd6w2\" (UID: \"c0251641-4f60-4eae-8970-dbc5f652936b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bd6w2"
Oct 30 12:57:08.516580 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:08.516557    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0251641-4f60-4eae-8970-dbc5f652936b-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bd6w2\" (UID: \"c0251641-4f60-4eae-8970-dbc5f652936b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bd6w2"
Oct 30 12:57:08.516841 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:08.516823    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0251641-4f60-4eae-8970-dbc5f652936b-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bd6w2\" (UID: \"c0251641-4f60-4eae-8970-dbc5f652936b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bd6w2"
Oct 30 12:57:08.538330 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:08.538255    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2chk4\" (UniqueName: \"kubernetes.io/projected/c0251641-4f60-4eae-8970-dbc5f652936b-kube-api-access-2chk4\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bd6w2\" (UID: \"c0251641-4f60-4eae-8970-dbc5f652936b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bd6w2"
Oct 30 12:57:08.575310 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:08.575253    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bd6w2"
Oct 30 12:57:08.878048 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:08.877954    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-bd6w2"]
Oct 30 12:57:08.882578 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:57:08.882548    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0251641_4f60_4eae_8970_dbc5f652936b.slice/crio-d4294f75ab93b9f371689b63bb419bdace23b8a18b18cf29bcc9bd31001f6ba9 WatchSource:0}: Error finding container d4294f75ab93b9f371689b63bb419bdace23b8a18b18cf29bcc9bd31001f6ba9: Status 404 returned error can't find the container with id d4294f75ab93b9f371689b63bb419bdace23b8a18b18cf29bcc9bd31001f6ba9
Oct 30 12:57:09.020263 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:09.020202    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bd6w2" event={"ID":"c0251641-4f60-4eae-8970-dbc5f652936b","Type":"ContainerStarted","Data":"d4294f75ab93b9f371689b63bb419bdace23b8a18b18cf29bcc9bd31001f6ba9"}
Oct 30 12:57:09.021685 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:09.021657    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6" event={"ID":"4a23bc66-d828-423f-a2a1-e803aef031d1","Type":"ContainerStarted","Data":"c496283088c44a31ef9d63b51e8513773e77a7987aab561cf96492e48292124e"}
Oct 30 12:57:09.065059 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:09.065002    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6" podStartSLOduration=2.028499752 podStartE2EDuration="3.064987435s" totalImagesPullingTime="1.036487683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:09.063987224 +0000 UTC m=+724.111121177" watchObservedRunningTime="2026-10-30 12:57:09.064987435 +0000 UTC m=+724.112121386"
Oct 30 12:57:09.706772 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:09.706733    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6"
Oct 30 12:57:09.893493 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:09.893460    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:10.028656 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:10.028556    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bd6w2" event={"ID":"c0251641-4f60-4eae-8970-dbc5f652936b","Type":"ContainerStarted","Data":"b25800b7f27280f73e5e8f78c37f3537e982ce127cd4632f66bf0af19a9db148"}
Oct 30 12:57:10.072711 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:10.072656    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bd6w2" podStartSLOduration=1.866375251 podStartE2EDuration="2.072642742s" totalImagesPullingTime="206.267491ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:10.071709518 +0000 UTC m=+725.118843484" watchObservedRunningTime="2026-10-30 12:57:10.072642742 +0000 UTC m=+725.119776695"
Oct 30 12:57:10.576047 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:10.576002    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bd6w2"
Oct 30 12:57:11.696076 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:11.696044    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:14.357104 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:14.357057    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88"]
Oct 30 12:57:14.361034 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:14.361010    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88"
Oct 30 12:57:14.371125 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:14.371088    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62b23320-6cd9-4cbb-b58e-cf8de8a931a8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88\" (UID: \"62b23320-6cd9-4cbb-b58e-cf8de8a931a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88"
Oct 30 12:57:14.371336 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:14.371139    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62b23320-6cd9-4cbb-b58e-cf8de8a931a8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88\" (UID: \"62b23320-6cd9-4cbb-b58e-cf8de8a931a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88"
Oct 30 12:57:14.371409 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:14.371333    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62b23320-6cd9-4cbb-b58e-cf8de8a931a8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88\" (UID: \"62b23320-6cd9-4cbb-b58e-cf8de8a931a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88"
Oct 30 12:57:14.371409 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:14.371395    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w65sz\" (UniqueName: \"kubernetes.io/projected/62b23320-6cd9-4cbb-b58e-cf8de8a931a8-kube-api-access-w65sz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88\" (UID: \"62b23320-6cd9-4cbb-b58e-cf8de8a931a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88"
Oct 30 12:57:14.371590 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:14.371465    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62b23320-6cd9-4cbb-b58e-cf8de8a931a8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88\" (UID: \"62b23320-6cd9-4cbb-b58e-cf8de8a931a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88"
Oct 30 12:57:14.371590 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:14.371554    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62b23320-6cd9-4cbb-b58e-cf8de8a931a8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88\" (UID: \"62b23320-6cd9-4cbb-b58e-cf8de8a931a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88"
Oct 30 12:57:14.387714 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:14.387686    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 30 12:57:14.395947 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:14.395913    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88"]
Oct 30 12:57:14.457614 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:14.457579    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:14.472080 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:14.472043    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62b23320-6cd9-4cbb-b58e-cf8de8a931a8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88\" (UID: \"62b23320-6cd9-4cbb-b58e-cf8de8a931a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88"
Oct 30 12:57:14.472306 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:14.472099    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62b23320-6cd9-4cbb-b58e-cf8de8a931a8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88\" (UID: \"62b23320-6cd9-4cbb-b58e-cf8de8a931a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88"
Oct 30 12:57:14.472306 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:14.472130    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62b23320-6cd9-4cbb-b58e-cf8de8a931a8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88\" (UID: \"62b23320-6cd9-4cbb-b58e-cf8de8a931a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88"
Oct 30 12:57:14.472306 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:14.472177    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62b23320-6cd9-4cbb-b58e-cf8de8a931a8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88\" (UID: \"62b23320-6cd9-4cbb-b58e-cf8de8a931a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88"
Oct 30 12:57:14.472489 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:14.472374    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w65sz\" (UniqueName: \"kubernetes.io/projected/62b23320-6cd9-4cbb-b58e-cf8de8a931a8-kube-api-access-w65sz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88\" (UID: \"62b23320-6cd9-4cbb-b58e-cf8de8a931a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88"
Oct 30 12:57:14.472489 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:14.472449    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62b23320-6cd9-4cbb-b58e-cf8de8a931a8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88\" (UID: \"62b23320-6cd9-4cbb-b58e-cf8de8a931a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88"
Oct 30 12:57:14.472593 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:14.472551    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62b23320-6cd9-4cbb-b58e-cf8de8a931a8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88\" (UID: \"62b23320-6cd9-4cbb-b58e-cf8de8a931a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88"
Oct 30 12:57:14.472593 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:14.472573    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62b23320-6cd9-4cbb-b58e-cf8de8a931a8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88\" (UID: \"62b23320-6cd9-4cbb-b58e-cf8de8a931a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88"
Oct 30 12:57:14.472691 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:14.472613    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62b23320-6cd9-4cbb-b58e-cf8de8a931a8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88\" (UID: \"62b23320-6cd9-4cbb-b58e-cf8de8a931a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88"
Oct 30 12:57:14.474733 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:14.474703    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62b23320-6cd9-4cbb-b58e-cf8de8a931a8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88\" (UID: \"62b23320-6cd9-4cbb-b58e-cf8de8a931a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88"
Oct 30 12:57:14.475117 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:14.475095    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62b23320-6cd9-4cbb-b58e-cf8de8a931a8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88\" (UID: \"62b23320-6cd9-4cbb-b58e-cf8de8a931a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88"
Oct 30 12:57:14.498323 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:14.498290    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w65sz\" (UniqueName: \"kubernetes.io/projected/62b23320-6cd9-4cbb-b58e-cf8de8a931a8-kube-api-access-w65sz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88\" (UID: \"62b23320-6cd9-4cbb-b58e-cf8de8a931a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88"
Oct 30 12:57:14.676382 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:14.676284    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88"
Oct 30 12:57:14.979151 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:14.979035    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88"]
Oct 30 12:57:14.983600 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:57:14.983567    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62b23320_6cd9_4cbb_b58e_cf8de8a931a8.slice/crio-c5965d4f772d3c3e315379143e5b33cc1a5caa203c486813af100b3f90d7b863 WatchSource:0}: Error finding container c5965d4f772d3c3e315379143e5b33cc1a5caa203c486813af100b3f90d7b863: Status 404 returned error can't find the container with id c5965d4f772d3c3e315379143e5b33cc1a5caa203c486813af100b3f90d7b863
Oct 30 12:57:15.051135 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:15.051099    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88" event={"ID":"62b23320-6cd9-4cbb-b58e-cf8de8a931a8","Type":"ContainerStarted","Data":"c5965d4f772d3c3e315379143e5b33cc1a5caa203c486813af100b3f90d7b863"}
Oct 30 12:57:16.056799 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:16.056766    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88" event={"ID":"62b23320-6cd9-4cbb-b58e-cf8de8a931a8","Type":"ContainerStarted","Data":"b9385918af84c6a1acd27a4e8213ea0ba2790c711cbc62501e2dd873b8598e30"}
Oct 30 12:57:16.099520 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:16.099469    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88" podStartSLOduration=1.896664882 podStartE2EDuration="2.099453883s" totalImagesPullingTime="202.789001ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:16.098581036 +0000 UTC m=+731.145714988" watchObservedRunningTime="2026-10-30 12:57:16.099453883 +0000 UTC m=+731.146587835"
Oct 30 12:57:16.677002 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:16.676954    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88"
Oct 30 12:57:16.706481 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:16.706442    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6"
Oct 30 12:57:16.719007 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:16.718978    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6"
Oct 30 12:57:17.084089 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:17.084063    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6"
Oct 30 12:57:17.510539 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:17.510453    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:18.576038 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:18.576005    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bd6w2"
Oct 30 12:57:18.589603 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:18.589574    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bd6w2"
Oct 30 12:57:19.093695 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:19.093668    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bd6w2"
Oct 30 12:57:20.402281 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:20.402196    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:24.022965 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:24.022919    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:24.676860 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:24.676815    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88"
Oct 30 12:57:24.689863 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:24.689823    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88"
Oct 30 12:57:25.122295 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:25.122263    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88"
Oct 30 12:57:29.006969 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:57:29.006928    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 12:57:39.243142 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.243103    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"]
Oct 30 12:57:39.248514 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.248445    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 12:57:39.286321 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.286281    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"]
Oct 30 12:57:39.416824 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.416784    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-7462m\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 12:57:39.417001 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.416845    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-7462m\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 12:57:39.417001 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.416894    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-7462m\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 12:57:39.417079 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.417003    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-7462m\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 12:57:39.417079 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.417033    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7462m\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 12:57:39.417079 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.417062    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xld8z\" (UniqueName: \"kubernetes.io/projected/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-kube-api-access-xld8z\") pod \"maas-default-gateway-openshift-default-58c6585447-7462m\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 12:57:39.417176 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.417123    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7462m\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 12:57:39.417213 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.417195    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-7462m\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 12:57:39.417309 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.417251    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-7462m\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 12:57:39.518553 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.518461    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-7462m\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 12:57:39.518553 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.518517    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-7462m\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 12:57:39.518553 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.518532    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-7462m\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 12:57:39.518855 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.518561    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-7462m\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 12:57:39.518855 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.518593    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7462m\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 12:57:39.518855 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.518609    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xld8z\" (UniqueName: \"kubernetes.io/projected/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-kube-api-access-xld8z\") pod \"maas-default-gateway-openshift-default-58c6585447-7462m\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 12:57:39.518855 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.518642    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7462m\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 12:57:39.519070 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.519016    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7462m\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 12:57:39.519155 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.519123    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7462m\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 12:57:39.519215 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.519154    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-7462m\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 12:57:39.519215 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.519190    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-7462m\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 12:57:39.519342 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.519209    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-7462m\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 12:57:39.519458 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.519435    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-7462m\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 12:57:39.519458 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.519445    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-7462m\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 12:57:39.521014 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.520992    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-7462m\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 12:57:39.521270 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.521246    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-7462m\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 12:57:39.547620 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.547580    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-7462m\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 12:57:39.547773 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.547685    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xld8z\" (UniqueName: \"kubernetes.io/projected/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-kube-api-access-xld8z\") pod \"maas-default-gateway-openshift-default-58c6585447-7462m\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 12:57:39.562408 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.562376    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 12:57:39.903547 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.903514    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"]
Oct 30 12:57:39.907487 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:57:39.907458    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a3bf11c_7722_47cd_a2ff_0e8cf5bacdcf.slice/crio-aa6999a2e89b62387e2c6216f171cb1b5186db82aebe8be3c2297ede7cd6f5a0 WatchSource:0}: Error finding container aa6999a2e89b62387e2c6216f171cb1b5186db82aebe8be3c2297ede7cd6f5a0: Status 404 returned error can't find the container with id aa6999a2e89b62387e2c6216f171cb1b5186db82aebe8be3c2297ede7cd6f5a0
Oct 30 12:57:39.909531 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.909497    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:57:39.909617 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.909566    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:57:39.909617 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:39.909596    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:57:40.178043 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:40.177941    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m" event={"ID":"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf","Type":"ContainerStarted","Data":"e72b6fd6e3230ccb48f7e93bfb1bb132da0338be2ca04492b8bf7f9a42306f51"}
Oct 30 12:57:40.178327 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:40.178052    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m" event={"ID":"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf","Type":"ContainerStarted","Data":"aa6999a2e89b62387e2c6216f171cb1b5186db82aebe8be3c2297ede7cd6f5a0"}
Oct 30 12:57:40.225511 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:40.225458    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m" podStartSLOduration=1.225443063 podStartE2EDuration="1.225443063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:40.223828881 +0000 UTC m=+755.270962834" watchObservedRunningTime="2026-10-30 12:57:40.225443063 +0000 UTC m=+755.272577015"
Oct 30 12:57:40.376651 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:40.376591    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-d8c8ccb-k7dz5"]
Oct 30 12:57:40.380349 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:40.380330    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-d8c8ccb-k7dz5"
Oct 30 12:57:40.383953 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:40.383927    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:40.395342 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:40.395315    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 30 12:57:40.395476 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:40.395408    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-gbh2n\""
Oct 30 12:57:40.395476 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:40.395453    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 12:57:40.401290 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:40.401208    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-d8c8ccb-k7dz5"]
Oct 30 12:57:40.553906 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:40.553806    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/878edd36-7a7d-4955-adf6-dec4d65714c2-metrics-certs\") pod \"ai-gateway-controller-d8c8ccb-k7dz5\" (UID: \"878edd36-7a7d-4955-adf6-dec4d65714c2\") " pod="opendatahub/ai-gateway-controller-d8c8ccb-k7dz5"
Oct 30 12:57:40.553906 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:40.553879    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbjqt\" (UniqueName: \"kubernetes.io/projected/878edd36-7a7d-4955-adf6-dec4d65714c2-kube-api-access-jbjqt\") pod \"ai-gateway-controller-d8c8ccb-k7dz5\" (UID: \"878edd36-7a7d-4955-adf6-dec4d65714c2\") " pod="opendatahub/ai-gateway-controller-d8c8ccb-k7dz5"
Oct 30 12:57:40.562799 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:40.562777    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 12:57:40.564408 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:40.564381    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7462m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 12:57:40.564522 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:40.564439    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m" podUID="3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 12:57:40.654918 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:40.654873    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/878edd36-7a7d-4955-adf6-dec4d65714c2-metrics-certs\") pod \"ai-gateway-controller-d8c8ccb-k7dz5\" (UID: \"878edd36-7a7d-4955-adf6-dec4d65714c2\") " pod="opendatahub/ai-gateway-controller-d8c8ccb-k7dz5"
Oct 30 12:57:40.655132 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:40.654966    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbjqt\" (UniqueName: \"kubernetes.io/projected/878edd36-7a7d-4955-adf6-dec4d65714c2-kube-api-access-jbjqt\") pod \"ai-gateway-controller-d8c8ccb-k7dz5\" (UID: \"878edd36-7a7d-4955-adf6-dec4d65714c2\") " pod="opendatahub/ai-gateway-controller-d8c8ccb-k7dz5"
Oct 30 12:57:40.657463 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:40.657432    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/878edd36-7a7d-4955-adf6-dec4d65714c2-metrics-certs\") pod \"ai-gateway-controller-d8c8ccb-k7dz5\" (UID: \"878edd36-7a7d-4955-adf6-dec4d65714c2\") " pod="opendatahub/ai-gateway-controller-d8c8ccb-k7dz5"
Oct 30 12:57:40.686425 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:40.686389    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbjqt\" (UniqueName: \"kubernetes.io/projected/878edd36-7a7d-4955-adf6-dec4d65714c2-kube-api-access-jbjqt\") pod \"ai-gateway-controller-d8c8ccb-k7dz5\" (UID: \"878edd36-7a7d-4955-adf6-dec4d65714c2\") " pod="opendatahub/ai-gateway-controller-d8c8ccb-k7dz5"
Oct 30 12:57:40.686761 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:40.686733    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:40.694735 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:40.694713    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-d8c8ccb-k7dz5"
Oct 30 12:57:41.022295 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:41.022190    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-d8c8ccb-k7dz5"]
Oct 30 12:57:41.183817 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:41.183781    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-d8c8ccb-k7dz5" event={"ID":"878edd36-7a7d-4955-adf6-dec4d65714c2","Type":"ContainerStarted","Data":"51b1cff867972cb19f7544b695ae0c7c84c13fd2d724f5bac190c16b9268934a"}
Oct 30 12:57:41.563492 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:41.563454    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7462m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 12:57:41.563924 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:41.563520    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m" podUID="3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 12:57:42.563909 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:42.563876    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7462m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 12:57:42.564367 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:42.563946    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m" podUID="3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 12:57:43.563629 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:43.563582    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7462m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 12:57:43.563827 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:43.563660    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m" podUID="3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 12:57:44.200500 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:44.200461    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-d8c8ccb-k7dz5" event={"ID":"878edd36-7a7d-4955-adf6-dec4d65714c2","Type":"ContainerStarted","Data":"0b62b32b26093a90a23bf902a9423dda2536ad54494d189d9daa2c914c134008"}
Oct 30 12:57:44.200919 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:44.200590    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-d8c8ccb-k7dz5"
Oct 30 12:57:44.249039 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:44.248985    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-d8c8ccb-k7dz5" podStartSLOduration=1.281877423 podStartE2EDuration="4.24896676s" totalImagesPullingTime="2.967089337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:44.247372872 +0000 UTC m=+759.294506825" watchObservedRunningTime="2026-10-30 12:57:44.24896676 +0000 UTC m=+759.296100712"
Oct 30 12:57:44.563886 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:44.563844    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7462m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 12:57:44.564090 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:44.563906    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m" podUID="3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 12:57:45.563586 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:45.563541    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7462m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 12:57:45.564073 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:45.563599    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m" podUID="3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 12:57:46.376186 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:46.376140    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:46.493779 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:46.493746    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:46.536301 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:46.536268    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-8lgfp"]
Oct 30 12:57:46.540170 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:46.540146    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8lgfp"
Oct 30 12:57:46.560431 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:46.560399    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 30 12:57:46.562718 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:46.562691    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7462m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 12:57:46.562870 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:46.562750    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m" podUID="3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 12:57:46.570550 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:46.570519    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-52sqw\""
Oct 30 12:57:46.572625 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:46.572584    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-8lgfp"]
Oct 30 12:57:46.619046 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:46.606479    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/097cd592-eba6-4113-9944-3cdbd57400f4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-8lgfp\" (UID: \"097cd592-eba6-4113-9944-3cdbd57400f4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-8lgfp"
Oct 30 12:57:46.619046 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:46.606536    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz6v5\" (UniqueName: \"kubernetes.io/projected/097cd592-eba6-4113-9944-3cdbd57400f4-kube-api-access-kz6v5\") pod \"payload-pre-processing-7b98bb678c-8lgfp\" (UID: \"097cd592-eba6-4113-9944-3cdbd57400f4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-8lgfp"
Oct 30 12:57:46.643978 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:46.643889    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-wrtq4"]
Oct 30 12:57:46.647600 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:46.647565    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-wrtq4"
Oct 30 12:57:46.668142 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:46.668110    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-wrtq4"]
Oct 30 12:57:46.707070 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:46.707032    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l2nj\" (UniqueName: \"kubernetes.io/projected/7568a93f-5fdb-444f-86e7-b600ab7c7a78-kube-api-access-6l2nj\") pod \"payload-processing-5f78687d5b-wrtq4\" (UID: \"7568a93f-5fdb-444f-86e7-b600ab7c7a78\") " pod="openshift-ingress/payload-processing-5f78687d5b-wrtq4"
Oct 30 12:57:46.707070 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:46.707072    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7568a93f-5fdb-444f-86e7-b600ab7c7a78-plugins-config-volume\") pod \"payload-processing-5f78687d5b-wrtq4\" (UID: \"7568a93f-5fdb-444f-86e7-b600ab7c7a78\") " pod="openshift-ingress/payload-processing-5f78687d5b-wrtq4"
Oct 30 12:57:46.707367 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:46.707149    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/097cd592-eba6-4113-9944-3cdbd57400f4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-8lgfp\" (UID: \"097cd592-eba6-4113-9944-3cdbd57400f4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-8lgfp"
Oct 30 12:57:46.707367 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:46.707195    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz6v5\" (UniqueName: \"kubernetes.io/projected/097cd592-eba6-4113-9944-3cdbd57400f4-kube-api-access-kz6v5\") pod \"payload-pre-processing-7b98bb678c-8lgfp\" (UID: \"097cd592-eba6-4113-9944-3cdbd57400f4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-8lgfp"
Oct 30 12:57:46.707897 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:46.707878    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/097cd592-eba6-4113-9944-3cdbd57400f4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-8lgfp\" (UID: \"097cd592-eba6-4113-9944-3cdbd57400f4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-8lgfp"
Oct 30 12:57:46.731253 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:46.731202    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz6v5\" (UniqueName: \"kubernetes.io/projected/097cd592-eba6-4113-9944-3cdbd57400f4-kube-api-access-kz6v5\") pod \"payload-pre-processing-7b98bb678c-8lgfp\" (UID: \"097cd592-eba6-4113-9944-3cdbd57400f4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-8lgfp"
Oct 30 12:57:46.808390 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:46.808354    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l2nj\" (UniqueName: \"kubernetes.io/projected/7568a93f-5fdb-444f-86e7-b600ab7c7a78-kube-api-access-6l2nj\") pod \"payload-processing-5f78687d5b-wrtq4\" (UID: \"7568a93f-5fdb-444f-86e7-b600ab7c7a78\") " pod="openshift-ingress/payload-processing-5f78687d5b-wrtq4"
Oct 30 12:57:46.808390 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:46.808396    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7568a93f-5fdb-444f-86e7-b600ab7c7a78-plugins-config-volume\") pod \"payload-processing-5f78687d5b-wrtq4\" (UID: \"7568a93f-5fdb-444f-86e7-b600ab7c7a78\") " pod="openshift-ingress/payload-processing-5f78687d5b-wrtq4"
Oct 30 12:57:46.809085 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:46.809064    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7568a93f-5fdb-444f-86e7-b600ab7c7a78-plugins-config-volume\") pod \"payload-processing-5f78687d5b-wrtq4\" (UID: \"7568a93f-5fdb-444f-86e7-b600ab7c7a78\") " pod="openshift-ingress/payload-processing-5f78687d5b-wrtq4"
Oct 30 12:57:46.831714 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:46.831683    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l2nj\" (UniqueName: \"kubernetes.io/projected/7568a93f-5fdb-444f-86e7-b600ab7c7a78-kube-api-access-6l2nj\") pod \"payload-processing-5f78687d5b-wrtq4\" (UID: \"7568a93f-5fdb-444f-86e7-b600ab7c7a78\") " pod="openshift-ingress/payload-processing-5f78687d5b-wrtq4"
Oct 30 12:57:46.857535 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:46.857496    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8lgfp"
Oct 30 12:57:46.964919 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:46.964872    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-wrtq4"
Oct 30 12:57:47.161335 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:47.161214    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-8lgfp"]
Oct 30 12:57:47.164701 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:57:47.164671    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod097cd592_eba6_4113_9944_3cdbd57400f4.slice/crio-a093c23c43da2f0548c82ed134ad73a398a8bf0a1b76111b6e1b7a655e63b0bd WatchSource:0}: Error finding container a093c23c43da2f0548c82ed134ad73a398a8bf0a1b76111b6e1b7a655e63b0bd: Status 404 returned error can't find the container with id a093c23c43da2f0548c82ed134ad73a398a8bf0a1b76111b6e1b7a655e63b0bd
Oct 30 12:57:47.216754 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:47.216713    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8lgfp" event={"ID":"097cd592-eba6-4113-9944-3cdbd57400f4","Type":"ContainerStarted","Data":"a093c23c43da2f0548c82ed134ad73a398a8bf0a1b76111b6e1b7a655e63b0bd"}
Oct 30 12:57:47.271598 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:47.271561    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-wrtq4"]
Oct 30 12:57:47.278974 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:57:47.278941    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7568a93f_5fdb_444f_86e7_b600ab7c7a78.slice/crio-b10e7b6d910801dccb82e3a533d79bc39db46eefe0ae58cdfdb54d372fd3c3f7 WatchSource:0}: Error finding container b10e7b6d910801dccb82e3a533d79bc39db46eefe0ae58cdfdb54d372fd3c3f7: Status 404 returned error can't find the container with id b10e7b6d910801dccb82e3a533d79bc39db46eefe0ae58cdfdb54d372fd3c3f7
Oct 30 12:57:48.226773 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:48.226727    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-wrtq4" event={"ID":"7568a93f-5fdb-444f-86e7-b600ab7c7a78","Type":"ContainerStarted","Data":"b10e7b6d910801dccb82e3a533d79bc39db46eefe0ae58cdfdb54d372fd3c3f7"}
Oct 30 12:57:48.606885 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:48.606406    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7462m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:57:48.606885 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:48.606502    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m" podUID="3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:49.607809 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:49.607758    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7462m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:57:49.608382 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:49.607836    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m" podUID="3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:50.240499 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:50.240391    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8lgfp" event={"ID":"097cd592-eba6-4113-9944-3cdbd57400f4","Type":"ContainerStarted","Data":"5f0472cf903a203b82baf6b2730f22468476c2059a5dc0f1b39ee181746daa04"}
Oct 30 12:57:50.240683 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:50.240510    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8lgfp"
Oct 30 12:57:50.242419 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:50.242386    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-wrtq4" event={"ID":"7568a93f-5fdb-444f-86e7-b600ab7c7a78","Type":"ContainerStarted","Data":"5cfd199badca2b56033e7f6fb06f7cae3d4baed3174f1fedb6bb66dba0913b6c"}
Oct 30 12:57:50.242561 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:50.242519    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-wrtq4"
Oct 30 12:57:50.281420 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:50.281359    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8lgfp" podStartSLOduration=1.552625832 podStartE2EDuration="4.28134252s" totalImagesPullingTime="2.728716688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:50.280234612 +0000 UTC m=+765.327368555" watchObservedRunningTime="2026-10-30 12:57:50.28134252 +0000 UTC m=+765.328476472"
Oct 30 12:57:50.322795 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:50.322720    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-wrtq4" podStartSLOduration=1.7522834010000001 podStartE2EDuration="4.322697444s" totalImagesPullingTime="2.570414043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:50.319678066 +0000 UTC m=+765.366812020" watchObservedRunningTime="2026-10-30 12:57:50.322697444 +0000 UTC m=+765.369831397"
Oct 30 12:57:50.649478 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:50.649429    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7462m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:57:50.650056 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:50.649510    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m" podUID="3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:57:51.046677 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:51.046593    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 12:57:51.046901 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:51.046869    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 12:57:51.047668 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:51.047650    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 12:57:52.491513 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:52.491482    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:57:54.304286 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.304247    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"]
Oct 30 12:57:54.307970 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.307942    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 12:57:54.345315 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.345277    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"]
Oct 30 12:57:54.369194 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.369163    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"]
Oct 30 12:57:54.373010 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.372983    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 12:57:54.384914 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.384889    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-7mx6s\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 12:57:54.385035 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.384936    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7mx6s\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 12:57:54.385035 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.384952    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zghds\" (UniqueName: \"kubernetes.io/projected/9c24a948-3812-4568-a7a0-e58641a7427f-kube-api-access-zghds\") pod \"maas-default-gateway-openshift-default-58c6585447-7mx6s\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 12:57:54.385171 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.385151    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-7mx6s\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 12:57:54.385215 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.385182    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7mx6s\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 12:57:54.385215 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.385200    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c24a948-3812-4568-a7a0-e58641a7427f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-7mx6s\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 12:57:54.385308 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.385271    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-7mx6s\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 12:57:54.385308 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.385292    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c24a948-3812-4568-a7a0-e58641a7427f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-7mx6s\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 12:57:54.385399 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.385308    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c24a948-3812-4568-a7a0-e58641a7427f-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-7mx6s\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 12:57:54.401864 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.401835    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"]
Oct 30 12:57:54.486839 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.486800    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c24a948-3812-4568-a7a0-e58641a7427f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-7mx6s\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 12:57:54.487043 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.486851    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-w27kj\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 12:57:54.487043 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.486883    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c24a948-3812-4568-a7a0-e58641a7427f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-7mx6s\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 12:57:54.487043 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.486916    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/287ea8cd-7fc1-4967-a291-adf6973f5967-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-w27kj\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 12:57:54.487043 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.486946    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k76nq\" (UniqueName: \"kubernetes.io/projected/287ea8cd-7fc1-4967-a291-adf6973f5967-kube-api-access-k76nq\") pod \"maas-default-gateway-openshift-default-58c6585447-w27kj\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 12:57:54.487298 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.487133    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-w27kj\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 12:57:54.487298 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.487180    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7mx6s\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 12:57:54.487298 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.487253    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zghds\" (UniqueName: \"kubernetes.io/projected/9c24a948-3812-4568-a7a0-e58641a7427f-kube-api-access-zghds\") pod \"maas-default-gateway-openshift-default-58c6585447-7mx6s\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 12:57:54.487454 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.487299    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/287ea8cd-7fc1-4967-a291-adf6973f5967-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-w27kj\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 12:57:54.487454 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.487325    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/287ea8cd-7fc1-4967-a291-adf6973f5967-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-w27kj\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 12:57:54.487454 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.487356    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-w27kj\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 12:57:54.487454 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.487388    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-w27kj\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 12:57:54.487615 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.487503    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7mx6s\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 12:57:54.487615 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.487525    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7mx6s\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 12:57:54.487615 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.487547    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c24a948-3812-4568-a7a0-e58641a7427f-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-7mx6s\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 12:57:54.487615 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.487597    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-7mx6s\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 12:57:54.487615 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.487605    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c24a948-3812-4568-a7a0-e58641a7427f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-7mx6s\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 12:57:54.487842 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.487623    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-7mx6s\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 12:57:54.487842 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.487681    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-7mx6s\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 12:57:54.487842 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.487714    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-w27kj\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 12:57:54.487842 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.487761    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7mx6s\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 12:57:54.487985 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.487965    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-7mx6s\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 12:57:54.488113 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.488092    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-7mx6s\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 12:57:54.489631 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.489607    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-7mx6s\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 12:57:54.489747 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.489684    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c24a948-3812-4568-a7a0-e58641a7427f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-7mx6s\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 12:57:54.510462 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.510434    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c24a948-3812-4568-a7a0-e58641a7427f-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-7mx6s\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 12:57:54.510577 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.510507    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zghds\" (UniqueName: \"kubernetes.io/projected/9c24a948-3812-4568-a7a0-e58641a7427f-kube-api-access-zghds\") pod \"maas-default-gateway-openshift-default-58c6585447-7mx6s\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 12:57:54.588678 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.588635    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-w27kj\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 12:57:54.588678 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.588684    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-w27kj\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 12:57:54.588893 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.588732    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-w27kj\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 12:57:54.588893 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.588774    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-w27kj\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 12:57:54.588893 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.588797    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/287ea8cd-7fc1-4967-a291-adf6973f5967-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-w27kj\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 12:57:54.588893 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.588824    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k76nq\" (UniqueName: \"kubernetes.io/projected/287ea8cd-7fc1-4967-a291-adf6973f5967-kube-api-access-k76nq\") pod \"maas-default-gateway-openshift-default-58c6585447-w27kj\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 12:57:54.588893 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.588853    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-w27kj\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 12:57:54.588893 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.588876    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/287ea8cd-7fc1-4967-a291-adf6973f5967-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-w27kj\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 12:57:54.589240 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.588903    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/287ea8cd-7fc1-4967-a291-adf6973f5967-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-w27kj\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 12:57:54.589240 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.589041    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-w27kj\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 12:57:54.589240 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.589095    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-w27kj\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 12:57:54.589240 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.589150    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-w27kj\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 12:57:54.589441 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.589339    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-w27kj\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 12:57:54.589571 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.589554    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/287ea8cd-7fc1-4967-a291-adf6973f5967-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-w27kj\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 12:57:54.590989 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.590967    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-w27kj\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 12:57:54.591390 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.591371    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/287ea8cd-7fc1-4967-a291-adf6973f5967-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-w27kj\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 12:57:54.614692 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.614665    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/287ea8cd-7fc1-4967-a291-adf6973f5967-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-w27kj\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 12:57:54.614948 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.614931    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k76nq\" (UniqueName: \"kubernetes.io/projected/287ea8cd-7fc1-4967-a291-adf6973f5967-kube-api-access-k76nq\") pod \"maas-default-gateway-openshift-default-58c6585447-w27kj\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 12:57:54.620775 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.620760    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 12:57:54.685503 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.685474    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 12:57:54.942184 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.942106    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"]
Oct 30 12:57:54.946572 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:57:54.946546    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c24a948_3812_4568_a7a0_e58641a7427f.slice/crio-80db4a4a138695eaf976a0507c460dd7c72e140144dde65d20d89c4c276020d2 WatchSource:0}: Error finding container 80db4a4a138695eaf976a0507c460dd7c72e140144dde65d20d89c4c276020d2: Status 404 returned error can't find the container with id 80db4a4a138695eaf976a0507c460dd7c72e140144dde65d20d89c4c276020d2
Oct 30 12:57:54.948657 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.948629    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:57:54.948727 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.948690    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:57:54.948768 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.948757    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:57:54.996798 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:54.996771    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"]
Oct 30 12:57:55.001867 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:57:55.001835    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod287ea8cd_7fc1_4967_a291_adf6973f5967.slice/crio-6e9651e2d28d618d0ad35d88509b54c9b88a3192341d0e5a1c199f7b9ac2beb6 WatchSource:0}: Error finding container 6e9651e2d28d618d0ad35d88509b54c9b88a3192341d0e5a1c199f7b9ac2beb6: Status 404 returned error can't find the container with id 6e9651e2d28d618d0ad35d88509b54c9b88a3192341d0e5a1c199f7b9ac2beb6
Oct 30 12:57:55.004617 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:55.004582    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:57:55.004720 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:55.004662    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:57:55.004720 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:55.004712    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:57:55.208287 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:55.208197    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-d8c8ccb-k7dz5"
Oct 30 12:57:55.269339 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:55.269275    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s" event={"ID":"9c24a948-3812-4568-a7a0-e58641a7427f","Type":"ContainerStarted","Data":"457f928f5e6b51b4d2611a04116db827bc438cbe60f1ad787f08093538e028f6"}
Oct 30 12:57:55.269503 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:55.269348    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s" event={"ID":"9c24a948-3812-4568-a7a0-e58641a7427f","Type":"ContainerStarted","Data":"80db4a4a138695eaf976a0507c460dd7c72e140144dde65d20d89c4c276020d2"}
Oct 30 12:57:55.272704 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:55.272666    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj" event={"ID":"287ea8cd-7fc1-4967-a291-adf6973f5967","Type":"ContainerStarted","Data":"8811fa1f9883d0d3b4048ba57103e1e34881d03683a3c5ee38215dfba4b6fc35"}
Oct 30 12:57:55.272859 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:55.272738    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj" event={"ID":"287ea8cd-7fc1-4967-a291-adf6973f5967","Type":"ContainerStarted","Data":"6e9651e2d28d618d0ad35d88509b54c9b88a3192341d0e5a1c199f7b9ac2beb6"}
Oct 30 12:57:55.316947 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:55.316888    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s" podStartSLOduration=1.316868805 podStartE2EDuration="1.316868805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:55.315824808 +0000 UTC m=+770.362958762" watchObservedRunningTime="2026-10-30 12:57:55.316868805 +0000 UTC m=+770.364002761"
Oct 30 12:57:55.360746 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:55.360690    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj" podStartSLOduration=1.360671376 podStartE2EDuration="1.360671376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:55.360424227 +0000 UTC m=+770.407558178" watchObservedRunningTime="2026-10-30 12:57:55.360671376 +0000 UTC m=+770.407805331"
Oct 30 12:57:55.635245 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:55.634392    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 12:57:55.686559 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:55.686528    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 12:57:56.626071 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:56.626030    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7mx6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 12:57:56.626713 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:56.626682    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s" podUID="9c24a948-3812-4568-a7a0-e58641a7427f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 12:57:56.687765 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:56.687729    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-w27kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 12:57:56.687933 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:56.687799    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj" podUID="287ea8cd-7fc1-4967-a291-adf6973f5967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 12:57:57.630262 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:57.629714    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7mx6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 12:57:57.630262 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:57.629785    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s" podUID="9c24a948-3812-4568-a7a0-e58641a7427f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 12:57:57.686546 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:57.686508    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-w27kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 12:57:57.686693 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:57.686569    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj" podUID="287ea8cd-7fc1-4967-a291-adf6973f5967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 12:57:58.624701 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:58.624660    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7mx6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 12:57:58.624991 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:58.624962    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s" podUID="9c24a948-3812-4568-a7a0-e58641a7427f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 12:57:58.686750 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:58.686716    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-w27kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 12:57:58.687268 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:58.687208    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj" podUID="287ea8cd-7fc1-4967-a291-adf6973f5967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 12:57:59.006576 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:57:59.006490    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 12:57:59.621789 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:59.621750    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7mx6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 12:57:59.622093 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:59.622051    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s" podUID="9c24a948-3812-4568-a7a0-e58641a7427f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 12:57:59.686112 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:59.686077    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-w27kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 12:57:59.686356 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:57:59.686332    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj" podUID="287ea8cd-7fc1-4967-a291-adf6973f5967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 12:58:00.624729 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:00.624685    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7mx6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 12:58:00.625408 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:00.625369    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s" podUID="9c24a948-3812-4568-a7a0-e58641a7427f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 12:58:00.685894 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:00.685853    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-w27kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 12:58:00.686049 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:00.685909    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj" podUID="287ea8cd-7fc1-4967-a291-adf6973f5967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 12:58:01.250506 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:01.250475    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-wrtq4"
Oct 30 12:58:01.250819 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:01.250540    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-8lgfp"
Oct 30 12:58:01.621292 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:01.621243    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7mx6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 12:58:01.621504 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:01.621340    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s" podUID="9c24a948-3812-4568-a7a0-e58641a7427f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 12:58:01.686754 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:01.686718    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-w27kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 12:58:01.687162 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:01.686774    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj" podUID="287ea8cd-7fc1-4967-a291-adf6973f5967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 12:58:02.598386 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:02.598353    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:03.430266 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:03.430198    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 12:58:03.430970 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:03.430713    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 12:58:03.432015 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:03.431985    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 12:58:03.490235 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:03.490188    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 12:58:03.490599 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:03.490572    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 12:58:03.492646 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:03.492624    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 12:58:10.857615 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:10.857575    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-d8c8ccb-k7dz5"]
Oct 30 12:58:10.858168 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:10.857894    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-d8c8ccb-k7dz5" podUID="878edd36-7a7d-4955-adf6-dec4d65714c2" containerName="manager" containerID="cri-o://0b62b32b26093a90a23bf902a9423dda2536ad54494d189d9daa2c914c134008" gracePeriod=30
Oct 30 12:58:11.159344 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:11.159306    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-d8c8ccb-k7dz5"
Oct 30 12:58:11.296603 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:11.296565    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/878edd36-7a7d-4955-adf6-dec4d65714c2-kube-api-access-jbjqt\" (UniqueName: \"kubernetes.io/projected/878edd36-7a7d-4955-adf6-dec4d65714c2-kube-api-access-jbjqt\") pod \"878edd36-7a7d-4955-adf6-dec4d65714c2\" (UID: \"878edd36-7a7d-4955-adf6-dec4d65714c2\") "
Oct 30 12:58:11.296795 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:11.296623    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/878edd36-7a7d-4955-adf6-dec4d65714c2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/878edd36-7a7d-4955-adf6-dec4d65714c2-metrics-certs\") pod \"878edd36-7a7d-4955-adf6-dec4d65714c2\" (UID: \"878edd36-7a7d-4955-adf6-dec4d65714c2\") "
Oct 30 12:58:11.298825 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:11.298790    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/878edd36-7a7d-4955-adf6-dec4d65714c2-metrics-certs" pod "878edd36-7a7d-4955-adf6-dec4d65714c2" (UID: "878edd36-7a7d-4955-adf6-dec4d65714c2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:58:11.298825 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:11.298796    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/878edd36-7a7d-4955-adf6-dec4d65714c2-kube-api-access-jbjqt" pod "878edd36-7a7d-4955-adf6-dec4d65714c2" (UID: "878edd36-7a7d-4955-adf6-dec4d65714c2"). InnerVolumeSpecName "kube-api-access-jbjqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:58:11.358109 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:11.358074    4544 generic.go:356] "Generic (PLEG): container finished" podID="878edd36-7a7d-4955-adf6-dec4d65714c2" containerID="0b62b32b26093a90a23bf902a9423dda2536ad54494d189d9daa2c914c134008" exitCode=0
Oct 30 12:58:11.358298 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:11.358126    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-d8c8ccb-k7dz5"
Oct 30 12:58:11.358298 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:11.358158    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-d8c8ccb-k7dz5" event={"ID":"878edd36-7a7d-4955-adf6-dec4d65714c2","Type":"ContainerDied","Data":"0b62b32b26093a90a23bf902a9423dda2536ad54494d189d9daa2c914c134008"}
Oct 30 12:58:11.358298 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:11.358243    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-d8c8ccb-k7dz5" event={"ID":"878edd36-7a7d-4955-adf6-dec4d65714c2","Type":"ContainerDied","Data":"51b1cff867972cb19f7544b695ae0c7c84c13fd2d724f5bac190c16b9268934a"}
Oct 30 12:58:11.358298 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:11.358245    4544 scope.go:122] "RemoveContainer" containerID="0b62b32b26093a90a23bf902a9423dda2536ad54494d189d9daa2c914c134008"
Oct 30 12:58:11.367973 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:11.367955    4544 scope.go:122] "RemoveContainer" containerID="0b62b32b26093a90a23bf902a9423dda2536ad54494d189d9daa2c914c134008"
Oct 30 12:58:11.368307 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:58:11.368277    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b62b32b26093a90a23bf902a9423dda2536ad54494d189d9daa2c914c134008\": container with ID starting with 0b62b32b26093a90a23bf902a9423dda2536ad54494d189d9daa2c914c134008 not found: ID does not exist" containerID="0b62b32b26093a90a23bf902a9423dda2536ad54494d189d9daa2c914c134008"
Oct 30 12:58:11.368406 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:11.368315    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b62b32b26093a90a23bf902a9423dda2536ad54494d189d9daa2c914c134008"} err="failed to get container status \"0b62b32b26093a90a23bf902a9423dda2536ad54494d189d9daa2c914c134008\": rpc error: code = NotFound desc = could not find container \"0b62b32b26093a90a23bf902a9423dda2536ad54494d189d9daa2c914c134008\": container with ID starting with 0b62b32b26093a90a23bf902a9423dda2536ad54494d189d9daa2c914c134008 not found: ID does not exist"
Oct 30 12:58:11.398127 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:11.398043    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbjqt\" (UniqueName: \"kubernetes.io/projected/878edd36-7a7d-4955-adf6-dec4d65714c2-kube-api-access-jbjqt\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:58:11.398127 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:11.398071    4544 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/878edd36-7a7d-4955-adf6-dec4d65714c2-metrics-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 12:58:11.409265 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:11.409230    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-d8c8ccb-k7dz5"]
Oct 30 12:58:11.415786 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:11.415754    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-d8c8ccb-k7dz5"]
Oct 30 12:58:12.611095 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:12.611058    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:13.084433 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:13.084400    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="878edd36-7a7d-4955-adf6-dec4d65714c2" path="/var/lib/kubelet/pods/878edd36-7a7d-4955-adf6-dec4d65714c2/volumes"
Oct 30 12:58:13.148483 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:13.148450    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:14.886915 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:14.886880    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:15.027936 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:15.027900    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:16.528843 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:16.528809    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:16.656566 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:16.656534    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:29.004736 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:58:29.004699    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 12:58:38.120688 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.120657    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:38.231750 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.231720    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:38.725919 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.725888    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:38.803898 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.803868    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"]
Oct 30 12:58:38.804681 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.804648    4544 cpu_manager.go:414] "containerMap: removing container" podUID="878edd36-7a7d-4955-adf6-dec4d65714c2" containerName="manager"
Oct 30 12:58:38.804681 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.804675    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="878edd36-7a7d-4955-adf6-dec4d65714c2" containerName="manager"
Oct 30 12:58:38.804957 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.804878    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="878edd36-7a7d-4955-adf6-dec4d65714c2" containerName="manager"
Oct 30 12:58:38.808460 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.808439    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:58:38.826455 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.826432    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-dd90cb-openshift-default-dockercfg-9nwsm\""
Oct 30 12:58:38.838969 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.838946    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"]
Oct 30 12:58:38.854045 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.854017    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcjbm\" (UniqueName: \"kubernetes.io/projected/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-kube-api-access-bcjbm\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:58:38.854327 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.854061    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istio-data\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:58:38.854327 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.854088    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-credential-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:58:38.854327 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.854237    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-workload-certs\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:58:38.854561 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.854374    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istiod-ca-cert\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:58:38.854561 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.854433    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istio-podinfo\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:58:38.854561 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.854468    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istio-envoy\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:58:38.854704 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.854565    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istio-token\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:58:38.854704 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.854683    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-workload-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:58:38.912736 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.912696    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:38.955690 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.955655    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istiod-ca-cert\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:58:38.955690 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.955690    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istio-podinfo\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:58:38.955961 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.955715    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istio-envoy\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:58:38.955961 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.955766    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istio-token\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:58:38.955961 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.955786    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-workload-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:58:38.955961 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.955818    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcjbm\" (UniqueName: \"kubernetes.io/projected/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-kube-api-access-bcjbm\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:58:38.955961 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.955842    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istio-data\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:58:38.955961 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.955877    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-credential-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:58:38.955961 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.955902    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-workload-certs\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:58:38.956391 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.956272    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-workload-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:58:38.956391 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.956303    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-workload-certs\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:58:38.956510 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.956444    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istio-data\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:58:38.956510 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.956464    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-credential-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:58:38.956639 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.956612    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istiod-ca-cert\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:58:38.958184 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.958166    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istio-envoy\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:58:38.958429 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.958411    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istio-podinfo\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:58:38.981433 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.981367    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcjbm\" (UniqueName: \"kubernetes.io/projected/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-kube-api-access-bcjbm\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:58:38.986093 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.986067    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istio-token\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:58:38.997912 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:38.997882    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"]
Oct 30 12:58:39.004799 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.004778    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:58:39.029059 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.029018    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-7a3603-openshift-default-dockercfg-dh98z\""
Oct 30 12:58:39.033993 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.033966    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"]
Oct 30 12:58:39.056713 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.056688    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-workload-certs\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:58:39.056864 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.056720    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-workload-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:58:39.056864 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.056745    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-istio-envoy\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:58:39.056978 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.056905    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3381ce8c-5918-4339-aa75-dd37e7723ee8-istio-podinfo\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:58:39.056978 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.056961    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3381ce8c-5918-4339-aa75-dd37e7723ee8-istiod-ca-cert\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:58:39.057084 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.057007    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3381ce8c-5918-4339-aa75-dd37e7723ee8-istio-token\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:58:39.057138 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.057098    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-credential-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:58:39.057199 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.057137    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g77n\" (UniqueName: \"kubernetes.io/projected/3381ce8c-5918-4339-aa75-dd37e7723ee8-kube-api-access-6g77n\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:58:39.057199 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.057185    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-istio-data\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:58:39.083088 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.083058    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:39.124662 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.124639    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:58:39.158643 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.158613    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-workload-certs\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:58:39.158797 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.158651    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-workload-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:58:39.158797 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.158678    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-istio-envoy\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:58:39.158797 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.158723    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3381ce8c-5918-4339-aa75-dd37e7723ee8-istio-podinfo\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:58:39.158797 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.158741    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3381ce8c-5918-4339-aa75-dd37e7723ee8-istiod-ca-cert\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:58:39.158797 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.158776    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3381ce8c-5918-4339-aa75-dd37e7723ee8-istio-token\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:58:39.158797 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.158800    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-credential-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:58:39.159140 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.158820    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g77n\" (UniqueName: \"kubernetes.io/projected/3381ce8c-5918-4339-aa75-dd37e7723ee8-kube-api-access-6g77n\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:58:39.159140 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.158849    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-istio-data\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:58:39.159277 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.159169    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-istio-data\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:58:39.159394 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.159377    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-workload-certs\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:58:39.159556 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.159540    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-workload-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:58:39.160981 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.160409    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3381ce8c-5918-4339-aa75-dd37e7723ee8-istiod-ca-cert\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:58:39.160981 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.160683    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-credential-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:58:39.161774 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.161756    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-istio-envoy\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:58:39.163440 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.163420    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3381ce8c-5918-4339-aa75-dd37e7723ee8-istio-podinfo\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:58:39.197663 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.197584    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3381ce8c-5918-4339-aa75-dd37e7723ee8-istio-token\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:58:39.197910 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.197887    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g77n\" (UniqueName: \"kubernetes.io/projected/3381ce8c-5918-4339-aa75-dd37e7723ee8-kube-api-access-6g77n\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:58:39.219408 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.219381    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:39.313833 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.313736    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"]
Oct 30 12:58:39.319814 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.319785    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:58:39.321013 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.320974    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:58:39.339324 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.339296    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-67f56e-openshift-default-dockercfg-vvpqd\""
Oct 30 12:58:39.349411 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.349371    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"]
Oct 30 12:58:39.360462 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.360438    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6lvs\" (UniqueName: \"kubernetes.io/projected/1e40797f-db7b-4502-b06c-bb8b419e52b9-kube-api-access-t6lvs\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:58:39.360609 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.360494    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e40797f-db7b-4502-b06c-bb8b419e52b9-istio-podinfo\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:58:39.360609 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.360531    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-istio-data\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:58:39.360739 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.360631    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-istio-envoy\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:58:39.360739 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.360662    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-credential-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:58:39.360739 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.360698    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-workload-certs\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:58:39.360739 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.360724    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e40797f-db7b-4502-b06c-bb8b419e52b9-istio-token\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:58:39.360938 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.360761    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e40797f-db7b-4502-b06c-bb8b419e52b9-istiod-ca-cert\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:58:39.360938 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.360809    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-workload-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:58:39.364916 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.364894    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:39.453809 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.453783    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"]
Oct 30 12:58:39.456719 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:58:39.456690    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41b0db3d_0f0b_4791_aca0_0cd31a7c78f6.slice/crio-5f9bebd29999c575aabd41af53b9cfb2bbed8a3608ebcf80f275a122ad7462b1 WatchSource:0}: Error finding container 5f9bebd29999c575aabd41af53b9cfb2bbed8a3608ebcf80f275a122ad7462b1: Status 404 returned error can't find the container with id 5f9bebd29999c575aabd41af53b9cfb2bbed8a3608ebcf80f275a122ad7462b1
Oct 30 12:58:39.459014 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.458979    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:58:39.459092 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.459041    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:58:39.459092 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.459075    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:58:39.462257 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.462234    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-istio-envoy\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:58:39.462372 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.462279    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-credential-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:58:39.462372 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.462320    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-workload-certs\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:58:39.462372 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.462348    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e40797f-db7b-4502-b06c-bb8b419e52b9-istio-token\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:58:39.462533 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.462386    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e40797f-db7b-4502-b06c-bb8b419e52b9-istiod-ca-cert\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:58:39.462533 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.462434    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-workload-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:58:39.462533 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.462482    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6lvs\" (UniqueName: \"kubernetes.io/projected/1e40797f-db7b-4502-b06c-bb8b419e52b9-kube-api-access-t6lvs\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:58:39.462686 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.462531    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e40797f-db7b-4502-b06c-bb8b419e52b9-istio-podinfo\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:58:39.462686 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.462571    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-istio-data\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:58:39.462793 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.462668    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-credential-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:58:39.462938 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.462920    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-istio-data\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:58:39.463047 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.462978    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-workload-certs\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:58:39.463047 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.462996    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-workload-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:58:39.463701 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.463680    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e40797f-db7b-4502-b06c-bb8b419e52b9-istiod-ca-cert\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:58:39.465352 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.465334    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e40797f-db7b-4502-b06c-bb8b419e52b9-istio-podinfo\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:58:39.465434 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.465351    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-istio-envoy\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:58:39.498067 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.498029    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455" event={"ID":"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6","Type":"ContainerStarted","Data":"5f9bebd29999c575aabd41af53b9cfb2bbed8a3608ebcf80f275a122ad7462b1"}
Oct 30 12:58:39.512080 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.512042    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6lvs\" (UniqueName: \"kubernetes.io/projected/1e40797f-db7b-4502-b06c-bb8b419e52b9-kube-api-access-t6lvs\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:58:39.512626 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.512605    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e40797f-db7b-4502-b06c-bb8b419e52b9-istio-token\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:58:39.541125 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.541088    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:39.637756 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.637469    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:58:39.683664 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.679157    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"]
Oct 30 12:58:39.724297 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.708552    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"]
Oct 30 12:58:39.724297 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.717367    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:39.724297 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.718889    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:58:39.724297 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.719326    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:58:39.724297 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.719419    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:58:39.724297 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.719464    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:58:39.781239 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.776269    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-c90b9e-openshift-default-dockercfg-6pcbf\""
Oct 30 12:58:39.781239 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.777055    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istiod-ca-cert\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:58:39.781239 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.777097    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istio-podinfo\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:58:39.781239 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.777136    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-workload-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:58:39.781239 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.777180    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istio-token\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:58:39.788292 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.781728    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-credential-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:58:39.788292 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.781797    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-workload-certs\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:58:39.788292 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.781842    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istio-data\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:58:39.788292 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.781903    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44tv2\" (UniqueName: \"kubernetes.io/projected/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-kube-api-access-44tv2\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:58:39.788292 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.781946    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istio-envoy\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:58:39.839659 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.839624    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"]
Oct 30 12:58:39.863157 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.863115    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:39.882880 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.882842    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istiod-ca-cert\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:58:39.883031 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.882897    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istio-podinfo\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:58:39.883031 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.882925    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-workload-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:58:39.883031 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.882960    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istio-token\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:58:39.883031 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.882994    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-credential-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:58:39.883031 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.883024    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-workload-certs\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:58:39.883330 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.883055    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istio-data\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:58:39.883330 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.883106    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44tv2\" (UniqueName: \"kubernetes.io/projected/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-kube-api-access-44tv2\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:58:39.883330 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.883144    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istio-envoy\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:58:39.885961 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.885891    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istiod-ca-cert\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:58:39.886638 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.886615    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-workload-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:58:39.887035 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.887016    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-workload-certs\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:58:39.887397 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.887379    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-credential-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:58:39.887739 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.887720    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istio-data\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:58:39.892355 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.891113    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istio-podinfo\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:58:39.899654 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.899630    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istio-envoy\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:58:39.908625 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.908601    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istio-token\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:58:39.912038 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:39.912017    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44tv2\" (UniqueName: \"kubernetes.io/projected/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-kube-api-access-44tv2\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:58:40.094174 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:40.094142    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:58:40.128802 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:40.128768    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"]
Oct 30 12:58:40.138849 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:58:40.138815    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e40797f_db7b_4502_b06c_bb8b419e52b9.slice/crio-8df7c8d982968c65d2668f56c0ecd8d5097aa10e4be9a11bf39e36ce6009c0ac WatchSource:0}: Error finding container 8df7c8d982968c65d2668f56c0ecd8d5097aa10e4be9a11bf39e36ce6009c0ac: Status 404 returned error can't find the container with id 8df7c8d982968c65d2668f56c0ecd8d5097aa10e4be9a11bf39e36ce6009c0ac
Oct 30 12:58:40.142744 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:40.142420    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:58:40.142744 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:40.142482    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:58:40.142744 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:40.142510    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:58:40.329702 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:40.329617    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:40.417563 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:40.417476    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"]
Oct 30 12:58:40.422640 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:58:40.422607    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb72c176a_dc4d_4b4e_82e9_b1e15fceaed3.slice/crio-2966b83548118af0b63eea1c965c32210721b021a4f1ee51ad8dce04d4ca0c75 WatchSource:0}: Error finding container 2966b83548118af0b63eea1c965c32210721b021a4f1ee51ad8dce04d4ca0c75: Status 404 returned error can't find the container with id 2966b83548118af0b63eea1c965c32210721b021a4f1ee51ad8dce04d4ca0c75
Oct 30 12:58:40.425116 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:40.425078    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:58:40.425195 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:40.425161    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:58:40.425263 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:40.425200    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:58:40.495099 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:40.495069    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:40.505892 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:40.505857    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz" event={"ID":"3381ce8c-5918-4339-aa75-dd37e7723ee8","Type":"ContainerStarted","Data":"76d71583d1b23504e34a005ae7664b45d7effd76ef14cfabba1f253e5c7bee59"}
Oct 30 12:58:40.506053 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:40.505924    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz" event={"ID":"3381ce8c-5918-4339-aa75-dd37e7723ee8","Type":"ContainerStarted","Data":"7b6292574bce6f1aeba42c62bcf6be45710b1617da3434e42f162719e2e56636"}
Oct 30 12:58:40.508115 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:40.508083    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4" event={"ID":"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3","Type":"ContainerStarted","Data":"4bbfd2bb3e5cde47f000197b8d0e4614382d950ec815493d55462c8b59f9206d"}
Oct 30 12:58:40.508249 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:40.508137    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4" event={"ID":"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3","Type":"ContainerStarted","Data":"2966b83548118af0b63eea1c965c32210721b021a4f1ee51ad8dce04d4ca0c75"}
Oct 30 12:58:40.509447 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:40.509426    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455" event={"ID":"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6","Type":"ContainerStarted","Data":"ae17880baa1f57fc9fb2a30b8f7dc4d53366af63e088ac3e0303490b44cd49e8"}
Oct 30 12:58:40.511119 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:40.511098    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v" event={"ID":"1e40797f-db7b-4502-b06c-bb8b419e52b9","Type":"ContainerStarted","Data":"ceeaed7d2a60a6a4a239bfbe002009ebb873cc0df96aaa62c30588e4636c020f"}
Oct 30 12:58:40.511200 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:40.511134    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v" event={"ID":"1e40797f-db7b-4502-b06c-bb8b419e52b9","Type":"ContainerStarted","Data":"8df7c8d982968c65d2668f56c0ecd8d5097aa10e4be9a11bf39e36ce6009c0ac"}
Oct 30 12:58:40.558929 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:40.558867    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz" podStartSLOduration=2.558848124 podStartE2EDuration="2.558848124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:40.555547164 +0000 UTC m=+815.602681118" watchObservedRunningTime="2026-10-30 12:58:40.558848124 +0000 UTC m=+815.605982078"
Oct 30 12:58:40.609934 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:40.609886    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v" podStartSLOduration=1.609874246 podStartE2EDuration="1.609874246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:40.609383722 +0000 UTC m=+815.656517675" watchObservedRunningTime="2026-10-30 12:58:40.609874246 +0000 UTC m=+815.657008198"
Oct 30 12:58:40.638516 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:40.638490    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:58:40.655324 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:40.655265    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455" podStartSLOduration=2.655247999 podStartE2EDuration="2.655247999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:40.652802096 +0000 UTC m=+815.699936051" watchObservedRunningTime="2026-10-30 12:58:40.655247999 +0000 UTC m=+815.702381949"
Oct 30 12:58:40.714071 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:40.713957    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4" podStartSLOduration=1.713938008 podStartE2EDuration="1.713938008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:40.709204538 +0000 UTC m=+815.756338494" watchObservedRunningTime="2026-10-30 12:58:40.713938008 +0000 UTC m=+815.761071962"
Oct 30 12:58:41.094975 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:41.094938    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:58:41.101137 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:41.101107    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:58:41.125288 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:41.125263    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:58:41.129988 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:41.129968    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:58:41.321723 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:41.321689    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:58:41.327259 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:41.327235    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:58:41.517240 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:41.517136    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:58:41.517240 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:41.517193    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:58:41.517467 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:41.517278    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:58:41.517467 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:41.517338    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:58:41.517467 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:41.517360    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:58:41.518123 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:41.518101    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:58:41.645729 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:41.645699    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:58:42.520795 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:42.520756    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:58:42.521290 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:42.521273    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:58:42.751005 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:42.750976    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:42.912796 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:42.912763    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:43.094939 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:43.094903    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:43.319279 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:43.319164    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:43.500977 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:43.500943    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:43.658787 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:43.658746    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:43.980485 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:43.980404    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:44.194078 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:44.194045    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:44.348296 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:44.348260    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:44.488794 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:44.488763    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:45.124796 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:45.124759    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:45.267071 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:45.267038    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:45.277837 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:45.277733    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4"]
Oct 30 12:58:45.281750 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:45.281734    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4"
Oct 30 12:58:45.310493 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:45.310468    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-09db16\""
Oct 30 12:58:45.310640 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:45.310491    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-09db16-dockercfg-mjg7j\""
Oct 30 12:58:45.313053 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:45.313031    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4"]
Oct 30 12:58:45.347512 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:45.347473    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzdk8\" (UniqueName: \"kubernetes.io/projected/fa3df092-b18c-47fb-8793-90f9869a1078-kube-api-access-fzdk8\") pod \"payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4\" (UID: \"fa3df092-b18c-47fb-8793-90f9869a1078\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4"
Oct 30 12:58:45.347663 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:45.347587    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa3df092-b18c-47fb-8793-90f9869a1078-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4\" (UID: \"fa3df092-b18c-47fb-8793-90f9869a1078\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4"
Oct 30 12:58:45.449293 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:45.448862    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzdk8\" (UniqueName: \"kubernetes.io/projected/fa3df092-b18c-47fb-8793-90f9869a1078-kube-api-access-fzdk8\") pod \"payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4\" (UID: \"fa3df092-b18c-47fb-8793-90f9869a1078\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4"
Oct 30 12:58:45.449293 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:45.448933    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa3df092-b18c-47fb-8793-90f9869a1078-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4\" (UID: \"fa3df092-b18c-47fb-8793-90f9869a1078\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4"
Oct 30 12:58:45.449806 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:45.449783    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa3df092-b18c-47fb-8793-90f9869a1078-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4\" (UID: \"fa3df092-b18c-47fb-8793-90f9869a1078\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4"
Oct 30 12:58:45.478786 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:45.478757    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzdk8\" (UniqueName: \"kubernetes.io/projected/fa3df092-b18c-47fb-8793-90f9869a1078-kube-api-access-fzdk8\") pod \"payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4\" (UID: \"fa3df092-b18c-47fb-8793-90f9869a1078\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4"
Oct 30 12:58:45.595299 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:45.595265    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4"
Oct 30 12:58:45.893008 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:45.892973    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4"]
Oct 30 12:58:45.896847 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:58:45.896820    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa3df092_b18c_47fb_8793_90f9869a1078.slice/crio-821861ec49d958e9af2b872c0b60a4cae3c68833574ef71aacb20069c672df68 WatchSource:0}: Error finding container 821861ec49d958e9af2b872c0b60a4cae3c68833574ef71aacb20069c672df68: Status 404 returned error can't find the container with id 821861ec49d958e9af2b872c0b60a4cae3c68833574ef71aacb20069c672df68
Oct 30 12:58:46.541635 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:46.541601    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4" event={"ID":"fa3df092-b18c-47fb-8793-90f9869a1078","Type":"ContainerStarted","Data":"04ff21d5441999697d91f7826ef964e1d1c457a79ac66a37b767125b11cdd8fb"}
Oct 30 12:58:46.542150 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:46.541660    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4" event={"ID":"fa3df092-b18c-47fb-8793-90f9869a1078","Type":"ContainerStarted","Data":"821861ec49d958e9af2b872c0b60a4cae3c68833574ef71aacb20069c672df68"}
Oct 30 12:58:46.542150 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:46.541714    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4"
Oct 30 12:58:46.583492 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:46.583442    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4" podStartSLOduration=1.583425412 podStartE2EDuration="1.583425412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:46.582059362 +0000 UTC m=+821.629193315" watchObservedRunningTime="2026-10-30 12:58:46.583425412 +0000 UTC m=+821.630559364"
Oct 30 12:58:46.847768 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:46.847733    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:46.992702 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:46.992669    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:47.134381 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:47.134296    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:47.229724 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:47.229685    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5"]
Oct 30 12:58:47.233772 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:47.233751    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5"
Oct 30 12:58:47.258404 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:47.258370    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-c2ce75\""
Oct 30 12:58:47.258724 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:47.258700    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-c2ce75-dockercfg-pvfvd\""
Oct 30 12:58:47.258963 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:47.258939    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5"]
Oct 30 12:58:47.287243 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:47.283920    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw"]
Oct 30 12:58:47.344257 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:47.343633    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw"
Oct 30 12:58:47.348873 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:47.348837    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw"]
Oct 30 12:58:47.435764 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:47.435671    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d82c543-383d-4498-8d3c-d6c2a0716736-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5\" (UID: \"8d82c543-383d-4498-8d3c-d6c2a0716736\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5"
Oct 30 12:58:47.435764 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:47.435711    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm26z\" (UniqueName: \"kubernetes.io/projected/8d82c543-383d-4498-8d3c-d6c2a0716736-kube-api-access-lm26z\") pod \"payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5\" (UID: \"8d82c543-383d-4498-8d3c-d6c2a0716736\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5"
Oct 30 12:58:47.537209 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:47.537175    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68hfg\" (UniqueName: \"kubernetes.io/projected/db2e31d3-d423-4671-8449-56d1f380fbd1-kube-api-access-68hfg\") pod \"payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw\" (UID: \"db2e31d3-d423-4671-8449-56d1f380fbd1\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw"
Oct 30 12:58:47.537412 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:47.537257    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d82c543-383d-4498-8d3c-d6c2a0716736-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5\" (UID: \"8d82c543-383d-4498-8d3c-d6c2a0716736\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5"
Oct 30 12:58:47.537412 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:47.537314    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm26z\" (UniqueName: \"kubernetes.io/projected/8d82c543-383d-4498-8d3c-d6c2a0716736-kube-api-access-lm26z\") pod \"payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5\" (UID: \"8d82c543-383d-4498-8d3c-d6c2a0716736\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5"
Oct 30 12:58:47.537412 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:47.537371    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db2e31d3-d423-4671-8449-56d1f380fbd1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw\" (UID: \"db2e31d3-d423-4671-8449-56d1f380fbd1\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw"
Oct 30 12:58:47.537981 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:47.537962    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d82c543-383d-4498-8d3c-d6c2a0716736-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5\" (UID: \"8d82c543-383d-4498-8d3c-d6c2a0716736\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5"
Oct 30 12:58:47.562668 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:47.562636    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm26z\" (UniqueName: \"kubernetes.io/projected/8d82c543-383d-4498-8d3c-d6c2a0716736-kube-api-access-lm26z\") pod \"payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5\" (UID: \"8d82c543-383d-4498-8d3c-d6c2a0716736\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5"
Oct 30 12:58:47.638167 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:47.638131    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68hfg\" (UniqueName: \"kubernetes.io/projected/db2e31d3-d423-4671-8449-56d1f380fbd1-kube-api-access-68hfg\") pod \"payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw\" (UID: \"db2e31d3-d423-4671-8449-56d1f380fbd1\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw"
Oct 30 12:58:47.638344 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:47.638207    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db2e31d3-d423-4671-8449-56d1f380fbd1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw\" (UID: \"db2e31d3-d423-4671-8449-56d1f380fbd1\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw"
Oct 30 12:58:47.639010 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:47.638988    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db2e31d3-d423-4671-8449-56d1f380fbd1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw\" (UID: \"db2e31d3-d423-4671-8449-56d1f380fbd1\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw"
Oct 30 12:58:47.681270 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:47.681215    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68hfg\" (UniqueName: \"kubernetes.io/projected/db2e31d3-d423-4671-8449-56d1f380fbd1-kube-api-access-68hfg\") pod \"payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw\" (UID: \"db2e31d3-d423-4671-8449-56d1f380fbd1\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw"
Oct 30 12:58:47.846474 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:47.846431    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5"
Oct 30 12:58:47.972025 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:47.971988    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw"
Oct 30 12:58:48.179073 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.178979    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5"]
Oct 30 12:58:48.182920 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:58:48.182889    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d82c543_383d_4498_8d3c_d6c2a0716736.slice/crio-89b304f72a336d720b46b05e6ac23a7f37705b862dda335d0adaf7169abb395a WatchSource:0}: Error finding container 89b304f72a336d720b46b05e6ac23a7f37705b862dda335d0adaf7169abb395a: Status 404 returned error can't find the container with id 89b304f72a336d720b46b05e6ac23a7f37705b862dda335d0adaf7169abb395a
Oct 30 12:58:48.203064 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.203031    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:48.280613 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.280580    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw"]
Oct 30 12:58:48.284918 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:58:48.284889    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb2e31d3_d423_4671_8449_56d1f380fbd1.slice/crio-bc028ccb28e9553ea0c95dcab0cab3882a8a68c4d59c923d568cd7a8c304025b WatchSource:0}: Error finding container bc028ccb28e9553ea0c95dcab0cab3882a8a68c4d59c923d568cd7a8c304025b: Status 404 returned error can't find the container with id bc028ccb28e9553ea0c95dcab0cab3882a8a68c4d59c923d568cd7a8c304025b
Oct 30 12:58:48.310524 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.310495    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p"]
Oct 30 12:58:48.315753 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.315732    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p"
Oct 30 12:58:48.334356 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.334327    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-c90b9e\""
Oct 30 12:58:48.334521 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.334419    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-c90b9e-dockercfg-pdmbf\""
Oct 30 12:58:48.337333 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.337303    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p"]
Oct 30 12:58:48.341911 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.341815    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:48.446950 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.446858    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88adbb69-5dc6-4d77-96f4-5b6f61d80e8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p\" (UID: \"88adbb69-5dc6-4d77-96f4-5b6f61d80e8b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p"
Oct 30 12:58:48.447101 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.447041    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8xn4\" (UniqueName: \"kubernetes.io/projected/88adbb69-5dc6-4d77-96f4-5b6f61d80e8b-kube-api-access-l8xn4\") pod \"payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p\" (UID: \"88adbb69-5dc6-4d77-96f4-5b6f61d80e8b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p"
Oct 30 12:58:48.448926 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.448897    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9"]
Oct 30 12:58:48.452618 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.452589    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9"
Oct 30 12:58:48.473628 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.473598    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9"]
Oct 30 12:58:48.563104 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.558186    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8xn4\" (UniqueName: \"kubernetes.io/projected/88adbb69-5dc6-4d77-96f4-5b6f61d80e8b-kube-api-access-l8xn4\") pod \"payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p\" (UID: \"88adbb69-5dc6-4d77-96f4-5b6f61d80e8b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p"
Oct 30 12:58:48.563789 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.563761    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88adbb69-5dc6-4d77-96f4-5b6f61d80e8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p\" (UID: \"88adbb69-5dc6-4d77-96f4-5b6f61d80e8b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p"
Oct 30 12:58:48.563974 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.563959    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbnkl\" (UniqueName: \"kubernetes.io/projected/9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69-kube-api-access-kbnkl\") pod \"payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9\" (UID: \"9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9"
Oct 30 12:58:48.564186 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.564170    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9\" (UID: \"9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9"
Oct 30 12:58:48.565270 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.565244    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88adbb69-5dc6-4d77-96f4-5b6f61d80e8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p\" (UID: \"88adbb69-5dc6-4d77-96f4-5b6f61d80e8b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p"
Oct 30 12:58:48.595771 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.595732    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw" event={"ID":"db2e31d3-d423-4671-8449-56d1f380fbd1","Type":"ContainerStarted","Data":"22d31b784325aa478dba1c843a61e3dc2db7964039fc1a4a989dc90f8acd5862"}
Oct 30 12:58:48.595937 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.595796    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw" event={"ID":"db2e31d3-d423-4671-8449-56d1f380fbd1","Type":"ContainerStarted","Data":"bc028ccb28e9553ea0c95dcab0cab3882a8a68c4d59c923d568cd7a8c304025b"}
Oct 30 12:58:48.595937 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.595848    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw"
Oct 30 12:58:48.597768 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.597741    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8xn4\" (UniqueName: \"kubernetes.io/projected/88adbb69-5dc6-4d77-96f4-5b6f61d80e8b-kube-api-access-l8xn4\") pod \"payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p\" (UID: \"88adbb69-5dc6-4d77-96f4-5b6f61d80e8b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p"
Oct 30 12:58:48.598193 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.598170    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5" event={"ID":"8d82c543-383d-4498-8d3c-d6c2a0716736","Type":"ContainerStarted","Data":"564b7c69980224c672c885ff533d40a4f70c9a6e43a071491ae939a39f83cfd6"}
Oct 30 12:58:48.598305 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.598232    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5" event={"ID":"8d82c543-383d-4498-8d3c-d6c2a0716736","Type":"ContainerStarted","Data":"89b304f72a336d720b46b05e6ac23a7f37705b862dda335d0adaf7169abb395a"}
Oct 30 12:58:48.598525 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.598505    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5"
Oct 30 12:58:48.634646 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.634615    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p"
Oct 30 12:58:48.636925 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.636883    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw" podStartSLOduration=1.636808748 podStartE2EDuration="1.636808748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:48.636297784 +0000 UTC m=+823.683431739" watchObservedRunningTime="2026-10-30 12:58:48.636808748 +0000 UTC m=+823.683942761"
Oct 30 12:58:48.665673 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.665642    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbnkl\" (UniqueName: \"kubernetes.io/projected/9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69-kube-api-access-kbnkl\") pod \"payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9\" (UID: \"9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9"
Oct 30 12:58:48.665875 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.665701    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9\" (UID: \"9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9"
Oct 30 12:58:48.666466 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.666443    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9\" (UID: \"9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9"
Oct 30 12:58:48.679024 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.678968    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5" podStartSLOduration=1.678954048 podStartE2EDuration="1.678954048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:48.676992348 +0000 UTC m=+823.724126301" watchObservedRunningTime="2026-10-30 12:58:48.678954048 +0000 UTC m=+823.726088001"
Oct 30 12:58:48.692341 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.692295    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbnkl\" (UniqueName: \"kubernetes.io/projected/9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69-kube-api-access-kbnkl\") pod \"payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9\" (UID: \"9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9"
Oct 30 12:58:48.767135 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.767027    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9"
Oct 30 12:58:48.940827 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:48.940787    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p"]
Oct 30 12:58:48.951216 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:58:48.951187    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88adbb69_5dc6_4d77_96f4_5b6f61d80e8b.slice/crio-cb22d3ec01f839afec503fb3f2d9f8fe47cf14cdaadb3f4e7b61cd195d065245 WatchSource:0}: Error finding container cb22d3ec01f839afec503fb3f2d9f8fe47cf14cdaadb3f4e7b61cd195d065245: Status 404 returned error can't find the container with id cb22d3ec01f839afec503fb3f2d9f8fe47cf14cdaadb3f4e7b61cd195d065245
Oct 30 12:58:49.099680 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:49.099615    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9"]
Oct 30 12:58:49.106271 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:58:49.106209    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fa9c95e_f9d7_44b5_b6f0_ccd15df41f69.slice/crio-8196ea15f2043214ef2b0d305a97dfe73a98924f76578073c77858e5b8c8b9ae WatchSource:0}: Error finding container 8196ea15f2043214ef2b0d305a97dfe73a98924f76578073c77858e5b8c8b9ae: Status 404 returned error can't find the container with id 8196ea15f2043214ef2b0d305a97dfe73a98924f76578073c77858e5b8c8b9ae
Oct 30 12:58:49.163273 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:49.163242    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:49.282772 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:49.282734    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:49.486298 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:49.486183    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg"]
Oct 30 12:58:49.536866 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:49.536834    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg"]
Oct 30 12:58:49.537261 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:49.537230    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg"
Oct 30 12:58:49.555550 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:49.555516    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-dd90cb\""
Oct 30 12:58:49.566921 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:49.566897    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-dd90cb-dockercfg-m6xs4\""
Oct 30 12:58:49.583651 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:49.583622    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c07693a-fc68-4472-a5d0-d7d7ba02bdc2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg\" (UID: \"3c07693a-fc68-4472-a5d0-d7d7ba02bdc2\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg"
Oct 30 12:58:49.583806 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:49.583791    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl4z9\" (UniqueName: \"kubernetes.io/projected/3c07693a-fc68-4472-a5d0-d7d7ba02bdc2-kube-api-access-cl4z9\") pod \"payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg\" (UID: \"3c07693a-fc68-4472-a5d0-d7d7ba02bdc2\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg"
Oct 30 12:58:49.604591 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:49.604550    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p" event={"ID":"88adbb69-5dc6-4d77-96f4-5b6f61d80e8b","Type":"ContainerStarted","Data":"f28609be50de347a74439be7a1d643291e94f00937792c40b6551fb64848aaa3"}
Oct 30 12:58:49.604757 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:49.604627    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p"
Oct 30 12:58:49.604757 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:49.604656    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p" event={"ID":"88adbb69-5dc6-4d77-96f4-5b6f61d80e8b","Type":"ContainerStarted","Data":"cb22d3ec01f839afec503fb3f2d9f8fe47cf14cdaadb3f4e7b61cd195d065245"}
Oct 30 12:58:49.606062 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:49.606042    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9" event={"ID":"9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69","Type":"ContainerStarted","Data":"904ec00d53d4370c85ee979df2ca10b3d90f92e5d6ea03c6d4b5b3a0369006b7"}
Oct 30 12:58:49.606144 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:49.606080    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9" event={"ID":"9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69","Type":"ContainerStarted","Data":"8196ea15f2043214ef2b0d305a97dfe73a98924f76578073c77858e5b8c8b9ae"}
Oct 30 12:58:49.606324 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:49.606306    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9"
Oct 30 12:58:49.645707 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:49.645649    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p" podStartSLOduration=1.6456357910000001 podStartE2EDuration="1.645635791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:49.645109991 +0000 UTC m=+824.692243946" watchObservedRunningTime="2026-10-30 12:58:49.645635791 +0000 UTC m=+824.692769744"
Oct 30 12:58:49.684932 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:49.684885    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl4z9\" (UniqueName: \"kubernetes.io/projected/3c07693a-fc68-4472-a5d0-d7d7ba02bdc2-kube-api-access-cl4z9\") pod \"payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg\" (UID: \"3c07693a-fc68-4472-a5d0-d7d7ba02bdc2\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg"
Oct 30 12:58:49.685143 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:49.684965    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c07693a-fc68-4472-a5d0-d7d7ba02bdc2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg\" (UID: \"3c07693a-fc68-4472-a5d0-d7d7ba02bdc2\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg"
Oct 30 12:58:49.685874 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:49.685849    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c07693a-fc68-4472-a5d0-d7d7ba02bdc2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg\" (UID: \"3c07693a-fc68-4472-a5d0-d7d7ba02bdc2\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg"
Oct 30 12:58:49.689021 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:49.688979    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9" podStartSLOduration=1.688966823 podStartE2EDuration="1.688966823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:49.687456084 +0000 UTC m=+824.734590040" watchObservedRunningTime="2026-10-30 12:58:49.688966823 +0000 UTC m=+824.736100869"
Oct 30 12:58:49.709180 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:49.709145    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl4z9\" (UniqueName: \"kubernetes.io/projected/3c07693a-fc68-4472-a5d0-d7d7ba02bdc2-kube-api-access-cl4z9\") pod \"payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg\" (UID: \"3c07693a-fc68-4472-a5d0-d7d7ba02bdc2\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg"
Oct 30 12:58:49.856240 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:49.856196    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg"
Oct 30 12:58:50.162038 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:50.161954    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg"]
Oct 30 12:58:50.165259 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:58:50.165203    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c07693a_fc68_4472_a5d0_d7d7ba02bdc2.slice/crio-f5cd851ed43674df3715e9e6532e0e371d8679503020e0a67cd4d47cfa506454 WatchSource:0}: Error finding container f5cd851ed43674df3715e9e6532e0e371d8679503020e0a67cd4d47cfa506454: Status 404 returned error can't find the container with id f5cd851ed43674df3715e9e6532e0e371d8679503020e0a67cd4d47cfa506454
Oct 30 12:58:50.612464 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:50.612430    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg" event={"ID":"3c07693a-fc68-4472-a5d0-d7d7ba02bdc2","Type":"ContainerStarted","Data":"7fb4b4efc97765b6fe2f8085ac576ada6731188d1a4f6d6c353a1f6be7c83a68"}
Oct 30 12:58:50.612887 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:50.612483    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg" event={"ID":"3c07693a-fc68-4472-a5d0-d7d7ba02bdc2","Type":"ContainerStarted","Data":"f5cd851ed43674df3715e9e6532e0e371d8679503020e0a67cd4d47cfa506454"}
Oct 30 12:58:50.612887 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:50.612512    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg"
Oct 30 12:58:50.656365 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:50.656305    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg" podStartSLOduration=1.656289304 podStartE2EDuration="1.656289304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:50.653287172 +0000 UTC m=+825.700421130" watchObservedRunningTime="2026-10-30 12:58:50.656289304 +0000 UTC m=+825.703423258"
Oct 30 12:58:50.935924 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:50.935831    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:51.077492 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:51.077457    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:51.096126 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:51.096090    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj"]
Oct 30 12:58:51.101683 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:51.101657    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj"
Oct 30 12:58:51.120075 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:51.120042    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj"]
Oct 30 12:58:51.121913 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:51.121882    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-7a3603\""
Oct 30 12:58:51.157909 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:51.157868    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-7a3603-dockercfg-cntmd\""
Oct 30 12:58:51.217702 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:51.217599    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c724287-666b-47b4-8c6b-428a20ff1164-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj\" (UID: \"3c724287-666b-47b4-8c6b-428a20ff1164\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj"
Oct 30 12:58:51.217931 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:51.217733    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntkp2\" (UniqueName: \"kubernetes.io/projected/3c724287-666b-47b4-8c6b-428a20ff1164-kube-api-access-ntkp2\") pod \"payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj\" (UID: \"3c724287-666b-47b4-8c6b-428a20ff1164\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj"
Oct 30 12:58:51.228713 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:51.228680    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:51.319064 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:51.319025    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c724287-666b-47b4-8c6b-428a20ff1164-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj\" (UID: \"3c724287-666b-47b4-8c6b-428a20ff1164\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj"
Oct 30 12:58:51.320310 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:51.320285    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntkp2\" (UniqueName: \"kubernetes.io/projected/3c724287-666b-47b4-8c6b-428a20ff1164-kube-api-access-ntkp2\") pod \"payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj\" (UID: \"3c724287-666b-47b4-8c6b-428a20ff1164\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj"
Oct 30 12:58:51.321931 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:51.320134    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c724287-666b-47b4-8c6b-428a20ff1164-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj\" (UID: \"3c724287-666b-47b4-8c6b-428a20ff1164\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj"
Oct 30 12:58:51.357959 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:51.357928    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:51.378386 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:51.378353    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntkp2\" (UniqueName: \"kubernetes.io/projected/3c724287-666b-47b4-8c6b-428a20ff1164-kube-api-access-ntkp2\") pod \"payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj\" (UID: \"3c724287-666b-47b4-8c6b-428a20ff1164\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj"
Oct 30 12:58:51.425926 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:51.425894    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj"
Oct 30 12:58:51.733370 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:51.733267    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj"]
Oct 30 12:58:51.737483 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:58:51.737449    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c724287_666b_47b4_8c6b_428a20ff1164.slice/crio-0dfed796d08d67214b5443a0770e65b76bde330b88b9efbb93f6274065eb536d WatchSource:0}: Error finding container 0dfed796d08d67214b5443a0770e65b76bde330b88b9efbb93f6274065eb536d: Status 404 returned error can't find the container with id 0dfed796d08d67214b5443a0770e65b76bde330b88b9efbb93f6274065eb536d
Oct 30 12:58:51.976137 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:51.976101    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:52.087256 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:52.087213    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:52.142808 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:52.142771    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj"]
Oct 30 12:58:52.146755 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:52.146730    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj"
Oct 30 12:58:52.161137 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:52.161110    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-67f56e\""
Oct 30 12:58:52.161286 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:52.161198    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-67f56e-dockercfg-qb622\""
Oct 30 12:58:52.167916 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:52.167876    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj"]
Oct 30 12:58:52.246246 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:52.244993    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52424dad-1898-48d3-9529-3dc7f16cb1e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj\" (UID: \"52424dad-1898-48d3-9529-3dc7f16cb1e5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj"
Oct 30 12:58:52.246246 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:52.245089    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qgwk\" (UniqueName: \"kubernetes.io/projected/52424dad-1898-48d3-9529-3dc7f16cb1e5-kube-api-access-5qgwk\") pod \"payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj\" (UID: \"52424dad-1898-48d3-9529-3dc7f16cb1e5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj"
Oct 30 12:58:52.362927 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:52.362783    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qgwk\" (UniqueName: \"kubernetes.io/projected/52424dad-1898-48d3-9529-3dc7f16cb1e5-kube-api-access-5qgwk\") pod \"payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj\" (UID: \"52424dad-1898-48d3-9529-3dc7f16cb1e5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj"
Oct 30 12:58:52.364175 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:52.364143    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52424dad-1898-48d3-9529-3dc7f16cb1e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj\" (UID: \"52424dad-1898-48d3-9529-3dc7f16cb1e5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj"
Oct 30 12:58:52.364944 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:52.364923    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52424dad-1898-48d3-9529-3dc7f16cb1e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj\" (UID: \"52424dad-1898-48d3-9529-3dc7f16cb1e5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj"
Oct 30 12:58:52.386464 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:52.386427    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qgwk\" (UniqueName: \"kubernetes.io/projected/52424dad-1898-48d3-9529-3dc7f16cb1e5-kube-api-access-5qgwk\") pod \"payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj\" (UID: \"52424dad-1898-48d3-9529-3dc7f16cb1e5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj"
Oct 30 12:58:52.461586 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:52.461544    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj"
Oct 30 12:58:52.607440 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:52.607400    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:52.652072 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:52.651339    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj" event={"ID":"3c724287-666b-47b4-8c6b-428a20ff1164","Type":"ContainerStarted","Data":"a87305396d2376abcd260fc0f333ce7c51c54539a32566ff5badd859f36702a7"}
Oct 30 12:58:52.652072 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:52.651410    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj" event={"ID":"3c724287-666b-47b4-8c6b-428a20ff1164","Type":"ContainerStarted","Data":"0dfed796d08d67214b5443a0770e65b76bde330b88b9efbb93f6274065eb536d"}
Oct 30 12:58:52.652421 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:52.652403    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj"
Oct 30 12:58:52.694534 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:52.694478    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj" podStartSLOduration=1.6944629519999999 podStartE2EDuration="1.694462952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:52.693246451 +0000 UTC m=+827.740380401" watchObservedRunningTime="2026-10-30 12:58:52.694462952 +0000 UTC m=+827.741596915"
Oct 30 12:58:52.787938 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:52.787899    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj"]
Oct 30 12:58:52.791880 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:58:52.791852    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52424dad_1898_48d3_9529_3dc7f16cb1e5.slice/crio-841a05d578ac07f2668ad444398c731ee8fe72df4915f9973bfd0162fc11c16e WatchSource:0}: Error finding container 841a05d578ac07f2668ad444398c731ee8fe72df4915f9973bfd0162fc11c16e: Status 404 returned error can't find the container with id 841a05d578ac07f2668ad444398c731ee8fe72df4915f9973bfd0162fc11c16e
Oct 30 12:58:53.175465 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:53.175385    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:53.312483 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:53.310623    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:53.429306 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:53.428813    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:53.659306 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:53.659258    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj" event={"ID":"52424dad-1898-48d3-9529-3dc7f16cb1e5","Type":"ContainerStarted","Data":"529296cefffe234a9b3ab3d24164b1e366d0dcea32cce448f93f35ca2fb2a87a"}
Oct 30 12:58:53.659512 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:53.659325    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj" event={"ID":"52424dad-1898-48d3-9529-3dc7f16cb1e5","Type":"ContainerStarted","Data":"841a05d578ac07f2668ad444398c731ee8fe72df4915f9973bfd0162fc11c16e"}
Oct 30 12:58:53.659585 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:53.659562    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj"
Oct 30 12:58:53.708807 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:53.708691    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj" podStartSLOduration=1.70867296 podStartE2EDuration="1.70867296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:53.705536921 +0000 UTC m=+828.752670874" watchObservedRunningTime="2026-10-30 12:58:53.70867296 +0000 UTC m=+828.755806914"
Oct 30 12:58:54.429304 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:54.429142    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:54.539385 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:54.539340    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9"]
Oct 30 12:58:54.543943 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:54.543917    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9"
Oct 30 12:58:54.584562 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:54.584510    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-8b46c9\""
Oct 30 12:58:54.584562 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:54.584521    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-8b46c9-dockercfg-4mq5z\""
Oct 30 12:58:54.594137 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:54.594105    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9"]
Oct 30 12:58:54.604474 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:54.604438    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:54.646918 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:54.646882    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j"]
Oct 30 12:58:54.699541 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:54.698482    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j"
Oct 30 12:58:54.701851 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:54.701813    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqhst\" (UniqueName: \"kubernetes.io/projected/7692a16a-4d89-4f37-85ae-6c43dad93d35-kube-api-access-tqhst\") pod \"payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9\" (UID: \"7692a16a-4d89-4f37-85ae-6c43dad93d35\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9"
Oct 30 12:58:54.702140 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:54.702123    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7692a16a-4d89-4f37-85ae-6c43dad93d35-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9\" (UID: \"7692a16a-4d89-4f37-85ae-6c43dad93d35\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9"
Oct 30 12:58:54.762369 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:54.762309    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j"]
Oct 30 12:58:54.803480 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:54.803438    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j\" (UID: \"cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j"
Oct 30 12:58:54.803657 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:54.803497    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqhst\" (UniqueName: \"kubernetes.io/projected/7692a16a-4d89-4f37-85ae-6c43dad93d35-kube-api-access-tqhst\") pod \"payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9\" (UID: \"7692a16a-4d89-4f37-85ae-6c43dad93d35\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9"
Oct 30 12:58:54.803758 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:54.803643    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7692a16a-4d89-4f37-85ae-6c43dad93d35-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9\" (UID: \"7692a16a-4d89-4f37-85ae-6c43dad93d35\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9"
Oct 30 12:58:54.803817 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:54.803799    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g7g6\" (UniqueName: \"kubernetes.io/projected/cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb-kube-api-access-8g7g6\") pod \"payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j\" (UID: \"cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j"
Oct 30 12:58:54.804405 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:54.804385    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7692a16a-4d89-4f37-85ae-6c43dad93d35-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9\" (UID: \"7692a16a-4d89-4f37-85ae-6c43dad93d35\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9"
Oct 30 12:58:54.839754 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:54.839718    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqhst\" (UniqueName: \"kubernetes.io/projected/7692a16a-4d89-4f37-85ae-6c43dad93d35-kube-api-access-tqhst\") pod \"payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9\" (UID: \"7692a16a-4d89-4f37-85ae-6c43dad93d35\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9"
Oct 30 12:58:54.857581 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:54.857555    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9"
Oct 30 12:58:54.905124 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:54.905084    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g7g6\" (UniqueName: \"kubernetes.io/projected/cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb-kube-api-access-8g7g6\") pod \"payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j\" (UID: \"cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j"
Oct 30 12:58:54.905335 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:54.905176    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j\" (UID: \"cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j"
Oct 30 12:58:54.906524 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:54.906488    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j\" (UID: \"cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j"
Oct 30 12:58:54.968293 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:54.967520    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g7g6\" (UniqueName: \"kubernetes.io/projected/cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb-kube-api-access-8g7g6\") pod \"payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j\" (UID: \"cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j"
Oct 30 12:58:55.088249 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:55.087973    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j"
Oct 30 12:58:55.195990 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:55.195953    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9"]
Oct 30 12:58:55.203376 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:58:55.203343    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7692a16a_4d89_4f37_85ae_6c43dad93d35.slice/crio-391e33c57549068c7a602a1afd6e44961c87e10dbbbfb3ec9751ac063534867e WatchSource:0}: Error finding container 391e33c57549068c7a602a1afd6e44961c87e10dbbbfb3ec9751ac063534867e: Status 404 returned error can't find the container with id 391e33c57549068c7a602a1afd6e44961c87e10dbbbfb3ec9751ac063534867e
Oct 30 12:58:55.338697 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:55.338662    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:55.442434 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:58:55.442405    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc60a80e_9ec6_4cbb_9e64_5b1cf0de90bb.slice/crio-6b70c1f5251089853c3a4158694e28abc66219cf5ea7ae909fb7e527f1e6118b WatchSource:0}: Error finding container 6b70c1f5251089853c3a4158694e28abc66219cf5ea7ae909fb7e527f1e6118b: Status 404 returned error can't find the container with id 6b70c1f5251089853c3a4158694e28abc66219cf5ea7ae909fb7e527f1e6118b
Oct 30 12:58:55.442768 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:55.442525    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j"]
Oct 30 12:58:55.807618 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:55.807519    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9" event={"ID":"7692a16a-4d89-4f37-85ae-6c43dad93d35","Type":"ContainerStarted","Data":"7477f63cb419fb6a54ccbdb71cd5379f790de4aceb42ecd453a5cc3ce70f76d2"}
Oct 30 12:58:55.807954 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:55.807925    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9" event={"ID":"7692a16a-4d89-4f37-85ae-6c43dad93d35","Type":"ContainerStarted","Data":"391e33c57549068c7a602a1afd6e44961c87e10dbbbfb3ec9751ac063534867e"}
Oct 30 12:58:55.808169 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:55.808156    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9"
Oct 30 12:58:55.825697 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:55.825652    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j" event={"ID":"cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb","Type":"ContainerStarted","Data":"abc3b345cdc771fc118889cfa8168a951045cee88b541355d43b90069fc23793"}
Oct 30 12:58:55.825891 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:55.825726    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j" event={"ID":"cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb","Type":"ContainerStarted","Data":"6b70c1f5251089853c3a4158694e28abc66219cf5ea7ae909fb7e527f1e6118b"}
Oct 30 12:58:55.836249 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:55.826417    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j"
Oct 30 12:58:55.856080 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:55.856009    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9" podStartSLOduration=1.8559939239999999 podStartE2EDuration="1.855993924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:55.854025686 +0000 UTC m=+830.901159651" watchObservedRunningTime="2026-10-30 12:58:55.855993924 +0000 UTC m=+830.903127876"
Oct 30 12:58:55.897766 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:55.897712    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j" podStartSLOduration=1.8976970359999998 podStartE2EDuration="1.897697036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:55.896277151 +0000 UTC m=+830.943411097" watchObservedRunningTime="2026-10-30 12:58:55.897697036 +0000 UTC m=+830.944830989"
Oct 30 12:58:56.112243 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:56.112058    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:57.607259 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:57.607110    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4"
Oct 30 12:58:57.608345 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:57.608014    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:57.806956 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:57.806917    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:58.362302 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:58.361412    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:58:59.007200 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:58:59.007162    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 12:58:59.609350 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:59.609325    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw"
Oct 30 12:58:59.609611 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:59.609599    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5"
Oct 30 12:58:59.939313 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:58:59.935537    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:00.649247 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:00.648706    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p"
Oct 30 12:59:00.658247 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:00.655441    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9"
Oct 30 12:59:01.662571 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:01.662518    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg"
Oct 30 12:59:02.151127 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:02.150912    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:02.303790 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:02.303702    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:02.443951 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:02.443906    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:03.439108 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:03.438998    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:03.989985 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:03.989946    4544 generic.go:356] "Generic (PLEG): container finished" podID="b72c176a-dc4d-4b4e-82e9-b1e15fceaed3" containerID="4bbfd2bb3e5cde47f000197b8d0e4614382d950ec815493d55462c8b59f9206d" exitCode=0
Oct 30 12:59:03.990199 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:03.989998    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4" event={"ID":"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3","Type":"ContainerDied","Data":"4bbfd2bb3e5cde47f000197b8d0e4614382d950ec815493d55462c8b59f9206d"}
Oct 30 12:59:03.990449 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:03.990435    4544 scope.go:122] "RemoveContainer" containerID="4bbfd2bb3e5cde47f000197b8d0e4614382d950ec815493d55462c8b59f9206d"
Oct 30 12:59:03.991947 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:03.991910    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:59:03.992055 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:03.992000    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:59:03.992055 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:03.992048    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:59:04.159943 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:04.159831    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:04.327837 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:04.311911    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:04.360706 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:04.360639    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp"]
Oct 30 12:59:04.368944 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:04.367800    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp"
Oct 30 12:59:04.385290 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:04.384269    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp"]
Oct 30 12:59:04.430845 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:04.430723    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-8b46c9-serving-cert\""
Oct 30 12:59:04.431132 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:04.431046    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 12:59:04.431319 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:04.431250    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nb7kb\""
Oct 30 12:59:04.442641 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:04.442473    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db19ff5-f476-4582-8153-b2b750f45032-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp\" (UID: \"6db19ff5-f476-4582-8153-b2b750f45032\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp"
Oct 30 12:59:04.442641 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:04.442539    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6db19ff5-f476-4582-8153-b2b750f45032-metrics-certs\") pod \"maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp\" (UID: \"6db19ff5-f476-4582-8153-b2b750f45032\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp"
Oct 30 12:59:04.443169 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:04.442701    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db19ff5-f476-4582-8153-b2b750f45032-maas-api-tls\") pod \"maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp\" (UID: \"6db19ff5-f476-4582-8153-b2b750f45032\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp"
Oct 30 12:59:04.443169 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:04.442796    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmsvb\" (UniqueName: \"kubernetes.io/projected/6db19ff5-f476-4582-8153-b2b750f45032-kube-api-access-gmsvb\") pod \"maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp\" (UID: \"6db19ff5-f476-4582-8153-b2b750f45032\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp"
Oct 30 12:59:04.487462 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:04.487430    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:04.544468 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:04.544423    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db19ff5-f476-4582-8153-b2b750f45032-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp\" (UID: \"6db19ff5-f476-4582-8153-b2b750f45032\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp"
Oct 30 12:59:04.544682 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:04.544504    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6db19ff5-f476-4582-8153-b2b750f45032-metrics-certs\") pod \"maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp\" (UID: \"6db19ff5-f476-4582-8153-b2b750f45032\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp"
Oct 30 12:59:04.544682 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:04.544595    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db19ff5-f476-4582-8153-b2b750f45032-maas-api-tls\") pod \"maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp\" (UID: \"6db19ff5-f476-4582-8153-b2b750f45032\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp"
Oct 30 12:59:04.544682 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:04.544646    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmsvb\" (UniqueName: \"kubernetes.io/projected/6db19ff5-f476-4582-8153-b2b750f45032-kube-api-access-gmsvb\") pod \"maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp\" (UID: \"6db19ff5-f476-4582-8153-b2b750f45032\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp"
Oct 30 12:59:04.545349 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:04.545323    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db19ff5-f476-4582-8153-b2b750f45032-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp\" (UID: \"6db19ff5-f476-4582-8153-b2b750f45032\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp"
Oct 30 12:59:04.547174 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:04.547145    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db19ff5-f476-4582-8153-b2b750f45032-maas-api-tls\") pod \"maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp\" (UID: \"6db19ff5-f476-4582-8153-b2b750f45032\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp"
Oct 30 12:59:04.547491 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:04.547471    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6db19ff5-f476-4582-8153-b2b750f45032-metrics-certs\") pod \"maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp\" (UID: \"6db19ff5-f476-4582-8153-b2b750f45032\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp"
Oct 30 12:59:04.569098 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:04.569014    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmsvb\" (UniqueName: \"kubernetes.io/projected/6db19ff5-f476-4582-8153-b2b750f45032-kube-api-access-gmsvb\") pod \"maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp\" (UID: \"6db19ff5-f476-4582-8153-b2b750f45032\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp"
Oct 30 12:59:04.631547 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:04.631417    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:04.696701 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:04.696597    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp"
Oct 30 12:59:04.768914 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:04.768882    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj"
Oct 30 12:59:04.769879 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:04.769401    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj"
Oct 30 12:59:05.004350 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:05.003459    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4" event={"ID":"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3","Type":"ContainerStarted","Data":"4608c08ab20b056ea6221d80488d26a79c048c7c9841ca6297096f123ffd2d5a"}
Oct 30 12:59:05.062549 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:05.060505    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp"]
Oct 30 12:59:05.094978 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:05.094941    4544 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 12:59:05.095139 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:05.095004    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4" podUID="b72c176a-dc4d-4b4e-82e9-b1e15fceaed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 12:59:05.095139 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:05.095088    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:59:06.012150 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:06.012111    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp" event={"ID":"6db19ff5-f476-4582-8153-b2b750f45032","Type":"ContainerStarted","Data":"577d603f349b9d38046afe314cc1ab0cde451829dd935f24db68639abb769845"}
Oct 30 12:59:06.094657 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:06.094614    4544 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 12:59:06.094850 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:06.094679    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4" podUID="b72c176a-dc4d-4b4e-82e9-b1e15fceaed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 12:59:06.298784 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:06.298678    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:06.863358 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:06.863322    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9"
Oct 30 12:59:07.095020 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:07.094980    4544 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 12:59:07.097796 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:07.095540    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4" podUID="b72c176a-dc4d-4b4e-82e9-b1e15fceaed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 12:59:07.205823 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:07.205785    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:07.900070 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:07.900036    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j"
Oct 30 12:59:08.068328 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:08.067243    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp" event={"ID":"6db19ff5-f476-4582-8153-b2b750f45032","Type":"ContainerStarted","Data":"83f865d4709a52fcdd4269f4bef1f974e4e186574af701c25496ab8c06879851"}
Oct 30 12:59:08.068520 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:08.068321    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp"
Oct 30 12:59:08.080019 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:08.079982    4544 generic.go:356] "Generic (PLEG): container finished" podID="41b0db3d-0f0b-4791-aca0-0cd31a7c78f6" containerID="ae17880baa1f57fc9fb2a30b8f7dc4d53366af63e088ac3e0303490b44cd49e8" exitCode=0
Oct 30 12:59:08.080261 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:08.080096    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455" event={"ID":"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6","Type":"ContainerDied","Data":"ae17880baa1f57fc9fb2a30b8f7dc4d53366af63e088ac3e0303490b44cd49e8"}
Oct 30 12:59:08.080643 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:08.080606    4544 scope.go:122] "RemoveContainer" containerID="ae17880baa1f57fc9fb2a30b8f7dc4d53366af63e088ac3e0303490b44cd49e8"
Oct 30 12:59:08.083060 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:08.083020    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:59:08.083189 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:08.083127    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:59:08.083286 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:08.083179    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:59:08.088006 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:08.086950    4544 generic.go:356] "Generic (PLEG): container finished" podID="3381ce8c-5918-4339-aa75-dd37e7723ee8" containerID="76d71583d1b23504e34a005ae7664b45d7effd76ef14cfabba1f253e5c7bee59" exitCode=0
Oct 30 12:59:08.088006 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:08.087009    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz" event={"ID":"3381ce8c-5918-4339-aa75-dd37e7723ee8","Type":"ContainerDied","Data":"76d71583d1b23504e34a005ae7664b45d7effd76ef14cfabba1f253e5c7bee59"}
Oct 30 12:59:08.088006 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:08.087598    4544 scope.go:122] "RemoveContainer" containerID="76d71583d1b23504e34a005ae7664b45d7effd76ef14cfabba1f253e5c7bee59"
Oct 30 12:59:08.094777 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:08.094650    4544 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 12:59:08.094777 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:08.094710    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4" podUID="b72c176a-dc4d-4b4e-82e9-b1e15fceaed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 12:59:08.095923 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:08.095883    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:59:08.096529 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:08.096495    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:59:08.096730 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:08.096709    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:59:08.195332 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:08.195279    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp" podStartSLOduration=1.88954888 podStartE2EDuration="4.195264322s" totalImagesPullingTime="2.305715442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:08.123283282 +0000 UTC m=+843.170417237" watchObservedRunningTime="2026-10-30 12:59:08.195264322 +0000 UTC m=+843.242398275"
Oct 30 12:59:09.095195 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:09.094520    4544 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 12:59:09.095195 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:09.094599    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4" podUID="b72c176a-dc4d-4b4e-82e9-b1e15fceaed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 12:59:09.097152 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:09.097111    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455" event={"ID":"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6","Type":"ContainerStarted","Data":"77286f3d171b3dd8acf2c5fb348c27c083f88db45475d778e6fc06b2681d66b6"}
Oct 30 12:59:09.100737 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:09.100691    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz" event={"ID":"3381ce8c-5918-4339-aa75-dd37e7723ee8","Type":"ContainerStarted","Data":"1f2ab98f4f237119db700d8fde4d491f25a005ece9fc417356b7d007cb2473ba"}
Oct 30 12:59:09.129247 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:09.126242    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:59:09.129247 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:09.128081    4544 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 30 12:59:09.129247 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:09.128336    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455" podUID="41b0db3d-0f0b-4791-aca0-0cd31a7c78f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 30 12:59:09.323245 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:09.322600    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:59:09.327245 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:09.325419    4544 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 12:59:09.327245 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:09.325490    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz" podUID="3381ce8c-5918-4339-aa75-dd37e7723ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 12:59:09.882262 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:09.869692    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:10.095655 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:10.095616    4544 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 12:59:10.095925 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:10.095900    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4" podUID="b72c176a-dc4d-4b4e-82e9-b1e15fceaed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 12:59:10.127680 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:10.127644    4544 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 30 12:59:10.128314 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:10.128284    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455" podUID="41b0db3d-0f0b-4791-aca0-0cd31a7c78f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 30 12:59:10.321800 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:10.321683    4544 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 12:59:10.343146 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:10.343097    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz" podUID="3381ce8c-5918-4339-aa75-dd37e7723ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 12:59:11.107844 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:11.106174    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:59:11.107844 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:11.107431    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:59:11.110418 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:11.110392    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 12:59:11.133247 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:11.130896    4544 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 30 12:59:11.133247 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:11.130958    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455" podUID="41b0db3d-0f0b-4791-aca0-0cd31a7c78f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 30 12:59:11.330247 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:11.322671    4544 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 12:59:11.349481 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:11.349431    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz" podUID="3381ce8c-5918-4339-aa75-dd37e7723ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 12:59:11.516550 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:11.516435    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:59:11.516995 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:11.516959    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:59:12.127699 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:12.127654    4544 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 30 12:59:12.128018 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:12.127988    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455" podUID="41b0db3d-0f0b-4791-aca0-0cd31a7c78f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 30 12:59:12.324044 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:12.324003    4544 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 12:59:12.324722 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:12.324681    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz" podUID="3381ce8c-5918-4339-aa75-dd37e7723ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 12:59:12.678765 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:12.678717    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:13.126843 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:13.126792    4544 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 30 12:59:13.127171 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:13.127134    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455" podUID="41b0db3d-0f0b-4791-aca0-0cd31a7c78f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 30 12:59:13.326688 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:13.326647    4544 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 12:59:13.327343 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:13.327307    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz" podUID="3381ce8c-5918-4339-aa75-dd37e7723ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 12:59:13.747463 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:13.747398    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:13.887083 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:13.887019    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:14.137784 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:14.137654    4544 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 30 12:59:14.137784 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:14.137720    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455" podUID="41b0db3d-0f0b-4791-aca0-0cd31a7c78f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 30 12:59:14.377491 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:14.374386    4544 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 12:59:14.377491 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:14.374446    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz" podUID="3381ce8c-5918-4339-aa75-dd37e7723ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 12:59:14.592562 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:14.592522    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:14.804331 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:14.804171    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:15.160115 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:15.159264    4544 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 30 12:59:15.160115 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:15.159327    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455" podUID="41b0db3d-0f0b-4791-aca0-0cd31a7c78f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 30 12:59:15.167885 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:15.166843    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp"
Oct 30 12:59:15.326247 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:15.322678    4544 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 12:59:15.326247 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:15.322744    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz" podUID="3381ce8c-5918-4339-aa75-dd37e7723ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 12:59:15.505311 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:15.501671    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:16.130502 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:16.130473    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:59:16.132648 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:16.132616    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 12:59:16.327585 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:16.327549    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:59:16.329461 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:16.329435    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 12:59:17.678940 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:17.678855    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:18.225029 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:18.224991    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:18.881232 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:18.881197    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:19.052806 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:19.052726    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:19.204330 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:19.204296    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:19.431875 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:19.431839    4544 generic.go:356] "Generic (PLEG): container finished" podID="1e40797f-db7b-4502-b06c-bb8b419e52b9" containerID="ceeaed7d2a60a6a4a239bfbe002009ebb873cc0df96aaa62c30588e4636c020f" exitCode=0
Oct 30 12:59:19.432061 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:19.431914    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v" event={"ID":"1e40797f-db7b-4502-b06c-bb8b419e52b9","Type":"ContainerDied","Data":"ceeaed7d2a60a6a4a239bfbe002009ebb873cc0df96aaa62c30588e4636c020f"}
Oct 30 12:59:19.432466 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:19.432445    4544 scope.go:122] "RemoveContainer" containerID="ceeaed7d2a60a6a4a239bfbe002009ebb873cc0df96aaa62c30588e4636c020f"
Oct 30 12:59:19.434148 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:19.434104    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:59:19.434264 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:19.434244    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:59:19.434325 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:19.434308    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 12:59:19.709044 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:19.708943    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:19.823998 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:19.823967    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:20.444355 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:20.444317    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v" event={"ID":"1e40797f-db7b-4502-b06c-bb8b419e52b9","Type":"ContainerStarted","Data":"17c44f4538ec3162998590472db2bfc1c4d35bd29829331aeab2cfe6768df920"}
Oct 30 12:59:20.637974 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:20.637851    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:59:20.638192 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:20.638165    4544 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 12:59:20.638316 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:20.638252    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v" podUID="1e40797f-db7b-4502-b06c-bb8b419e52b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 12:59:21.544184 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:21.544143    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:21.639670 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:21.639625    4544 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 12:59:21.640010 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:21.639977    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v" podUID="1e40797f-db7b-4502-b06c-bb8b419e52b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 12:59:22.036245 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:22.034737    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:22.638750 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:22.638708    4544 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 12:59:22.639306 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:22.638794    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v" podUID="1e40797f-db7b-4502-b06c-bb8b419e52b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 12:59:23.473381 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:23.473348    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:23.638835 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:23.638794    4544 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 12:59:23.639272 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:23.638868    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v" podUID="1e40797f-db7b-4502-b06c-bb8b419e52b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 12:59:23.974840 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:23.974807    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:24.157188 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:24.157148    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:24.287868 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:24.287773    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:24.638679 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:24.638644    4544 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 12:59:24.638891 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:24.638700    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v" podUID="1e40797f-db7b-4502-b06c-bb8b419e52b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 12:59:25.638810 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:25.638774    4544 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 12:59:25.638993 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:25.638831    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v" podUID="1e40797f-db7b-4502-b06c-bb8b419e52b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 12:59:26.158854 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:26.158801    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:26.642741 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:26.642709    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:59:26.643165 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:26.642991    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:59:26.643678 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:26.643661    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 12:59:28.145653 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:28.145622    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:29.006241 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:59:29.006193    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 12:59:29.207744 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:29.207713    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:29.534926 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:29.534875    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:29.836949 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:29.836914    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:29.992183 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:29.992147    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:30.158052 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:30.157969    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:30.393229 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:30.393182    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:30.601891 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:30.601857    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:31.038484 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:31.038401    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"]
Oct 30 12:59:31.042870 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:31.042844    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 12:59:31.087011 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:31.086980    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"]
Oct 30 12:59:31.088184 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:31.088139    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-09db16\"/\"openshift-service-ca.crt\""
Oct 30 12:59:31.093998 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:31.093971    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-09db16\"/\"simulator-w3-09db16-kserve-self-signed-certs\""
Oct 30 12:59:31.094574 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:31.094040    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-09db16\"/\"kube-root-ca.crt\""
Oct 30 12:59:31.094788 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:31.094142    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-09db16\"/\"default-dockercfg-xg4df\""
Oct 30 12:59:31.169306 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:31.169269    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-home\") pod \"simulator-w3-09db16-kserve-687f7bfcc6-hl6tf\" (UID: \"91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2\") " pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 12:59:31.169482 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:31.169324    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-tmp-dir\") pod \"simulator-w3-09db16-kserve-687f7bfcc6-hl6tf\" (UID: \"91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2\") " pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 12:59:31.169482 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:31.169450    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-model-cache\") pod \"simulator-w3-09db16-kserve-687f7bfcc6-hl6tf\" (UID: \"91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2\") " pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 12:59:31.169601 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:31.169484    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-dshm\") pod \"simulator-w3-09db16-kserve-687f7bfcc6-hl6tf\" (UID: \"91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2\") " pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 12:59:31.169601 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:31.169548    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsqbf\" (UniqueName: \"kubernetes.io/projected/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-kube-api-access-hsqbf\") pod \"simulator-w3-09db16-kserve-687f7bfcc6-hl6tf\" (UID: \"91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2\") " pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 12:59:31.169745 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:31.169724    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-tls-certs\") pod \"simulator-w3-09db16-kserve-687f7bfcc6-hl6tf\" (UID: \"91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2\") " pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 12:59:31.184816 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:31.184787    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:31.271965 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:31.271933    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-tls-certs\") pod \"simulator-w3-09db16-kserve-687f7bfcc6-hl6tf\" (UID: \"91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2\") " pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 12:59:31.272174 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:31.271985    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-home\") pod \"simulator-w3-09db16-kserve-687f7bfcc6-hl6tf\" (UID: \"91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2\") " pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 12:59:31.272174 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:31.272017    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-tmp-dir\") pod \"simulator-w3-09db16-kserve-687f7bfcc6-hl6tf\" (UID: \"91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2\") " pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 12:59:31.272174 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:31.272064    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-model-cache\") pod \"simulator-w3-09db16-kserve-687f7bfcc6-hl6tf\" (UID: \"91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2\") " pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 12:59:31.272174 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:31.272095    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-dshm\") pod \"simulator-w3-09db16-kserve-687f7bfcc6-hl6tf\" (UID: \"91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2\") " pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 12:59:31.272174 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:31.272148    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsqbf\" (UniqueName: \"kubernetes.io/projected/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-kube-api-access-hsqbf\") pod \"simulator-w3-09db16-kserve-687f7bfcc6-hl6tf\" (UID: \"91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2\") " pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 12:59:31.272571 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:31.272543    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-model-cache\") pod \"simulator-w3-09db16-kserve-687f7bfcc6-hl6tf\" (UID: \"91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2\") " pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 12:59:31.272690 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:31.272543    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-tmp-dir\") pod \"simulator-w3-09db16-kserve-687f7bfcc6-hl6tf\" (UID: \"91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2\") " pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 12:59:31.272690 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:31.272660    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-home\") pod \"simulator-w3-09db16-kserve-687f7bfcc6-hl6tf\" (UID: \"91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2\") " pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 12:59:31.274500 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:31.274475    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-dshm\") pod \"simulator-w3-09db16-kserve-687f7bfcc6-hl6tf\" (UID: \"91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2\") " pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 12:59:31.274858 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:31.274838    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-tls-certs\") pod \"simulator-w3-09db16-kserve-687f7bfcc6-hl6tf\" (UID: \"91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2\") " pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 12:59:31.301063 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:31.300949    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsqbf\" (UniqueName: \"kubernetes.io/projected/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-kube-api-access-hsqbf\") pod \"simulator-w3-09db16-kserve-687f7bfcc6-hl6tf\" (UID: \"91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2\") " pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 12:59:31.357345 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:31.356452    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 12:59:31.720351 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:31.720315    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"]
Oct 30 12:59:31.724479 ip-10-0-143-38 kubenswrapper[4544]: W1030 12:59:31.724452    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91c6b5c7_0c42_4a3b_95d3_ebd7fdac21e2.slice/crio-925dfece6b0b78b9171040b7322218a7032d41ad764799e11d1dcdac855fb6de WatchSource:0}: Error finding container 925dfece6b0b78b9171040b7322218a7032d41ad764799e11d1dcdac855fb6de: Status 404 returned error can't find the container with id 925dfece6b0b78b9171040b7322218a7032d41ad764799e11d1dcdac855fb6de
Oct 30 12:59:32.224820 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:32.224324    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:32.526506 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:32.526415    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" event={"ID":"91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2","Type":"ContainerStarted","Data":"ec8d4660164a489650899c9662278b1fd4ce42117fef8bfa0bf32b7155990a26"}
Oct 30 12:59:32.526506 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:32.526457    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" event={"ID":"91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2","Type":"ContainerStarted","Data":"925dfece6b0b78b9171040b7322218a7032d41ad764799e11d1dcdac855fb6de"}
Oct 30 12:59:32.570053 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:32.569983    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podStartSLOduration=2.5699621649999997 podStartE2EDuration="2.569962165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:32.568377881 +0000 UTC m=+867.615511834" watchObservedRunningTime="2026-10-30 12:59:32.569962165 +0000 UTC m=+867.617096119"
Oct 30 12:59:33.357041 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:33.357001    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 12:59:34.009174 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:34.009139    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:35.691798 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:35.691762    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:35.958788 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:35.958701    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:37.429043 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:37.429009    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:37.705267 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:37.705150    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:39.490852 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:39.490811    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:41.268926 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:41.268896    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:41.356914 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:41.356884    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 12:59:41.369717 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:41.369692    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 12:59:41.588136 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:41.588108    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 12:59:43.178532 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:43.178494    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:44.900631 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:44.900593    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:45.360500 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:45.360468    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:46.805651 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:46.805607    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:46.929280 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:46.924866    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:48.566972 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:48.566928    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:49.931149 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:49.931117    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:50.363421 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:50.363391    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:52.329121 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:52.329086    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:52.716527 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:52.716437    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:54.079054 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:54.079022    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:55.435817 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:55.435779    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:55.906031 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:55.905978    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:57.673157 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:57.673124    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 12:59:59.004952 ip-10-0-143-38 kubenswrapper[4544]: E1030 12:59:59.004910    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 12:59:59.596358 ip-10-0-143-38 kubenswrapper[4544]: I1030 12:59:59.596322    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:00.165341 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.165310    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs"]
Oct 30 13:00:00.169355 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.169339    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs"
Oct 30 13:00:00.181420 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.181397    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr"]
Oct 30 13:00:00.185193 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.185171    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54"]
Oct 30 13:00:00.185324 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.185303    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr"
Oct 30 13:00:00.188854 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.188836    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54"
Oct 30 13:00:00.190053 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.190034    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:00:00.217308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.217283    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr"]
Oct 30 13:00:00.217548 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.217508    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs"]
Oct 30 13:00:00.235816 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.235672    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54"]
Oct 30 13:00:00.239865 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.239834    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp"]
Oct 30 13:00:00.244337 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.244312    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp"
Oct 30 13:00:00.251849 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.251824    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63284ee7-cdee-4456-a64a-72ea663b19a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs\" (UID: \"63284ee7-cdee-4456-a64a-72ea663b19a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs"
Oct 30 13:00:00.251970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.251896    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwmt5\" (UniqueName: \"kubernetes.io/projected/63284ee7-cdee-4456-a64a-72ea663b19a4-kube-api-access-rwmt5\") pod \"maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs\" (UID: \"63284ee7-cdee-4456-a64a-72ea663b19a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs"
Oct 30 13:00:00.261886 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.261862    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp"]
Oct 30 13:00:00.353163 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.353124    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c52ae920-7884-4f8b-b7cf-7e853416af2c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54\" (UID: \"c52ae920-7884-4f8b-b7cf-7e853416af2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54"
Oct 30 13:00:00.353394 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.353190    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63284ee7-cdee-4456-a64a-72ea663b19a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs\" (UID: \"63284ee7-cdee-4456-a64a-72ea663b19a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs"
Oct 30 13:00:00.353394 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.353290    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58kkd\" (UniqueName: \"kubernetes.io/projected/4e0369cc-9862-4842-b130-171298422b33-kube-api-access-58kkd\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr\" (UID: \"4e0369cc-9862-4842-b130-171298422b33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr"
Oct 30 13:00:00.353394 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.353330    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msqx6\" (UniqueName: \"kubernetes.io/projected/82aafa40-b786-4399-8e45-96982391939c-kube-api-access-msqx6\") pod \"maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp\" (UID: \"82aafa40-b786-4399-8e45-96982391939c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp"
Oct 30 13:00:00.353552 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.353497    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfzmp\" (UniqueName: \"kubernetes.io/projected/c52ae920-7884-4f8b-b7cf-7e853416af2c-kube-api-access-zfzmp\") pod \"maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54\" (UID: \"c52ae920-7884-4f8b-b7cf-7e853416af2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54"
Oct 30 13:00:00.353552 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.353542    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82aafa40-b786-4399-8e45-96982391939c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp\" (UID: \"82aafa40-b786-4399-8e45-96982391939c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp"
Oct 30 13:00:00.353659 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.353576    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwmt5\" (UniqueName: \"kubernetes.io/projected/63284ee7-cdee-4456-a64a-72ea663b19a4-kube-api-access-rwmt5\") pod \"maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs\" (UID: \"63284ee7-cdee-4456-a64a-72ea663b19a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs"
Oct 30 13:00:00.353712 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.353677    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e0369cc-9862-4842-b130-171298422b33-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr\" (UID: \"4e0369cc-9862-4842-b130-171298422b33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr"
Oct 30 13:00:00.354004 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.353984    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63284ee7-cdee-4456-a64a-72ea663b19a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs\" (UID: \"63284ee7-cdee-4456-a64a-72ea663b19a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs"
Oct 30 13:00:00.380322 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.380290    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwmt5\" (UniqueName: \"kubernetes.io/projected/63284ee7-cdee-4456-a64a-72ea663b19a4-kube-api-access-rwmt5\") pod \"maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs\" (UID: \"63284ee7-cdee-4456-a64a-72ea663b19a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs"
Oct 30 13:00:00.455272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.455098    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58kkd\" (UniqueName: \"kubernetes.io/projected/4e0369cc-9862-4842-b130-171298422b33-kube-api-access-58kkd\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr\" (UID: \"4e0369cc-9862-4842-b130-171298422b33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr"
Oct 30 13:00:00.455272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.455151    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msqx6\" (UniqueName: \"kubernetes.io/projected/82aafa40-b786-4399-8e45-96982391939c-kube-api-access-msqx6\") pod \"maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp\" (UID: \"82aafa40-b786-4399-8e45-96982391939c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp"
Oct 30 13:00:00.455272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.455265    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfzmp\" (UniqueName: \"kubernetes.io/projected/c52ae920-7884-4f8b-b7cf-7e853416af2c-kube-api-access-zfzmp\") pod \"maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54\" (UID: \"c52ae920-7884-4f8b-b7cf-7e853416af2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54"
Oct 30 13:00:00.455692 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.455295    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82aafa40-b786-4399-8e45-96982391939c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp\" (UID: \"82aafa40-b786-4399-8e45-96982391939c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp"
Oct 30 13:00:00.455692 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.455345    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e0369cc-9862-4842-b130-171298422b33-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr\" (UID: \"4e0369cc-9862-4842-b130-171298422b33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr"
Oct 30 13:00:00.455692 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.455421    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c52ae920-7884-4f8b-b7cf-7e853416af2c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54\" (UID: \"c52ae920-7884-4f8b-b7cf-7e853416af2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54"
Oct 30 13:00:00.456361 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.456328    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c52ae920-7884-4f8b-b7cf-7e853416af2c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54\" (UID: \"c52ae920-7884-4f8b-b7cf-7e853416af2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54"
Oct 30 13:00:00.456522 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.456338    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e0369cc-9862-4842-b130-171298422b33-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr\" (UID: \"4e0369cc-9862-4842-b130-171298422b33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr"
Oct 30 13:00:00.456522 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.456371    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82aafa40-b786-4399-8e45-96982391939c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp\" (UID: \"82aafa40-b786-4399-8e45-96982391939c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp"
Oct 30 13:00:00.483732 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.483696    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs"
Oct 30 13:00:00.484509 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.484489    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfzmp\" (UniqueName: \"kubernetes.io/projected/c52ae920-7884-4f8b-b7cf-7e853416af2c-kube-api-access-zfzmp\") pod \"maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54\" (UID: \"c52ae920-7884-4f8b-b7cf-7e853416af2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54"
Oct 30 13:00:00.484623 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.484531    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58kkd\" (UniqueName: \"kubernetes.io/projected/4e0369cc-9862-4842-b130-171298422b33-kube-api-access-58kkd\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr\" (UID: \"4e0369cc-9862-4842-b130-171298422b33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr"
Oct 30 13:00:00.488705 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.488671    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msqx6\" (UniqueName: \"kubernetes.io/projected/82aafa40-b786-4399-8e45-96982391939c-kube-api-access-msqx6\") pod \"maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp\" (UID: \"82aafa40-b786-4399-8e45-96982391939c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp"
Oct 30 13:00:00.499583 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.499557    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr"
Oct 30 13:00:00.505989 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.505966    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54"
Oct 30 13:00:00.563262 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.562696    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp"
Oct 30 13:00:00.852643 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.852613    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs"]
Oct 30 13:00:00.853897 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:00:00.853424    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e0369cc_9862_4842_b130_171298422b33.slice/crio-aeec8e675178ca69ff1feabdd56484bfd4391c4f982051e56b4b100a3dbc44ea WatchSource:0}: Error finding container aeec8e675178ca69ff1feabdd56484bfd4391c4f982051e56b4b100a3dbc44ea: Status 404 returned error can't find the container with id aeec8e675178ca69ff1feabdd56484bfd4391c4f982051e56b4b100a3dbc44ea
Oct 30 13:00:00.855290 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.855263    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr"]
Oct 30 13:00:00.869035 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.869010    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54"]
Oct 30 13:00:00.872728 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:00:00.872698    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc52ae920_7884_4f8b_b7cf_7e853416af2c.slice/crio-c1e808f95e2855036a8828bbaecaf1f33b5a7238bf4db80c120721f31b6f8c35 WatchSource:0}: Error finding container c1e808f95e2855036a8828bbaecaf1f33b5a7238bf4db80c120721f31b6f8c35: Status 404 returned error can't find the container with id c1e808f95e2855036a8828bbaecaf1f33b5a7238bf4db80c120721f31b6f8c35
Oct 30 13:00:00.909331 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:00.909300    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp"]
Oct 30 13:00:00.914578 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:00:00.914553    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82aafa40_b786_4399_8e45_96982391939c.slice/crio-b4bb4ca54c82c1fbb931b0c04bc4de0030b52ab94b9ea60be8bf0d02e2e7cdfc WatchSource:0}: Error finding container b4bb4ca54c82c1fbb931b0c04bc4de0030b52ab94b9ea60be8bf0d02e2e7cdfc: Status 404 returned error can't find the container with id b4bb4ca54c82c1fbb931b0c04bc4de0030b52ab94b9ea60be8bf0d02e2e7cdfc
Oct 30 13:00:01.347537 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:01.347504    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:01.488047 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:01.487932    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:01.718252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:01.718155    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs" event={"ID":"63284ee7-cdee-4456-a64a-72ea663b19a4","Type":"ContainerStarted","Data":"c1e83aca0371d31820f3e384bde03e1b65bc3ffced58326a5768e0716aec8c5f"}
Oct 30 13:00:01.719315 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:01.719279    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp" event={"ID":"82aafa40-b786-4399-8e45-96982391939c","Type":"ContainerStarted","Data":"b4bb4ca54c82c1fbb931b0c04bc4de0030b52ab94b9ea60be8bf0d02e2e7cdfc"}
Oct 30 13:00:01.720438 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:01.720418    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr" event={"ID":"4e0369cc-9862-4842-b130-171298422b33","Type":"ContainerStarted","Data":"aeec8e675178ca69ff1feabdd56484bfd4391c4f982051e56b4b100a3dbc44ea"}
Oct 30 13:00:01.721554 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:01.721532    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54" event={"ID":"c52ae920-7884-4f8b-b7cf-7e853416af2c","Type":"ContainerStarted","Data":"c1e808f95e2855036a8828bbaecaf1f33b5a7238bf4db80c120721f31b6f8c35"}
Oct 30 13:00:03.283346 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:03.283277    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:04.745120 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:04.745084    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr" event={"ID":"4e0369cc-9862-4842-b130-171298422b33","Type":"ContainerStarted","Data":"fd3ccfba258703d6c437a3cedcf60bdd80255e9e6933905a3cdb407abf102863"}
Oct 30 13:00:04.788215 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:04.788166    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr" podStartSLOduration=1.597857608 podStartE2EDuration="4.788153745s" totalImagesPullingTime="3.190296137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:04.787532264 +0000 UTC m=+899.834666218" watchObservedRunningTime="2026-10-30 13:00:04.788153745 +0000 UTC m=+899.835287742"
Oct 30 13:00:05.294912 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:05.294808    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:05.755581 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:05.755530    4544 generic.go:356] "Generic (PLEG): container finished" podID="82aafa40-b786-4399-8e45-96982391939c" containerID="3bcfbf64e340fffbb547cbd58704aa8662f26b57e312e7bb450b196ba9f68f98" exitCode=60
Oct 30 13:00:05.756072 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:05.755622    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp" event={"ID":"82aafa40-b786-4399-8e45-96982391939c","Type":"ContainerDied","Data":"3bcfbf64e340fffbb547cbd58704aa8662f26b57e312e7bb450b196ba9f68f98"}
Oct 30 13:00:05.756072 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:05.755872    4544 scope.go:122] "RemoveContainer" containerID="3bcfbf64e340fffbb547cbd58704aa8662f26b57e312e7bb450b196ba9f68f98"
Oct 30 13:00:05.757662 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:05.757644    4544 generic.go:356] "Generic (PLEG): container finished" podID="4e0369cc-9862-4842-b130-171298422b33" containerID="fd3ccfba258703d6c437a3cedcf60bdd80255e9e6933905a3cdb407abf102863" exitCode=60
Oct 30 13:00:05.757740 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:05.757717    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr" event={"ID":"4e0369cc-9862-4842-b130-171298422b33","Type":"ContainerDied","Data":"fd3ccfba258703d6c437a3cedcf60bdd80255e9e6933905a3cdb407abf102863"}
Oct 30 13:00:05.757968 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:05.757949    4544 scope.go:122] "RemoveContainer" containerID="fd3ccfba258703d6c437a3cedcf60bdd80255e9e6933905a3cdb407abf102863"
Oct 30 13:00:05.760647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:05.760623    4544 generic.go:356] "Generic (PLEG): container finished" podID="c52ae920-7884-4f8b-b7cf-7e853416af2c" containerID="087142dcd47683c1c396cfb9816d86f9e8698611b9622f0c98969257bbd7644a" exitCode=60
Oct 30 13:00:05.760771 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:05.760737    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54" event={"ID":"c52ae920-7884-4f8b-b7cf-7e853416af2c","Type":"ContainerDied","Data":"087142dcd47683c1c396cfb9816d86f9e8698611b9622f0c98969257bbd7644a"}
Oct 30 13:00:05.761350 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:05.760944    4544 scope.go:122] "RemoveContainer" containerID="087142dcd47683c1c396cfb9816d86f9e8698611b9622f0c98969257bbd7644a"
Oct 30 13:00:05.762689 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:05.762667    4544 generic.go:356] "Generic (PLEG): container finished" podID="63284ee7-cdee-4456-a64a-72ea663b19a4" containerID="6bfad02ff093a41fd9697a10ae8e7d68cc150f336176bcb335d3d6195d718a95" exitCode=60
Oct 30 13:00:05.762803 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:05.762704    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs" event={"ID":"63284ee7-cdee-4456-a64a-72ea663b19a4","Type":"ContainerDied","Data":"6bfad02ff093a41fd9697a10ae8e7d68cc150f336176bcb335d3d6195d718a95"}
Oct 30 13:00:05.762934 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:05.762911    4544 scope.go:122] "RemoveContainer" containerID="6bfad02ff093a41fd9697a10ae8e7d68cc150f336176bcb335d3d6195d718a95"
Oct 30 13:00:06.532072 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:06.532039    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:06.770394 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:06.770337    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54" event={"ID":"c52ae920-7884-4f8b-b7cf-7e853416af2c","Type":"ContainerStarted","Data":"5439f38c197bcaf9f7a8c86e1bf0cb431136db678553df9c1df9dacf70f08446"}
Oct 30 13:00:06.772506 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:06.772473    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs" event={"ID":"63284ee7-cdee-4456-a64a-72ea663b19a4","Type":"ContainerStarted","Data":"004a7040be7751027759a1f906dccaaee6b610af822a18f322a46539b2df0b3f"}
Oct 30 13:00:06.774410 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:06.774385    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp" event={"ID":"82aafa40-b786-4399-8e45-96982391939c","Type":"ContainerStarted","Data":"f37847c7faf8aa987baf4ace84ae7de545841cf0eace68557d1bbc3dff48b8d7"}
Oct 30 13:00:06.776352 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:06.776328    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr" event={"ID":"4e0369cc-9862-4842-b130-171298422b33","Type":"ContainerStarted","Data":"d06049f9d583a71b2edf57e21f7ec3924e6bce0fec750ac587f8ff940ec597cb"}
Oct 30 13:00:06.823009 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:06.822956    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54" podStartSLOduration=3.100566076 podStartE2EDuration="6.822940307s" totalImagesPullingTime="3.722374231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:06.819873528 +0000 UTC m=+901.867007505" watchObservedRunningTime="2026-10-30 13:00:06.822940307 +0000 UTC m=+901.870074314"
Oct 30 13:00:06.863473 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:06.863389    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp" podStartSLOduration=3.147080977 podStartE2EDuration="6.86337348s" totalImagesPullingTime="3.716292503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:06.861733417 +0000 UTC m=+901.908867369" watchObservedRunningTime="2026-10-30 13:00:06.86337348 +0000 UTC m=+901.910507433"
Oct 30 13:00:06.903802 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:06.903749    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs" podStartSLOduration=3.169088409 podStartE2EDuration="6.903734477s" totalImagesPullingTime="3.734646068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:06.903550249 +0000 UTC m=+901.950684201" watchObservedRunningTime="2026-10-30 13:00:06.903734477 +0000 UTC m=+901.950868429"
Oct 30 13:00:07.379020 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:07.378984    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:07.784158 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:07.784055    4544 generic.go:356] "Generic (PLEG): container finished" podID="63284ee7-cdee-4456-a64a-72ea663b19a4" containerID="004a7040be7751027759a1f906dccaaee6b610af822a18f322a46539b2df0b3f" exitCode=60
Oct 30 13:00:07.784158 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:07.784142    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs" event={"ID":"63284ee7-cdee-4456-a64a-72ea663b19a4","Type":"ContainerDied","Data":"004a7040be7751027759a1f906dccaaee6b610af822a18f322a46539b2df0b3f"}
Oct 30 13:00:07.784721 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:07.784196    4544 scope.go:122] "RemoveContainer" containerID="6bfad02ff093a41fd9697a10ae8e7d68cc150f336176bcb335d3d6195d718a95"
Oct 30 13:00:07.784721 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:07.784506    4544 scope.go:122] "RemoveContainer" containerID="004a7040be7751027759a1f906dccaaee6b610af822a18f322a46539b2df0b3f"
Oct 30 13:00:07.784809 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:00:07.784779    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs_odh-ai-gateway-infra(63284ee7-cdee-4456-a64a-72ea663b19a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs" podUID="63284ee7-cdee-4456-a64a-72ea663b19a4"
Oct 30 13:00:07.786042 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:07.786020    4544 generic.go:356] "Generic (PLEG): container finished" podID="82aafa40-b786-4399-8e45-96982391939c" containerID="f37847c7faf8aa987baf4ace84ae7de545841cf0eace68557d1bbc3dff48b8d7" exitCode=60
Oct 30 13:00:07.786147 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:07.786104    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp" event={"ID":"82aafa40-b786-4399-8e45-96982391939c","Type":"ContainerDied","Data":"f37847c7faf8aa987baf4ace84ae7de545841cf0eace68557d1bbc3dff48b8d7"}
Oct 30 13:00:07.786381 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:07.786364    4544 scope.go:122] "RemoveContainer" containerID="f37847c7faf8aa987baf4ace84ae7de545841cf0eace68557d1bbc3dff48b8d7"
Oct 30 13:00:07.786654 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:00:07.786626    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp_odh-ai-gateway-infra(82aafa40-b786-4399-8e45-96982391939c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp" podUID="82aafa40-b786-4399-8e45-96982391939c"
Oct 30 13:00:07.788019 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:07.788000    4544 generic.go:356] "Generic (PLEG): container finished" podID="4e0369cc-9862-4842-b130-171298422b33" containerID="d06049f9d583a71b2edf57e21f7ec3924e6bce0fec750ac587f8ff940ec597cb" exitCode=60
Oct 30 13:00:07.788113 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:07.788080    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr" event={"ID":"4e0369cc-9862-4842-b130-171298422b33","Type":"ContainerDied","Data":"d06049f9d583a71b2edf57e21f7ec3924e6bce0fec750ac587f8ff940ec597cb"}
Oct 30 13:00:07.788269 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:07.788254    4544 scope.go:122] "RemoveContainer" containerID="d06049f9d583a71b2edf57e21f7ec3924e6bce0fec750ac587f8ff940ec597cb"
Oct 30 13:00:07.788503 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:00:07.788479    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr_odh-ai-gateway-infra(4e0369cc-9862-4842-b130-171298422b33)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr" podUID="4e0369cc-9862-4842-b130-171298422b33"
Oct 30 13:00:07.790097 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:07.790078    4544 generic.go:356] "Generic (PLEG): container finished" podID="c52ae920-7884-4f8b-b7cf-7e853416af2c" containerID="5439f38c197bcaf9f7a8c86e1bf0cb431136db678553df9c1df9dacf70f08446" exitCode=60
Oct 30 13:00:07.790180 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:07.790128    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54" event={"ID":"c52ae920-7884-4f8b-b7cf-7e853416af2c","Type":"ContainerDied","Data":"5439f38c197bcaf9f7a8c86e1bf0cb431136db678553df9c1df9dacf70f08446"}
Oct 30 13:00:07.790442 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:07.790424    4544 scope.go:122] "RemoveContainer" containerID="5439f38c197bcaf9f7a8c86e1bf0cb431136db678553df9c1df9dacf70f08446"
Oct 30 13:00:07.790698 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:00:07.790677    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54_odh-ai-gateway-infra(c52ae920-7884-4f8b-b7cf-7e853416af2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54" podUID="c52ae920-7884-4f8b-b7cf-7e853416af2c"
Oct 30 13:00:07.796867 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:07.796839    4544 scope.go:122] "RemoveContainer" containerID="3bcfbf64e340fffbb547cbd58704aa8662f26b57e312e7bb450b196ba9f68f98"
Oct 30 13:00:07.810361 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:07.810338    4544 scope.go:122] "RemoveContainer" containerID="fd3ccfba258703d6c437a3cedcf60bdd80255e9e6933905a3cdb407abf102863"
Oct 30 13:00:07.823866 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:07.823828    4544 scope.go:122] "RemoveContainer" containerID="087142dcd47683c1c396cfb9816d86f9e8698611b9622f0c98969257bbd7644a"
Oct 30 13:00:08.797806 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:08.797777    4544 scope.go:122] "RemoveContainer" containerID="5439f38c197bcaf9f7a8c86e1bf0cb431136db678553df9c1df9dacf70f08446"
Oct 30 13:00:08.798263 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:00:08.798126    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54_odh-ai-gateway-infra(c52ae920-7884-4f8b-b7cf-7e853416af2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54" podUID="c52ae920-7884-4f8b-b7cf-7e853416af2c"
Oct 30 13:00:08.799525 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:08.799503    4544 scope.go:122] "RemoveContainer" containerID="004a7040be7751027759a1f906dccaaee6b610af822a18f322a46539b2df0b3f"
Oct 30 13:00:08.799757 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:00:08.799736    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs_odh-ai-gateway-infra(63284ee7-cdee-4456-a64a-72ea663b19a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs" podUID="63284ee7-cdee-4456-a64a-72ea663b19a4"
Oct 30 13:00:08.801057 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:08.801027    4544 scope.go:122] "RemoveContainer" containerID="f37847c7faf8aa987baf4ace84ae7de545841cf0eace68557d1bbc3dff48b8d7"
Oct 30 13:00:08.801287 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:00:08.801268    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp_odh-ai-gateway-infra(82aafa40-b786-4399-8e45-96982391939c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp" podUID="82aafa40-b786-4399-8e45-96982391939c"
Oct 30 13:00:08.802434 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:08.802418    4544 scope.go:122] "RemoveContainer" containerID="d06049f9d583a71b2edf57e21f7ec3924e6bce0fec750ac587f8ff940ec597cb"
Oct 30 13:00:08.802603 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:00:08.802586    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr_odh-ai-gateway-infra(4e0369cc-9862-4842-b130-171298422b33)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr" podUID="4e0369cc-9862-4842-b130-171298422b33"
Oct 30 13:00:09.285159 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:09.285123    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:11.119700 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:11.119660    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:12.775924 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:12.775890    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:14.831188 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:14.828858    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:15.678491 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:15.678445    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"]
Oct 30 13:00:15.684109 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:15.684078    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:00:15.707171 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:15.707130    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"]
Oct 30 13:00:15.714241 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:15.714197    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-c2ce75\"/\"openshift-service-ca.crt\""
Oct 30 13:00:15.714767 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:15.714250    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c2ce75\"/\"default-dockercfg-mkmbm\""
Oct 30 13:00:15.714767 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:15.714425    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-c2ce75\"/\"kube-root-ca.crt\""
Oct 30 13:00:15.714767 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:15.714531    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c2ce75\"/\"premium-w2-c2ce75-kserve-self-signed-certs\""
Oct 30 13:00:15.748339 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:15.748300    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:15.773698 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:15.773663    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/404ea200-b989-4029-be47-f7075c3ed65e-tls-certs\") pod \"premium-w2-c2ce75-kserve-c454bd5f9-vgxtv\" (UID: \"404ea200-b989-4029-be47-f7075c3ed65e\") " pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:00:15.773902 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:15.773711    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/404ea200-b989-4029-be47-f7075c3ed65e-tmp-dir\") pod \"premium-w2-c2ce75-kserve-c454bd5f9-vgxtv\" (UID: \"404ea200-b989-4029-be47-f7075c3ed65e\") " pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:00:15.773902 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:15.773736    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/404ea200-b989-4029-be47-f7075c3ed65e-home\") pod \"premium-w2-c2ce75-kserve-c454bd5f9-vgxtv\" (UID: \"404ea200-b989-4029-be47-f7075c3ed65e\") " pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:00:15.773995 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:15.773962    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhpjc\" (UniqueName: \"kubernetes.io/projected/404ea200-b989-4029-be47-f7075c3ed65e-kube-api-access-mhpjc\") pod \"premium-w2-c2ce75-kserve-c454bd5f9-vgxtv\" (UID: \"404ea200-b989-4029-be47-f7075c3ed65e\") " pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:00:15.774048 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:15.774009    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/404ea200-b989-4029-be47-f7075c3ed65e-dshm\") pod \"premium-w2-c2ce75-kserve-c454bd5f9-vgxtv\" (UID: \"404ea200-b989-4029-be47-f7075c3ed65e\") " pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:00:15.774194 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:15.774155    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/404ea200-b989-4029-be47-f7075c3ed65e-model-cache\") pod \"premium-w2-c2ce75-kserve-c454bd5f9-vgxtv\" (UID: \"404ea200-b989-4029-be47-f7075c3ed65e\") " pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:00:15.882385 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:15.882343    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhpjc\" (UniqueName: \"kubernetes.io/projected/404ea200-b989-4029-be47-f7075c3ed65e-kube-api-access-mhpjc\") pod \"premium-w2-c2ce75-kserve-c454bd5f9-vgxtv\" (UID: \"404ea200-b989-4029-be47-f7075c3ed65e\") " pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:00:15.882898 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:15.882711    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/404ea200-b989-4029-be47-f7075c3ed65e-dshm\") pod \"premium-w2-c2ce75-kserve-c454bd5f9-vgxtv\" (UID: \"404ea200-b989-4029-be47-f7075c3ed65e\") " pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:00:15.882898 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:15.882814    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/404ea200-b989-4029-be47-f7075c3ed65e-model-cache\") pod \"premium-w2-c2ce75-kserve-c454bd5f9-vgxtv\" (UID: \"404ea200-b989-4029-be47-f7075c3ed65e\") " pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:00:15.883018 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:15.882947    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/404ea200-b989-4029-be47-f7075c3ed65e-tls-certs\") pod \"premium-w2-c2ce75-kserve-c454bd5f9-vgxtv\" (UID: \"404ea200-b989-4029-be47-f7075c3ed65e\") " pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:00:15.883018 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:15.882983    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/404ea200-b989-4029-be47-f7075c3ed65e-tmp-dir\") pod \"premium-w2-c2ce75-kserve-c454bd5f9-vgxtv\" (UID: \"404ea200-b989-4029-be47-f7075c3ed65e\") " pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:00:15.883018 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:15.883013    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/404ea200-b989-4029-be47-f7075c3ed65e-home\") pod \"premium-w2-c2ce75-kserve-c454bd5f9-vgxtv\" (UID: \"404ea200-b989-4029-be47-f7075c3ed65e\") " pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:00:15.883499 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:15.883475    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/404ea200-b989-4029-be47-f7075c3ed65e-home\") pod \"premium-w2-c2ce75-kserve-c454bd5f9-vgxtv\" (UID: \"404ea200-b989-4029-be47-f7075c3ed65e\") " pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:00:15.900244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:15.889916    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/404ea200-b989-4029-be47-f7075c3ed65e-model-cache\") pod \"premium-w2-c2ce75-kserve-c454bd5f9-vgxtv\" (UID: \"404ea200-b989-4029-be47-f7075c3ed65e\") " pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:00:15.900244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:15.890330    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/404ea200-b989-4029-be47-f7075c3ed65e-tmp-dir\") pod \"premium-w2-c2ce75-kserve-c454bd5f9-vgxtv\" (UID: \"404ea200-b989-4029-be47-f7075c3ed65e\") " pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:00:15.924555 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:15.924520    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/404ea200-b989-4029-be47-f7075c3ed65e-dshm\") pod \"premium-w2-c2ce75-kserve-c454bd5f9-vgxtv\" (UID: \"404ea200-b989-4029-be47-f7075c3ed65e\") " pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:00:15.926757 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:15.926675    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/404ea200-b989-4029-be47-f7075c3ed65e-tls-certs\") pod \"premium-w2-c2ce75-kserve-c454bd5f9-vgxtv\" (UID: \"404ea200-b989-4029-be47-f7075c3ed65e\") " pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:00:15.929324 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:15.929191    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhpjc\" (UniqueName: \"kubernetes.io/projected/404ea200-b989-4029-be47-f7075c3ed65e-kube-api-access-mhpjc\") pod \"premium-w2-c2ce75-kserve-c454bd5f9-vgxtv\" (UID: \"404ea200-b989-4029-be47-f7075c3ed65e\") " pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:00:16.001425 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:16.001381    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:00:16.317543 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:16.317452    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"]
Oct 30 13:00:16.321948 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:00:16.321905    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod404ea200_b989_4029_be47_f7075c3ed65e.slice/crio-532b4634d1dbd31c2485b98c67c01d808aa0f0c03453dc2e06866b967ffc21ba WatchSource:0}: Error finding container 532b4634d1dbd31c2485b98c67c01d808aa0f0c03453dc2e06866b967ffc21ba: Status 404 returned error can't find the container with id 532b4634d1dbd31c2485b98c67c01d808aa0f0c03453dc2e06866b967ffc21ba
Oct 30 13:00:16.768984 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:16.768948    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:16.872158 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:16.872121    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" event={"ID":"404ea200-b989-4029-be47-f7075c3ed65e","Type":"ContainerStarted","Data":"8919966ee1cfb5400f07f9d90a533886b4de9baf5d7c84e98271d101ffbc6ec0"}
Oct 30 13:00:16.872416 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:16.872169    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" event={"ID":"404ea200-b989-4029-be47-f7075c3ed65e","Type":"ContainerStarted","Data":"532b4634d1dbd31c2485b98c67c01d808aa0f0c03453dc2e06866b967ffc21ba"}
Oct 30 13:00:16.916582 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:16.916528    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podStartSLOduration=1.9165044230000001 podStartE2EDuration="1.916504423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:16.914439114 +0000 UTC m=+911.961573067" watchObservedRunningTime="2026-10-30 13:00:16.916504423 +0000 UTC m=+911.963638375"
Oct 30 13:00:17.001816 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:17.001773    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:00:17.078695 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:17.078660    4544 scope.go:122] "RemoveContainer" containerID="5439f38c197bcaf9f7a8c86e1bf0cb431136db678553df9c1df9dacf70f08446"
Oct 30 13:00:17.880214 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:17.880172    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54" event={"ID":"c52ae920-7884-4f8b-b7cf-7e853416af2c","Type":"ContainerStarted","Data":"edffbdcc75ca53edc9dc522b7eda3d0e3dfbff815779858785e637cdba6c52c8"}
Oct 30 13:00:18.078465 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:18.078427    4544 scope.go:122] "RemoveContainer" containerID="004a7040be7751027759a1f906dccaaee6b610af822a18f322a46539b2df0b3f"
Oct 30 13:00:18.393043 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:18.393009    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bbaf346-j2c57"]
Oct 30 13:00:18.398647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:18.398620    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbaf346-j2c57"
Oct 30 13:00:18.415909 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:18.415881    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:00:18.430790 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:18.430755    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bbaf346-j2c57"]
Oct 30 13:00:18.523905 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:18.523865    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dczqr\" (UniqueName: \"kubernetes.io/projected/6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c-kube-api-access-dczqr\") pod \"maas-api-revoke-sub-5bbaf346-j2c57\" (UID: \"6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbaf346-j2c57"
Oct 30 13:00:18.524101 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:18.523925    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bbaf346-j2c57\" (UID: \"6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbaf346-j2c57"
Oct 30 13:00:18.524101 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:18.524083    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bbaf346-j2c57\" (UID: \"6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbaf346-j2c57"
Oct 30 13:00:18.564466 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:18.564426    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:18.626400 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:18.626294    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bbaf346-j2c57\" (UID: \"6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbaf346-j2c57"
Oct 30 13:00:18.626598 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:18.626478    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dczqr\" (UniqueName: \"kubernetes.io/projected/6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c-kube-api-access-dczqr\") pod \"maas-api-revoke-sub-5bbaf346-j2c57\" (UID: \"6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbaf346-j2c57"
Oct 30 13:00:18.626598 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:18.626540    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bbaf346-j2c57\" (UID: \"6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbaf346-j2c57"
Oct 30 13:00:18.630927 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:18.627352    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bbaf346-j2c57\" (UID: \"6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbaf346-j2c57"
Oct 30 13:00:18.630927 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:18.630134    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bbaf346-j2c57\" (UID: \"6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbaf346-j2c57"
Oct 30 13:00:18.654105 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:18.654067    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dczqr\" (UniqueName: \"kubernetes.io/projected/6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c-kube-api-access-dczqr\") pod \"maas-api-revoke-sub-5bbaf346-j2c57\" (UID: \"6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbaf346-j2c57"
Oct 30 13:00:18.711776 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:18.711733    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbaf346-j2c57"
Oct 30 13:00:18.888766 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:18.888683    4544 generic.go:356] "Generic (PLEG): container finished" podID="c52ae920-7884-4f8b-b7cf-7e853416af2c" containerID="edffbdcc75ca53edc9dc522b7eda3d0e3dfbff815779858785e637cdba6c52c8" exitCode=60
Oct 30 13:00:18.888931 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:18.888753    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54" event={"ID":"c52ae920-7884-4f8b-b7cf-7e853416af2c","Type":"ContainerDied","Data":"edffbdcc75ca53edc9dc522b7eda3d0e3dfbff815779858785e637cdba6c52c8"}
Oct 30 13:00:18.888931 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:18.888820    4544 scope.go:122] "RemoveContainer" containerID="5439f38c197bcaf9f7a8c86e1bf0cb431136db678553df9c1df9dacf70f08446"
Oct 30 13:00:18.889243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:18.889211    4544 scope.go:122] "RemoveContainer" containerID="edffbdcc75ca53edc9dc522b7eda3d0e3dfbff815779858785e637cdba6c52c8"
Oct 30 13:00:18.889550 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:00:18.889524    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54_odh-ai-gateway-infra(c52ae920-7884-4f8b-b7cf-7e853416af2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54" podUID="c52ae920-7884-4f8b-b7cf-7e853416af2c"
Oct 30 13:00:18.890752 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:18.890727    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs" event={"ID":"63284ee7-cdee-4456-a64a-72ea663b19a4","Type":"ContainerStarted","Data":"a2799d1701174ebdd26c8fff9244fcc6535c77beb1a2617e1cb94e3ef4ae1d75"}
Oct 30 13:00:18.939446 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:18.939417    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54"]
Oct 30 13:00:19.016622 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:19.016591    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bbaf346-j2c57"]
Oct 30 13:00:19.020062 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:00:19.020032    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aeaad1a_ebfb_4bf0_8c6f_d281d2cc6f7c.slice/crio-14917a661e5717c97546058160002118928f5dd35ede3fbaa0d22868e76597f7 WatchSource:0}: Error finding container 14917a661e5717c97546058160002118928f5dd35ede3fbaa0d22868e76597f7: Status 404 returned error can't find the container with id 14917a661e5717c97546058160002118928f5dd35ede3fbaa0d22868e76597f7
Oct 30 13:00:19.900922 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:19.900879    4544 generic.go:356] "Generic (PLEG): container finished" podID="63284ee7-cdee-4456-a64a-72ea663b19a4" containerID="a2799d1701174ebdd26c8fff9244fcc6535c77beb1a2617e1cb94e3ef4ae1d75" exitCode=60
Oct 30 13:00:19.901360 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:19.900947    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs" event={"ID":"63284ee7-cdee-4456-a64a-72ea663b19a4","Type":"ContainerDied","Data":"a2799d1701174ebdd26c8fff9244fcc6535c77beb1a2617e1cb94e3ef4ae1d75"}
Oct 30 13:00:19.901360 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:19.901009    4544 scope.go:122] "RemoveContainer" containerID="004a7040be7751027759a1f906dccaaee6b610af822a18f322a46539b2df0b3f"
Oct 30 13:00:19.901468 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:19.901442    4544 scope.go:122] "RemoveContainer" containerID="a2799d1701174ebdd26c8fff9244fcc6535c77beb1a2617e1cb94e3ef4ae1d75"
Oct 30 13:00:19.901790 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:00:19.901763    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs_odh-ai-gateway-infra(63284ee7-cdee-4456-a64a-72ea663b19a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs" podUID="63284ee7-cdee-4456-a64a-72ea663b19a4"
Oct 30 13:00:19.906844 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:19.906813    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbaf346-j2c57" event={"ID":"6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c","Type":"ContainerStarted","Data":"125e98e769a7e32dfd706a14f747dc1e768928561225ba31468f5328eb20143f"}
Oct 30 13:00:19.907028 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:19.906875    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbaf346-j2c57" event={"ID":"6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c","Type":"ContainerStarted","Data":"14917a661e5717c97546058160002118928f5dd35ede3fbaa0d22868e76597f7"}
Oct 30 13:00:20.018781 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:20.018738    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs"]
Oct 30 13:00:20.063699 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:20.063669    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54"
Oct 30 13:00:20.073068 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:20.072748    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbaf346-j2c57" podStartSLOduration=2.072730188 podStartE2EDuration="2.072730188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:20.069913317 +0000 UTC m=+915.117047272" watchObservedRunningTime="2026-10-30 13:00:20.072730188 +0000 UTC m=+915.119864142"
Oct 30 13:00:20.078784 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:20.078755    4544 scope.go:122] "RemoveContainer" containerID="f37847c7faf8aa987baf4ace84ae7de545841cf0eace68557d1bbc3dff48b8d7"
Oct 30 13:00:20.078971 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:20.078795    4544 scope.go:122] "RemoveContainer" containerID="d06049f9d583a71b2edf57e21f7ec3924e6bce0fec750ac587f8ff940ec597cb"
Oct 30 13:00:20.143799 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:20.143768    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c52ae920-7884-4f8b-b7cf-7e853416af2c-kube-api-access-zfzmp\" (UniqueName: \"kubernetes.io/projected/c52ae920-7884-4f8b-b7cf-7e853416af2c-kube-api-access-zfzmp\") pod \"c52ae920-7884-4f8b-b7cf-7e853416af2c\" (UID: \"c52ae920-7884-4f8b-b7cf-7e853416af2c\") "
Oct 30 13:00:20.143962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:20.143859    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c52ae920-7884-4f8b-b7cf-7e853416af2c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c52ae920-7884-4f8b-b7cf-7e853416af2c-maas-api-service-ca\") pod \"c52ae920-7884-4f8b-b7cf-7e853416af2c\" (UID: \"c52ae920-7884-4f8b-b7cf-7e853416af2c\") "
Oct 30 13:00:20.144698 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:20.144256    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c52ae920-7884-4f8b-b7cf-7e853416af2c-maas-api-service-ca" pod "c52ae920-7884-4f8b-b7cf-7e853416af2c" (UID: "c52ae920-7884-4f8b-b7cf-7e853416af2c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:20.144698 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:20.144521    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c52ae920-7884-4f8b-b7cf-7e853416af2c-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:00:20.147420 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:20.147398    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c52ae920-7884-4f8b-b7cf-7e853416af2c-kube-api-access-zfzmp" pod "c52ae920-7884-4f8b-b7cf-7e853416af2c" (UID: "c52ae920-7884-4f8b-b7cf-7e853416af2c"). InnerVolumeSpecName "kube-api-access-zfzmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:20.245404 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:20.245363    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfzmp\" (UniqueName: \"kubernetes.io/projected/c52ae920-7884-4f8b-b7cf-7e853416af2c-kube-api-access-zfzmp\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:00:20.598920 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:20.598881    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:20.914571 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:20.914538    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54"
Oct 30 13:00:20.914571 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:20.914559    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54" event={"ID":"c52ae920-7884-4f8b-b7cf-7e853416af2c","Type":"ContainerDied","Data":"c1e808f95e2855036a8828bbaecaf1f33b5a7238bf4db80c120721f31b6f8c35"}
Oct 30 13:00:20.915144 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:20.914626    4544 scope.go:122] "RemoveContainer" containerID="edffbdcc75ca53edc9dc522b7eda3d0e3dfbff815779858785e637cdba6c52c8"
Oct 30 13:00:20.920410 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:20.920375    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp" event={"ID":"82aafa40-b786-4399-8e45-96982391939c","Type":"ContainerStarted","Data":"8faeb32ce8a35e5fd7ae1f86bd08d3d6c6446a660d46c5ad823747a9325fb8a0"}
Oct 30 13:00:20.922789 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:20.922755    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr" event={"ID":"4e0369cc-9862-4842-b130-171298422b33","Type":"ContainerStarted","Data":"9d8628d882b533d4ebe2b630ec802d1c7920fe4fa8ccdcba21562d8431d66ce8"}
Oct 30 13:00:20.924929 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:20.924761    4544 generic.go:356] "Generic (PLEG): container finished" podID="6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c" containerID="125e98e769a7e32dfd706a14f747dc1e768928561225ba31468f5328eb20143f" exitCode=0
Oct 30 13:00:20.924929 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:20.924834    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbaf346-j2c57" event={"ID":"6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c","Type":"ContainerDied","Data":"125e98e769a7e32dfd706a14f747dc1e768928561225ba31468f5328eb20143f"}
Oct 30 13:00:21.026500 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:21.026462    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54"]
Oct 30 13:00:21.031415 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:21.031372    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889420-q7h54"]
Oct 30 13:00:21.086418 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:21.086390    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs"
Oct 30 13:00:21.087801 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:21.087777    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c52ae920-7884-4f8b-b7cf-7e853416af2c" path="/var/lib/kubelet/pods/c52ae920-7884-4f8b-b7cf-7e853416af2c/volumes"
Oct 30 13:00:21.157009 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:21.156972    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63284ee7-cdee-4456-a64a-72ea663b19a4-kube-api-access-rwmt5\" (UniqueName: \"kubernetes.io/projected/63284ee7-cdee-4456-a64a-72ea663b19a4-kube-api-access-rwmt5\") pod \"63284ee7-cdee-4456-a64a-72ea663b19a4\" (UID: \"63284ee7-cdee-4456-a64a-72ea663b19a4\") "
Oct 30 13:00:21.157190 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:21.157162    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63284ee7-cdee-4456-a64a-72ea663b19a4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63284ee7-cdee-4456-a64a-72ea663b19a4-maas-api-service-ca\") pod \"63284ee7-cdee-4456-a64a-72ea663b19a4\" (UID: \"63284ee7-cdee-4456-a64a-72ea663b19a4\") "
Oct 30 13:00:21.157528 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:21.157503    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63284ee7-cdee-4456-a64a-72ea663b19a4-maas-api-service-ca" pod "63284ee7-cdee-4456-a64a-72ea663b19a4" (UID: "63284ee7-cdee-4456-a64a-72ea663b19a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:21.159183 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:21.159156    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63284ee7-cdee-4456-a64a-72ea663b19a4-kube-api-access-rwmt5" pod "63284ee7-cdee-4456-a64a-72ea663b19a4" (UID: "63284ee7-cdee-4456-a64a-72ea663b19a4"). InnerVolumeSpecName "kube-api-access-rwmt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:21.258784 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:21.258745    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwmt5\" (UniqueName: \"kubernetes.io/projected/63284ee7-cdee-4456-a64a-72ea663b19a4-kube-api-access-rwmt5\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:00:21.258784 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:21.258777    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63284ee7-cdee-4456-a64a-72ea663b19a4-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:00:21.934051 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:21.934013    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs" event={"ID":"63284ee7-cdee-4456-a64a-72ea663b19a4","Type":"ContainerDied","Data":"c1e83aca0371d31820f3e384bde03e1b65bc3ffced58326a5768e0716aec8c5f"}
Oct 30 13:00:21.934555 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:21.934084    4544 scope.go:122] "RemoveContainer" containerID="a2799d1701174ebdd26c8fff9244fcc6535c77beb1a2617e1cb94e3ef4ae1d75"
Oct 30 13:00:21.934555 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:21.934025    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs"
Oct 30 13:00:21.936199 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:21.936175    4544 generic.go:356] "Generic (PLEG): container finished" podID="82aafa40-b786-4399-8e45-96982391939c" containerID="8faeb32ce8a35e5fd7ae1f86bd08d3d6c6446a660d46c5ad823747a9325fb8a0" exitCode=60
Oct 30 13:00:21.936319 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:21.936250    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp" event={"ID":"82aafa40-b786-4399-8e45-96982391939c","Type":"ContainerDied","Data":"8faeb32ce8a35e5fd7ae1f86bd08d3d6c6446a660d46c5ad823747a9325fb8a0"}
Oct 30 13:00:21.936670 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:21.936651    4544 scope.go:122] "RemoveContainer" containerID="8faeb32ce8a35e5fd7ae1f86bd08d3d6c6446a660d46c5ad823747a9325fb8a0"
Oct 30 13:00:21.936990 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:00:21.936970    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp_odh-ai-gateway-infra(82aafa40-b786-4399-8e45-96982391939c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp" podUID="82aafa40-b786-4399-8e45-96982391939c"
Oct 30 13:00:21.938259 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:21.938207    4544 generic.go:356] "Generic (PLEG): container finished" podID="4e0369cc-9862-4842-b130-171298422b33" containerID="9d8628d882b533d4ebe2b630ec802d1c7920fe4fa8ccdcba21562d8431d66ce8" exitCode=60
Oct 30 13:00:21.938259 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:21.938248    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr" event={"ID":"4e0369cc-9862-4842-b130-171298422b33","Type":"ContainerDied","Data":"9d8628d882b533d4ebe2b630ec802d1c7920fe4fa8ccdcba21562d8431d66ce8"}
Oct 30 13:00:21.938620 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:21.938597    4544 scope.go:122] "RemoveContainer" containerID="9d8628d882b533d4ebe2b630ec802d1c7920fe4fa8ccdcba21562d8431d66ce8"
Oct 30 13:00:21.938868 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:00:21.938850    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr_odh-ai-gateway-infra(4e0369cc-9862-4842-b130-171298422b33)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr" podUID="4e0369cc-9862-4842-b130-171298422b33"
Oct 30 13:00:21.945633 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:21.945605    4544 scope.go:122] "RemoveContainer" containerID="f37847c7faf8aa987baf4ace84ae7de545841cf0eace68557d1bbc3dff48b8d7"
Oct 30 13:00:21.958952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:21.958926    4544 scope.go:122] "RemoveContainer" containerID="d06049f9d583a71b2edf57e21f7ec3924e6bce0fec750ac587f8ff940ec597cb"
Oct 30 13:00:21.992642 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:21.992605    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp"]
Oct 30 13:00:22.080650 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.080619    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs"]
Oct 30 13:00:22.091348 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.091314    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889420-m4trs"]
Oct 30 13:00:22.097095 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.097070    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbaf346-j2c57"
Oct 30 13:00:22.169639 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.169608    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c-kube-api-access-dczqr\" (UniqueName: \"kubernetes.io/projected/6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c-kube-api-access-dczqr\") pod \"6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c\" (UID: \"6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c\") "
Oct 30 13:00:22.169798 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.169715    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c-maas-api-xks-ca\") pod \"6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c\" (UID: \"6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c\") "
Oct 30 13:00:22.169798 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.169778    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c-maas-api-openshift-ca\") pod \"6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c\" (UID: \"6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c\") "
Oct 30 13:00:22.170199 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.170170    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c-maas-api-openshift-ca" pod "6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c" (UID: "6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:22.172514 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.172486    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c-kube-api-access-dczqr" pod "6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c" (UID: "6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c"). InnerVolumeSpecName "kube-api-access-dczqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:22.172632 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.172490    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c-maas-api-xks-ca" pod "6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c" (UID: "6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:00:22.196987 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.196898    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr"]
Oct 30 13:00:22.270813 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.270781    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:00:22.270813 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.270811    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:00:22.271039 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.270820    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dczqr\" (UniqueName: \"kubernetes.io/projected/6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c-kube-api-access-dczqr\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:00:22.451582 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.451486    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:22.602976 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.602940    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"]
Oct 30 13:00:22.603520 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.603502    4544 cpu_manager.go:414] "containerMap: removing container" podUID="63284ee7-cdee-4456-a64a-72ea663b19a4" containerName="cleanup"
Oct 30 13:00:22.603575 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.603524    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63284ee7-cdee-4456-a64a-72ea663b19a4" containerName="cleanup"
Oct 30 13:00:22.603575 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.603550    4544 cpu_manager.go:414] "containerMap: removing container" podUID="63284ee7-cdee-4456-a64a-72ea663b19a4" containerName="cleanup"
Oct 30 13:00:22.603575 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.603557    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63284ee7-cdee-4456-a64a-72ea663b19a4" containerName="cleanup"
Oct 30 13:00:22.603575 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.603572    4544 cpu_manager.go:414] "containerMap: removing container" podUID="c52ae920-7884-4f8b-b7cf-7e853416af2c" containerName="cleanup"
Oct 30 13:00:22.603690 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.603580    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c52ae920-7884-4f8b-b7cf-7e853416af2c" containerName="cleanup"
Oct 30 13:00:22.603690 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.603600    4544 cpu_manager.go:414] "containerMap: removing container" podUID="c52ae920-7884-4f8b-b7cf-7e853416af2c" containerName="cleanup"
Oct 30 13:00:22.603690 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.603604    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c52ae920-7884-4f8b-b7cf-7e853416af2c" containerName="cleanup"
Oct 30 13:00:22.603690 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.603612    4544 cpu_manager.go:414] "containerMap: removing container" podUID="6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c" containerName="revoke-keys"
Oct 30 13:00:22.603690 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.603616    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c" containerName="revoke-keys"
Oct 30 13:00:22.603690 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.603625    4544 cpu_manager.go:414] "containerMap: removing container" podUID="c52ae920-7884-4f8b-b7cf-7e853416af2c" containerName="cleanup"
Oct 30 13:00:22.603690 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.603630    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c52ae920-7884-4f8b-b7cf-7e853416af2c" containerName="cleanup"
Oct 30 13:00:22.603887 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.603704    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="63284ee7-cdee-4456-a64a-72ea663b19a4" containerName="cleanup"
Oct 30 13:00:22.603887 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.603712    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="c52ae920-7884-4f8b-b7cf-7e853416af2c" containerName="cleanup"
Oct 30 13:00:22.603887 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.603717    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c" containerName="revoke-keys"
Oct 30 13:00:22.603887 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.603725    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="c52ae920-7884-4f8b-b7cf-7e853416af2c" containerName="cleanup"
Oct 30 13:00:22.603887 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.603733    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="63284ee7-cdee-4456-a64a-72ea663b19a4" containerName="cleanup"
Oct 30 13:00:22.619545 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.619503    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:00:22.634903 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.634826    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-67f56e\"/\"openshift-service-ca.crt\""
Oct 30 13:00:22.635215 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.635193    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-67f56e\"/\"kube-root-ca.crt\""
Oct 30 13:00:22.635393 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.635249    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-67f56e\"/\"premium-w4-67f56e-kserve-self-signed-certs\""
Oct 30 13:00:22.635393 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.635299    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-67f56e\"/\"default-dockercfg-7ldk6\""
Oct 30 13:00:22.636427 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.636401    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"]
Oct 30 13:00:22.665844 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.665798    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:22.776095 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.776004    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8be6bd4f-3790-489b-8a6e-edcd85f33aec-home\") pod \"premium-w4-67f56e-kserve-5dcb66fdff-4jdgs\" (UID: \"8be6bd4f-3790-489b-8a6e-edcd85f33aec\") " pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:00:22.776095 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.776071    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjclj\" (UniqueName: \"kubernetes.io/projected/8be6bd4f-3790-489b-8a6e-edcd85f33aec-kube-api-access-kjclj\") pod \"premium-w4-67f56e-kserve-5dcb66fdff-4jdgs\" (UID: \"8be6bd4f-3790-489b-8a6e-edcd85f33aec\") " pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:00:22.776336 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.776277    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8be6bd4f-3790-489b-8a6e-edcd85f33aec-tmp-dir\") pod \"premium-w4-67f56e-kserve-5dcb66fdff-4jdgs\" (UID: \"8be6bd4f-3790-489b-8a6e-edcd85f33aec\") " pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:00:22.776336 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.776308    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8be6bd4f-3790-489b-8a6e-edcd85f33aec-model-cache\") pod \"premium-w4-67f56e-kserve-5dcb66fdff-4jdgs\" (UID: \"8be6bd4f-3790-489b-8a6e-edcd85f33aec\") " pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:00:22.776336 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.776328    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8be6bd4f-3790-489b-8a6e-edcd85f33aec-dshm\") pod \"premium-w4-67f56e-kserve-5dcb66fdff-4jdgs\" (UID: \"8be6bd4f-3790-489b-8a6e-edcd85f33aec\") " pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:00:22.776433 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.776416    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8be6bd4f-3790-489b-8a6e-edcd85f33aec-tls-certs\") pod \"premium-w4-67f56e-kserve-5dcb66fdff-4jdgs\" (UID: \"8be6bd4f-3790-489b-8a6e-edcd85f33aec\") " pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:00:22.880031 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.879979    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8be6bd4f-3790-489b-8a6e-edcd85f33aec-home\") pod \"premium-w4-67f56e-kserve-5dcb66fdff-4jdgs\" (UID: \"8be6bd4f-3790-489b-8a6e-edcd85f33aec\") " pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:00:22.880910 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.880884    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjclj\" (UniqueName: \"kubernetes.io/projected/8be6bd4f-3790-489b-8a6e-edcd85f33aec-kube-api-access-kjclj\") pod \"premium-w4-67f56e-kserve-5dcb66fdff-4jdgs\" (UID: \"8be6bd4f-3790-489b-8a6e-edcd85f33aec\") " pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:00:22.881912 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.881888    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8be6bd4f-3790-489b-8a6e-edcd85f33aec-tmp-dir\") pod \"premium-w4-67f56e-kserve-5dcb66fdff-4jdgs\" (UID: \"8be6bd4f-3790-489b-8a6e-edcd85f33aec\") " pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:00:22.882475 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.882456    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8be6bd4f-3790-489b-8a6e-edcd85f33aec-model-cache\") pod \"premium-w4-67f56e-kserve-5dcb66fdff-4jdgs\" (UID: \"8be6bd4f-3790-489b-8a6e-edcd85f33aec\") " pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:00:22.882880 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.882864    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8be6bd4f-3790-489b-8a6e-edcd85f33aec-dshm\") pod \"premium-w4-67f56e-kserve-5dcb66fdff-4jdgs\" (UID: \"8be6bd4f-3790-489b-8a6e-edcd85f33aec\") " pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:00:22.883058 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.883038    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8be6bd4f-3790-489b-8a6e-edcd85f33aec-tls-certs\") pod \"premium-w4-67f56e-kserve-5dcb66fdff-4jdgs\" (UID: \"8be6bd4f-3790-489b-8a6e-edcd85f33aec\") " pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:00:22.884517 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.882823    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8be6bd4f-3790-489b-8a6e-edcd85f33aec-model-cache\") pod \"premium-w4-67f56e-kserve-5dcb66fdff-4jdgs\" (UID: \"8be6bd4f-3790-489b-8a6e-edcd85f33aec\") " pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:00:22.884517 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.882390    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8be6bd4f-3790-489b-8a6e-edcd85f33aec-tmp-dir\") pod \"premium-w4-67f56e-kserve-5dcb66fdff-4jdgs\" (UID: \"8be6bd4f-3790-489b-8a6e-edcd85f33aec\") " pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:00:22.884517 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.880742    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8be6bd4f-3790-489b-8a6e-edcd85f33aec-home\") pod \"premium-w4-67f56e-kserve-5dcb66fdff-4jdgs\" (UID: \"8be6bd4f-3790-489b-8a6e-edcd85f33aec\") " pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:00:22.901078 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.901044    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8be6bd4f-3790-489b-8a6e-edcd85f33aec-tls-certs\") pod \"premium-w4-67f56e-kserve-5dcb66fdff-4jdgs\" (UID: \"8be6bd4f-3790-489b-8a6e-edcd85f33aec\") " pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:00:22.911796 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.911757    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8be6bd4f-3790-489b-8a6e-edcd85f33aec-dshm\") pod \"premium-w4-67f56e-kserve-5dcb66fdff-4jdgs\" (UID: \"8be6bd4f-3790-489b-8a6e-edcd85f33aec\") " pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:00:22.924559 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.914568    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjclj\" (UniqueName: \"kubernetes.io/projected/8be6bd4f-3790-489b-8a6e-edcd85f33aec-kube-api-access-kjclj\") pod \"premium-w4-67f56e-kserve-5dcb66fdff-4jdgs\" (UID: \"8be6bd4f-3790-489b-8a6e-edcd85f33aec\") " pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:00:22.934266 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.932159    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:00:22.962013 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.961965    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbaf346-j2c57" event={"ID":"6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c","Type":"ContainerDied","Data":"14917a661e5717c97546058160002118928f5dd35ede3fbaa0d22868e76597f7"}
Oct 30 13:00:22.962181 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.962055    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14917a661e5717c97546058160002118928f5dd35ede3fbaa0d22868e76597f7"
Oct 30 13:00:22.962181 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:22.962164    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbaf346-j2c57"
Oct 30 13:00:23.102086 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.102049    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63284ee7-cdee-4456-a64a-72ea663b19a4" path="/var/lib/kubelet/pods/63284ee7-cdee-4456-a64a-72ea663b19a4/volumes"
Oct 30 13:00:23.258347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.258308    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr"
Oct 30 13:00:23.264418 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.264371    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp"
Oct 30 13:00:23.275198 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.274089    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"]
Oct 30 13:00:23.278665 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:00:23.277964    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8be6bd4f_3790_489b_8a6e_edcd85f33aec.slice/crio-9c9f4939212f95284c056da0e3737f338d2b0d1053adc89bc419aef3ec4f41b9 WatchSource:0}: Error finding container 9c9f4939212f95284c056da0e3737f338d2b0d1053adc89bc419aef3ec4f41b9: Status 404 returned error can't find the container with id 9c9f4939212f95284c056da0e3737f338d2b0d1053adc89bc419aef3ec4f41b9
Oct 30 13:00:23.407103 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.407060    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82aafa40-b786-4399-8e45-96982391939c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82aafa40-b786-4399-8e45-96982391939c-maas-api-service-ca\") pod \"82aafa40-b786-4399-8e45-96982391939c\" (UID: \"82aafa40-b786-4399-8e45-96982391939c\") "
Oct 30 13:00:23.407462 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.407435    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82aafa40-b786-4399-8e45-96982391939c-maas-api-service-ca" pod "82aafa40-b786-4399-8e45-96982391939c" (UID: "82aafa40-b786-4399-8e45-96982391939c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:23.407708 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.407687    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82aafa40-b786-4399-8e45-96982391939c-kube-api-access-msqx6\" (UniqueName: \"kubernetes.io/projected/82aafa40-b786-4399-8e45-96982391939c-kube-api-access-msqx6\") pod \"82aafa40-b786-4399-8e45-96982391939c\" (UID: \"82aafa40-b786-4399-8e45-96982391939c\") "
Oct 30 13:00:23.408070 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.407961    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e0369cc-9862-4842-b130-171298422b33-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e0369cc-9862-4842-b130-171298422b33-maas-api-service-ca\") pod \"4e0369cc-9862-4842-b130-171298422b33\" (UID: \"4e0369cc-9862-4842-b130-171298422b33\") "
Oct 30 13:00:23.408719 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.408700    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e0369cc-9862-4842-b130-171298422b33-kube-api-access-58kkd\" (UniqueName: \"kubernetes.io/projected/4e0369cc-9862-4842-b130-171298422b33-kube-api-access-58kkd\") pod \"4e0369cc-9862-4842-b130-171298422b33\" (UID: \"4e0369cc-9862-4842-b130-171298422b33\") "
Oct 30 13:00:23.409117 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.409076    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e0369cc-9862-4842-b130-171298422b33-maas-api-service-ca" pod "4e0369cc-9862-4842-b130-171298422b33" (UID: "4e0369cc-9862-4842-b130-171298422b33"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:23.409732 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.409713    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e0369cc-9862-4842-b130-171298422b33-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:00:23.409825 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.409733    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82aafa40-b786-4399-8e45-96982391939c-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:00:23.411013 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.410942    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e0369cc-9862-4842-b130-171298422b33-kube-api-access-58kkd" pod "4e0369cc-9862-4842-b130-171298422b33" (UID: "4e0369cc-9862-4842-b130-171298422b33"). InnerVolumeSpecName "kube-api-access-58kkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:23.411619 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.411494    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82aafa40-b786-4399-8e45-96982391939c-kube-api-access-msqx6" pod "82aafa40-b786-4399-8e45-96982391939c" (UID: "82aafa40-b786-4399-8e45-96982391939c"). InnerVolumeSpecName "kube-api-access-msqx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:23.511405 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.511365    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msqx6\" (UniqueName: \"kubernetes.io/projected/82aafa40-b786-4399-8e45-96982391939c-kube-api-access-msqx6\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:00:23.511405 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.511399    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58kkd\" (UniqueName: \"kubernetes.io/projected/4e0369cc-9862-4842-b130-171298422b33-kube-api-access-58kkd\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:00:23.970349 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.970307    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp" event={"ID":"82aafa40-b786-4399-8e45-96982391939c","Type":"ContainerDied","Data":"b4bb4ca54c82c1fbb931b0c04bc4de0030b52ab94b9ea60be8bf0d02e2e7cdfc"}
Oct 30 13:00:23.970827 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.970362    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp"
Oct 30 13:00:23.970827 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.970375    4544 scope.go:122] "RemoveContainer" containerID="8faeb32ce8a35e5fd7ae1f86bd08d3d6c6446a660d46c5ad823747a9325fb8a0"
Oct 30 13:00:23.972237 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.972195    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" event={"ID":"8be6bd4f-3790-489b-8a6e-edcd85f33aec","Type":"ContainerStarted","Data":"09dcb1cc0ac2ab86742504cee72d627a19258fea0c0329501902ecb714d95d98"}
Oct 30 13:00:23.972343 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.972254    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" event={"ID":"8be6bd4f-3790-489b-8a6e-edcd85f33aec","Type":"ContainerStarted","Data":"9c9f4939212f95284c056da0e3737f338d2b0d1053adc89bc419aef3ec4f41b9"}
Oct 30 13:00:23.974182 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.974159    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr"
Oct 30 13:00:23.974315 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.974192    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr" event={"ID":"4e0369cc-9862-4842-b130-171298422b33","Type":"ContainerDied","Data":"aeec8e675178ca69ff1feabdd56484bfd4391c4f982051e56b4b100a3dbc44ea"}
Oct 30 13:00:23.979082 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.979053    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"]
Oct 30 13:00:23.979677 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.979655    4544 cpu_manager.go:414] "containerMap: removing container" podUID="4e0369cc-9862-4842-b130-171298422b33" containerName="cleanup"
Oct 30 13:00:23.979677 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.979676    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e0369cc-9862-4842-b130-171298422b33" containerName="cleanup"
Oct 30 13:00:23.979828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.979694    4544 cpu_manager.go:414] "containerMap: removing container" podUID="4e0369cc-9862-4842-b130-171298422b33" containerName="cleanup"
Oct 30 13:00:23.979828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.979703    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e0369cc-9862-4842-b130-171298422b33" containerName="cleanup"
Oct 30 13:00:23.979828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.979745    4544 cpu_manager.go:414] "containerMap: removing container" podUID="82aafa40-b786-4399-8e45-96982391939c" containerName="cleanup"
Oct 30 13:00:23.979828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.979753    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82aafa40-b786-4399-8e45-96982391939c" containerName="cleanup"
Oct 30 13:00:23.979828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.979767    4544 cpu_manager.go:414] "containerMap: removing container" podUID="82aafa40-b786-4399-8e45-96982391939c" containerName="cleanup"
Oct 30 13:00:23.979828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.979774    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82aafa40-b786-4399-8e45-96982391939c" containerName="cleanup"
Oct 30 13:00:23.979828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.979794    4544 cpu_manager.go:414] "containerMap: removing container" podUID="63284ee7-cdee-4456-a64a-72ea663b19a4" containerName="cleanup"
Oct 30 13:00:23.979828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.979801    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63284ee7-cdee-4456-a64a-72ea663b19a4" containerName="cleanup"
Oct 30 13:00:23.979828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.979817    4544 cpu_manager.go:414] "containerMap: removing container" podUID="82aafa40-b786-4399-8e45-96982391939c" containerName="cleanup"
Oct 30 13:00:23.979828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.979824    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82aafa40-b786-4399-8e45-96982391939c" containerName="cleanup"
Oct 30 13:00:23.980280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.979960    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="82aafa40-b786-4399-8e45-96982391939c" containerName="cleanup"
Oct 30 13:00:23.980280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.979973    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="63284ee7-cdee-4456-a64a-72ea663b19a4" containerName="cleanup"
Oct 30 13:00:23.980280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.979984    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e0369cc-9862-4842-b130-171298422b33" containerName="cleanup"
Oct 30 13:00:23.980280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.979993    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="82aafa40-b786-4399-8e45-96982391939c" containerName="cleanup"
Oct 30 13:00:23.980280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.980008    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e0369cc-9862-4842-b130-171298422b33" containerName="cleanup"
Oct 30 13:00:23.980280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.980018    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e0369cc-9862-4842-b130-171298422b33" containerName="cleanup"
Oct 30 13:00:23.980280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.980027    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="c52ae920-7884-4f8b-b7cf-7e853416af2c" containerName="cleanup"
Oct 30 13:00:23.982314 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.982289    4544 scope.go:122] "RemoveContainer" containerID="9d8628d882b533d4ebe2b630ec802d1c7920fe4fa8ccdcba21562d8431d66ce8"
Oct 30 13:00:23.985703 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:23.985680    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:00:24.002390 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.002358    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-09db16-openshift-default-dockercfg-l5hf4\""
Oct 30 13:00:24.014724 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.014126    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"]
Oct 30 13:00:24.043771 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.043707    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podStartSLOduration=2.043688534 podStartE2EDuration="2.043688534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:24.042031077 +0000 UTC m=+919.089165032" watchObservedRunningTime="2026-10-30 13:00:24.043688534 +0000 UTC m=+919.090822487"
Oct 30 13:00:24.094449 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.094412    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr"]
Oct 30 13:00:24.099769 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.099732    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889420-nbphr"]
Oct 30 13:00:24.119852 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.119816    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-workload-certs\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:00:24.120010 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.119859    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-istio-envoy\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:00:24.120069 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.119992    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/297febba-5574-48a5-9e02-71dc942697a9-istiod-ca-cert\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:00:24.120113 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.120084    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/297febba-5574-48a5-9e02-71dc942697a9-istio-podinfo\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:00:24.120190 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.120171    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-workload-socket\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:00:24.120260 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.120206    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-istio-data\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:00:24.120346 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.120330    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-credential-socket\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:00:24.120402 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.120358    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/297febba-5574-48a5-9e02-71dc942697a9-istio-token\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:00:24.120618 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.120596    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4jp8\" (UniqueName: \"kubernetes.io/projected/297febba-5574-48a5-9e02-71dc942697a9-kube-api-access-v4jp8\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:00:24.196687 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.196576    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp"]
Oct 30 13:00:24.206456 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.206420    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889420-czgnp"]
Oct 30 13:00:24.222397 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.222286    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/297febba-5574-48a5-9e02-71dc942697a9-istiod-ca-cert\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:00:24.222397 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.222383    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/297febba-5574-48a5-9e02-71dc942697a9-istio-podinfo\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:00:24.222646 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.222420    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-workload-socket\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:00:24.222646 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.222446    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-istio-data\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:00:24.222646 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.222630    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-credential-socket\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:00:24.222816 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.222673    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/297febba-5574-48a5-9e02-71dc942697a9-istio-token\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:00:24.222873 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.222847    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4jp8\" (UniqueName: \"kubernetes.io/projected/297febba-5574-48a5-9e02-71dc942697a9-kube-api-access-v4jp8\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:00:24.222935 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.222887    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-workload-certs\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:00:24.222935 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.222909    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-istio-envoy\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:00:24.223040 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.222846    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-istio-data\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:00:24.223653 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.223209    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-credential-socket\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:00:24.223653 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.223403    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-workload-socket\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:00:24.223653 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.223485    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-workload-certs\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:00:24.223856 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.223784    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/297febba-5574-48a5-9e02-71dc942697a9-istiod-ca-cert\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:00:24.225319 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.225299    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-istio-envoy\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:00:24.225595 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.225576    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/297febba-5574-48a5-9e02-71dc942697a9-istio-podinfo\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:00:24.250277 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.250213    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/297febba-5574-48a5-9e02-71dc942697a9-istio-token\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:00:24.250433 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.250407    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4jp8\" (UniqueName: \"kubernetes.io/projected/297febba-5574-48a5-9e02-71dc942697a9-kube-api-access-v4jp8\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:00:24.301434 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.301398    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:00:24.650205 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.650022    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"]
Oct 30 13:00:24.652551 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:00:24.652508    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod297febba_5574_48a5_9e02_71dc942697a9.slice/crio-5c3e85e772eee168f07a05d8645f436dd47470fa92cf773f4ac2578ca5fe8e9c WatchSource:0}: Error finding container 5c3e85e772eee168f07a05d8645f436dd47470fa92cf773f4ac2578ca5fe8e9c: Status 404 returned error can't find the container with id 5c3e85e772eee168f07a05d8645f436dd47470fa92cf773f4ac2578ca5fe8e9c
Oct 30 13:00:24.655111 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.655079    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:00:24.655198 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.655154    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:00:24.655198 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.655188    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:00:24.793501 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.793463    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:24.932768 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.932680    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:00:24.985795 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.985708    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q" event={"ID":"297febba-5574-48a5-9e02-71dc942697a9","Type":"ContainerStarted","Data":"b7356a1f4371755d5f71eac9e32dd702dd243387fbb9e1363a8f98a1cee05233"}
Oct 30 13:00:24.986509 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:24.986484    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q" event={"ID":"297febba-5574-48a5-9e02-71dc942697a9","Type":"ContainerStarted","Data":"5c3e85e772eee168f07a05d8645f436dd47470fa92cf773f4ac2578ca5fe8e9c"}
Oct 30 13:00:25.035713 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:25.035653    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q" podStartSLOduration=2.03563589 podStartE2EDuration="2.03563589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:25.033381551 +0000 UTC m=+920.080515517" watchObservedRunningTime="2026-10-30 13:00:25.03563589 +0000 UTC m=+920.082769842"
Oct 30 13:00:25.089475 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:25.089440    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e0369cc-9862-4842-b130-171298422b33" path="/var/lib/kubelet/pods/4e0369cc-9862-4842-b130-171298422b33/volumes"
Oct 30 13:00:25.089872 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:25.089858    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82aafa40-b786-4399-8e45-96982391939c" path="/var/lib/kubelet/pods/82aafa40-b786-4399-8e45-96982391939c/volumes"
Oct 30 13:00:25.301826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:25.301725    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:00:25.303702 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:25.303672    4544 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 13:00:25.303806 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:25.303735    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q" podUID="297febba-5574-48a5-9e02-71dc942697a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 13:00:26.001880 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:26.001840    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:00:26.039553 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:26.039511    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:00:26.302853 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:26.302764    4544 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 13:00:26.303002 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:26.302844    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q" podUID="297febba-5574-48a5-9e02-71dc942697a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 13:00:26.582194 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:26.582156    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:27.008857 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:27.008780    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:00:27.059816 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:27.059781    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:27.302670 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:27.302565    4544 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 13:00:27.302670 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:27.302622    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q" podUID="297febba-5574-48a5-9e02-71dc942697a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 13:00:28.302656 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:28.302609    4544 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 13:00:28.303132 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:28.302672    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q" podUID="297febba-5574-48a5-9e02-71dc942697a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 13:00:28.538855 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:28.538812    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:29.008060 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:00:29.008019    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:00:29.301854 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:29.301755    4544 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 13:00:29.302009 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:29.301842    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q" podUID="297febba-5574-48a5-9e02-71dc942697a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 13:00:30.302492 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:30.302454    4544 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 13:00:30.302960 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:30.302529    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q" podUID="297febba-5574-48a5-9e02-71dc942697a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 13:00:30.444485 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:30.444452    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:30.608018 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:30.607986    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:31.307995 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:31.307953    4544 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 13:00:31.309164 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:31.308656    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q" podUID="297febba-5574-48a5-9e02-71dc942697a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 13:00:32.307090 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:32.307058    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:00:32.307524 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:32.307501    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:00:32.308316 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:32.308295    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:00:32.620323 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:32.620283    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:32.934317 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:32.933950    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:00:32.948497 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:32.948457    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:00:33.056995 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:33.056965    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:00:33.231650 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:33.231550    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f4c3482-lvwn4"]
Oct 30 13:00:33.232711 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:33.232681    4544 cpu_manager.go:414] "containerMap: removing container" podUID="4e0369cc-9862-4842-b130-171298422b33" containerName="cleanup"
Oct 30 13:00:33.232912 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:33.232895    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e0369cc-9862-4842-b130-171298422b33" containerName="cleanup"
Oct 30 13:00:33.233246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:33.233214    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="82aafa40-b786-4399-8e45-96982391939c" containerName="cleanup"
Oct 30 13:00:33.238780 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:33.238756    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4c3482-lvwn4"
Oct 30 13:00:33.259923 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:33.259134    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:00:33.265151 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:33.265114    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f4c3482-lvwn4"]
Oct 30 13:00:33.265550 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:33.265525    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:00:33.373729 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:33.373681    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79fp6\" (UniqueName: \"kubernetes.io/projected/eb59d65b-955e-49b3-9bfb-ae2fa7f98fda-kube-api-access-79fp6\") pod \"maas-api-revoke-sub-2f4c3482-lvwn4\" (UID: \"eb59d65b-955e-49b3-9bfb-ae2fa7f98fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4c3482-lvwn4"
Oct 30 13:00:33.374665 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:33.374631    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb59d65b-955e-49b3-9bfb-ae2fa7f98fda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f4c3482-lvwn4\" (UID: \"eb59d65b-955e-49b3-9bfb-ae2fa7f98fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4c3482-lvwn4"
Oct 30 13:00:33.374834 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:33.374817    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb59d65b-955e-49b3-9bfb-ae2fa7f98fda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f4c3482-lvwn4\" (UID: \"eb59d65b-955e-49b3-9bfb-ae2fa7f98fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4c3482-lvwn4"
Oct 30 13:00:33.475857 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:33.475813    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb59d65b-955e-49b3-9bfb-ae2fa7f98fda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f4c3482-lvwn4\" (UID: \"eb59d65b-955e-49b3-9bfb-ae2fa7f98fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4c3482-lvwn4"
Oct 30 13:00:33.476049 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:33.475897    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb59d65b-955e-49b3-9bfb-ae2fa7f98fda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f4c3482-lvwn4\" (UID: \"eb59d65b-955e-49b3-9bfb-ae2fa7f98fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4c3482-lvwn4"
Oct 30 13:00:33.476049 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:33.475953    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79fp6\" (UniqueName: \"kubernetes.io/projected/eb59d65b-955e-49b3-9bfb-ae2fa7f98fda-kube-api-access-79fp6\") pod \"maas-api-revoke-sub-2f4c3482-lvwn4\" (UID: \"eb59d65b-955e-49b3-9bfb-ae2fa7f98fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4c3482-lvwn4"
Oct 30 13:00:33.476994 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:33.476962    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb59d65b-955e-49b3-9bfb-ae2fa7f98fda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f4c3482-lvwn4\" (UID: \"eb59d65b-955e-49b3-9bfb-ae2fa7f98fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4c3482-lvwn4"
Oct 30 13:00:33.479271 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:33.478936    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb59d65b-955e-49b3-9bfb-ae2fa7f98fda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f4c3482-lvwn4\" (UID: \"eb59d65b-955e-49b3-9bfb-ae2fa7f98fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4c3482-lvwn4"
Oct 30 13:00:33.499561 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:33.499485    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79fp6\" (UniqueName: \"kubernetes.io/projected/eb59d65b-955e-49b3-9bfb-ae2fa7f98fda-kube-api-access-79fp6\") pod \"maas-api-revoke-sub-2f4c3482-lvwn4\" (UID: \"eb59d65b-955e-49b3-9bfb-ae2fa7f98fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4c3482-lvwn4"
Oct 30 13:00:33.575563 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:33.575524    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4c3482-lvwn4"
Oct 30 13:00:33.882580 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:33.882539    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f4c3482-lvwn4"]
Oct 30 13:00:33.886703 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:00:33.886676    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb59d65b_955e_49b3_9bfb_ae2fa7f98fda.slice/crio-7dc00fb9c473d2958d1d2616b3555e59866fc08c9e808f50805ffc3224fb5a48 WatchSource:0}: Error finding container 7dc00fb9c473d2958d1d2616b3555e59866fc08c9e808f50805ffc3224fb5a48: Status 404 returned error can't find the container with id 7dc00fb9c473d2958d1d2616b3555e59866fc08c9e808f50805ffc3224fb5a48
Oct 30 13:00:34.052379 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:34.052347    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4c3482-lvwn4" event={"ID":"eb59d65b-955e-49b3-9bfb-ae2fa7f98fda","Type":"ContainerStarted","Data":"7dc00fb9c473d2958d1d2616b3555e59866fc08c9e808f50805ffc3224fb5a48"}
Oct 30 13:00:34.490107 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:34.490030    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:35.059435 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:35.059348    4544 generic.go:356] "Generic (PLEG): container finished" podID="eb59d65b-955e-49b3-9bfb-ae2fa7f98fda" containerID="6f93200ccff66676b5f9f32f4164c2cb92e8a855419f719eeb685de2ff402874" exitCode=0
Oct 30 13:00:35.059584 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:35.059434    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4c3482-lvwn4" event={"ID":"eb59d65b-955e-49b3-9bfb-ae2fa7f98fda","Type":"ContainerDied","Data":"6f93200ccff66676b5f9f32f4164c2cb92e8a855419f719eeb685de2ff402874"}
Oct 30 13:00:36.152311 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:36.152278    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:36.272144 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:36.272110    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4c3482-lvwn4"
Oct 30 13:00:36.410196 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:36.410156    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb59d65b-955e-49b3-9bfb-ae2fa7f98fda-kube-api-access-79fp6\" (UniqueName: \"kubernetes.io/projected/eb59d65b-955e-49b3-9bfb-ae2fa7f98fda-kube-api-access-79fp6\") pod \"eb59d65b-955e-49b3-9bfb-ae2fa7f98fda\" (UID: \"eb59d65b-955e-49b3-9bfb-ae2fa7f98fda\") "
Oct 30 13:00:36.410196 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:36.410202    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb59d65b-955e-49b3-9bfb-ae2fa7f98fda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb59d65b-955e-49b3-9bfb-ae2fa7f98fda-maas-api-openshift-ca\") pod \"eb59d65b-955e-49b3-9bfb-ae2fa7f98fda\" (UID: \"eb59d65b-955e-49b3-9bfb-ae2fa7f98fda\") "
Oct 30 13:00:36.410496 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:36.410292    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb59d65b-955e-49b3-9bfb-ae2fa7f98fda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb59d65b-955e-49b3-9bfb-ae2fa7f98fda-maas-api-xks-ca\") pod \"eb59d65b-955e-49b3-9bfb-ae2fa7f98fda\" (UID: \"eb59d65b-955e-49b3-9bfb-ae2fa7f98fda\") "
Oct 30 13:00:36.410642 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:36.410616    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb59d65b-955e-49b3-9bfb-ae2fa7f98fda-maas-api-openshift-ca" pod "eb59d65b-955e-49b3-9bfb-ae2fa7f98fda" (UID: "eb59d65b-955e-49b3-9bfb-ae2fa7f98fda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:36.410813 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:36.410799    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb59d65b-955e-49b3-9bfb-ae2fa7f98fda-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:00:36.412466 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:36.412438    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb59d65b-955e-49b3-9bfb-ae2fa7f98fda-maas-api-xks-ca" pod "eb59d65b-955e-49b3-9bfb-ae2fa7f98fda" (UID: "eb59d65b-955e-49b3-9bfb-ae2fa7f98fda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:00:36.412564 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:36.412543    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb59d65b-955e-49b3-9bfb-ae2fa7f98fda-kube-api-access-79fp6" pod "eb59d65b-955e-49b3-9bfb-ae2fa7f98fda" (UID: "eb59d65b-955e-49b3-9bfb-ae2fa7f98fda"). InnerVolumeSpecName "kube-api-access-79fp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:36.511744 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:36.511708    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79fp6\" (UniqueName: \"kubernetes.io/projected/eb59d65b-955e-49b3-9bfb-ae2fa7f98fda-kube-api-access-79fp6\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:00:36.511744 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:36.511741    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb59d65b-955e-49b3-9bfb-ae2fa7f98fda-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:00:37.073199 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:37.073166    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4c3482-lvwn4"
Oct 30 13:00:37.073406 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:37.073193    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f4c3482-lvwn4" event={"ID":"eb59d65b-955e-49b3-9bfb-ae2fa7f98fda","Type":"ContainerDied","Data":"7dc00fb9c473d2958d1d2616b3555e59866fc08c9e808f50805ffc3224fb5a48"}
Oct 30 13:00:37.073406 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:37.073315    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7dc00fb9c473d2958d1d2616b3555e59866fc08c9e808f50805ffc3224fb5a48"
Oct 30 13:00:37.900242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:37.900194    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:39.538113 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.538075    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"]
Oct 30 13:00:39.538659 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.538645    4544 cpu_manager.go:414] "containerMap: removing container" podUID="eb59d65b-955e-49b3-9bfb-ae2fa7f98fda" containerName="revoke-keys"
Oct 30 13:00:39.538705 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.538660    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb59d65b-955e-49b3-9bfb-ae2fa7f98fda" containerName="revoke-keys"
Oct 30 13:00:39.538765 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.538756    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb59d65b-955e-49b3-9bfb-ae2fa7f98fda" containerName="revoke-keys"
Oct 30 13:00:39.542552 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.542525    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:00:39.559582 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.559545    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-c2ce75-openshift-default-dockercfg-44wkt\""
Oct 30 13:00:39.574509 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.574467    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"]
Oct 30 13:00:39.647717 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.647680    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-istio-data\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:00:39.647717 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.647725    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4af86d-c526-4365-b56d-32a7f46247bf-istiod-ca-cert\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:00:39.647938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.647822    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xszpv\" (UniqueName: \"kubernetes.io/projected/9f4af86d-c526-4365-b56d-32a7f46247bf-kube-api-access-xszpv\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:00:39.647938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.647885    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4af86d-c526-4365-b56d-32a7f46247bf-istio-token\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:00:39.648007 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.647955    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-workload-certs\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:00:39.648039 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.648012    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-workload-socket\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:00:39.648039 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.648033    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4af86d-c526-4365-b56d-32a7f46247bf-istio-podinfo\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:00:39.648139 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.648124    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-credential-socket\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:00:39.648181 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.648164    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-istio-envoy\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:00:39.749103 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.749066    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-workload-certs\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:00:39.749302 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.749117    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-workload-socket\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:00:39.749302 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.749139    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4af86d-c526-4365-b56d-32a7f46247bf-istio-podinfo\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:00:39.749302 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.749201    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-credential-socket\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:00:39.749302 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.749285    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-istio-envoy\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:00:39.749556 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.749323    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-istio-data\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:00:39.749556 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.749361    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4af86d-c526-4365-b56d-32a7f46247bf-istiod-ca-cert\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:00:39.749556 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.749422    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xszpv\" (UniqueName: \"kubernetes.io/projected/9f4af86d-c526-4365-b56d-32a7f46247bf-kube-api-access-xszpv\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:00:39.749556 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.749489    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4af86d-c526-4365-b56d-32a7f46247bf-istio-token\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:00:39.749753 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.749599    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-workload-certs\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:00:39.749753 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.749709    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-istio-data\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:00:39.749753 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.749723    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-credential-socket\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:00:39.750050 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.750029    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-workload-socket\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:00:39.750138 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.750123    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4af86d-c526-4365-b56d-32a7f46247bf-istiod-ca-cert\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:00:39.752192 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.752161    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-istio-envoy\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:00:39.752333 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.752321    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4af86d-c526-4365-b56d-32a7f46247bf-istio-podinfo\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:00:39.757131 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.757098    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:39.775062 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.775027    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4af86d-c526-4365-b56d-32a7f46247bf-istio-token\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:00:39.775550 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.775526    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xszpv\" (UniqueName: \"kubernetes.io/projected/9f4af86d-c526-4365-b56d-32a7f46247bf-kube-api-access-xszpv\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:00:39.856959 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:39.856918    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:00:40.197212 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:40.197124    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"]
Oct 30 13:00:40.202515 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:00:40.202471    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f4af86d_c526_4365_b56d_32a7f46247bf.slice/crio-f4e7f7635f092ca98f4310b0c167e1e7813a19fe93e162e9bdf969f4d04f4d35 WatchSource:0}: Error finding container f4e7f7635f092ca98f4310b0c167e1e7813a19fe93e162e9bdf969f4d04f4d35: Status 404 returned error can't find the container with id f4e7f7635f092ca98f4310b0c167e1e7813a19fe93e162e9bdf969f4d04f4d35
Oct 30 13:00:40.205468 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:40.205422    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:00:40.205569 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:40.205527    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:00:40.205609 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:40.205570    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:00:41.159237 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:41.159115    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9" event={"ID":"9f4af86d-c526-4365-b56d-32a7f46247bf","Type":"ContainerStarted","Data":"adab87174c517cf155db0afc4bc2d2335c6d1077e35d5d1dc6eb43b050ae280c"}
Oct 30 13:00:41.159828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:41.159799    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9" event={"ID":"9f4af86d-c526-4365-b56d-32a7f46247bf","Type":"ContainerStarted","Data":"f4e7f7635f092ca98f4310b0c167e1e7813a19fe93e162e9bdf969f4d04f4d35"}
Oct 30 13:00:41.286055 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:41.285978    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9" podStartSLOduration=2.285955553 podStartE2EDuration="2.285955553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:41.282882991 +0000 UTC m=+936.330016944" watchObservedRunningTime="2026-10-30 13:00:41.285955553 +0000 UTC m=+936.333089508"
Oct 30 13:00:41.859308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:41.859261    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:00:41.859760 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:41.859727    4544 patch_prober.go:29] interesting pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 13:00:41.859957 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:41.859928    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9" podUID="9f4af86d-c526-4365-b56d-32a7f46247bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 13:00:42.161177 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:42.161065    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:42.857685 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:42.857647    4544 patch_prober.go:29] interesting pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 13:00:42.857890 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:42.857717    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9" podUID="9f4af86d-c526-4365-b56d-32a7f46247bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 13:00:43.858292 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:43.858248    4544 patch_prober.go:29] interesting pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 13:00:43.858838 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:43.858316    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9" podUID="9f4af86d-c526-4365-b56d-32a7f46247bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 13:00:43.955935 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:43.955895    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:44.858346 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:44.858300    4544 patch_prober.go:29] interesting pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 13:00:44.858769 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:44.858403    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9" podUID="9f4af86d-c526-4365-b56d-32a7f46247bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 13:00:45.857653 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:45.857616    4544 patch_prober.go:29] interesting pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 13:00:45.857863 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:45.857672    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9" podUID="9f4af86d-c526-4365-b56d-32a7f46247bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 13:00:46.095851 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:46.095811    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:46.857789 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:46.857753    4544 patch_prober.go:29] interesting pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 13:00:46.858022 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:46.857818    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9" podUID="9f4af86d-c526-4365-b56d-32a7f46247bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 13:00:47.862503 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:47.862470    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:00:47.863002 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:47.862879    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:00:47.863646 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:47.863615    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:00:47.975401 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:47.975356    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:48.413309 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:48.413272    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23911506-vdb5f"]
Oct 30 13:00:48.418528 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:48.418502    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23911506-vdb5f"
Oct 30 13:00:48.436253 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:48.436202    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:00:48.436451 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:48.436284    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:00:48.439557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:48.439524    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23911506-vdb5f"]
Oct 30 13:00:48.508980 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:48.508946    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37d468c2-42fa-4cf6-903b-167b4780e5b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23911506-vdb5f\" (UID: \"37d468c2-42fa-4cf6-903b-167b4780e5b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23911506-vdb5f"
Oct 30 13:00:48.508980 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:48.508990    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37d468c2-42fa-4cf6-903b-167b4780e5b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23911506-vdb5f\" (UID: \"37d468c2-42fa-4cf6-903b-167b4780e5b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23911506-vdb5f"
Oct 30 13:00:48.509266 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:48.509086    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxf7b\" (UniqueName: \"kubernetes.io/projected/37d468c2-42fa-4cf6-903b-167b4780e5b2-kube-api-access-gxf7b\") pod \"maas-api-revoke-sub-23911506-vdb5f\" (UID: \"37d468c2-42fa-4cf6-903b-167b4780e5b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23911506-vdb5f"
Oct 30 13:00:48.611106 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:48.611033    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37d468c2-42fa-4cf6-903b-167b4780e5b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23911506-vdb5f\" (UID: \"37d468c2-42fa-4cf6-903b-167b4780e5b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23911506-vdb5f"
Oct 30 13:00:48.612146 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:48.611114    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37d468c2-42fa-4cf6-903b-167b4780e5b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23911506-vdb5f\" (UID: \"37d468c2-42fa-4cf6-903b-167b4780e5b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23911506-vdb5f"
Oct 30 13:00:48.612146 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:48.611183    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxf7b\" (UniqueName: \"kubernetes.io/projected/37d468c2-42fa-4cf6-903b-167b4780e5b2-kube-api-access-gxf7b\") pod \"maas-api-revoke-sub-23911506-vdb5f\" (UID: \"37d468c2-42fa-4cf6-903b-167b4780e5b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23911506-vdb5f"
Oct 30 13:00:48.613092 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:48.613063    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37d468c2-42fa-4cf6-903b-167b4780e5b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23911506-vdb5f\" (UID: \"37d468c2-42fa-4cf6-903b-167b4780e5b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23911506-vdb5f"
Oct 30 13:00:48.614868 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:48.614826    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37d468c2-42fa-4cf6-903b-167b4780e5b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23911506-vdb5f\" (UID: \"37d468c2-42fa-4cf6-903b-167b4780e5b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23911506-vdb5f"
Oct 30 13:00:48.634440 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:48.634408    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxf7b\" (UniqueName: \"kubernetes.io/projected/37d468c2-42fa-4cf6-903b-167b4780e5b2-kube-api-access-gxf7b\") pod \"maas-api-revoke-sub-23911506-vdb5f\" (UID: \"37d468c2-42fa-4cf6-903b-167b4780e5b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23911506-vdb5f"
Oct 30 13:00:48.733390 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:48.733303    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23911506-vdb5f"
Oct 30 13:00:49.036410 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:49.036326    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23911506-vdb5f"]
Oct 30 13:00:49.039749 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:00:49.039718    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37d468c2_42fa_4cf6_903b_167b4780e5b2.slice/crio-334eaf570a350e4bbb96a2089d02a83598bf46d39e5c87866b7f777b1d1425e7 WatchSource:0}: Error finding container 334eaf570a350e4bbb96a2089d02a83598bf46d39e5c87866b7f777b1d1425e7: Status 404 returned error can't find the container with id 334eaf570a350e4bbb96a2089d02a83598bf46d39e5c87866b7f777b1d1425e7
Oct 30 13:00:49.272902 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:49.272867    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23911506-vdb5f" event={"ID":"37d468c2-42fa-4cf6-903b-167b4780e5b2","Type":"ContainerStarted","Data":"334eaf570a350e4bbb96a2089d02a83598bf46d39e5c87866b7f777b1d1425e7"}
Oct 30 13:00:49.846747 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:49.846717    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:50.280635 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:50.280546    4544 generic.go:356] "Generic (PLEG): container finished" podID="37d468c2-42fa-4cf6-903b-167b4780e5b2" containerID="7b546a120049a7ba22c8c12bf096f0674f2cef176ce24e5c00acca266d7e4467" exitCode=0
Oct 30 13:00:50.280635 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:50.280611    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23911506-vdb5f" event={"ID":"37d468c2-42fa-4cf6-903b-167b4780e5b2","Type":"ContainerDied","Data":"7b546a120049a7ba22c8c12bf096f0674f2cef176ce24e5c00acca266d7e4467"}
Oct 30 13:00:51.035456 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:51.035414    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3838e9d-pdzg7"]
Oct 30 13:00:51.039447 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:51.039422    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3838e9d-pdzg7"
Oct 30 13:00:51.060430 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:51.060393    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3838e9d-pdzg7"]
Oct 30 13:00:51.144660 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:51.144615    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76d4505d-2bd3-491d-99c2-e4e3e6942327-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3838e9d-pdzg7\" (UID: \"76d4505d-2bd3-491d-99c2-e4e3e6942327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3838e9d-pdzg7"
Oct 30 13:00:51.144660 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:51.144661    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phdqq\" (UniqueName: \"kubernetes.io/projected/76d4505d-2bd3-491d-99c2-e4e3e6942327-kube-api-access-phdqq\") pod \"maas-api-revoke-sub-c3838e9d-pdzg7\" (UID: \"76d4505d-2bd3-491d-99c2-e4e3e6942327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3838e9d-pdzg7"
Oct 30 13:00:51.144857 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:51.144801    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76d4505d-2bd3-491d-99c2-e4e3e6942327-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3838e9d-pdzg7\" (UID: \"76d4505d-2bd3-491d-99c2-e4e3e6942327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3838e9d-pdzg7"
Oct 30 13:00:51.246544 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:51.246491    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76d4505d-2bd3-491d-99c2-e4e3e6942327-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3838e9d-pdzg7\" (UID: \"76d4505d-2bd3-491d-99c2-e4e3e6942327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3838e9d-pdzg7"
Oct 30 13:00:51.246732 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:51.246703    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76d4505d-2bd3-491d-99c2-e4e3e6942327-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3838e9d-pdzg7\" (UID: \"76d4505d-2bd3-491d-99c2-e4e3e6942327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3838e9d-pdzg7"
Oct 30 13:00:51.246779 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:51.246736    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phdqq\" (UniqueName: \"kubernetes.io/projected/76d4505d-2bd3-491d-99c2-e4e3e6942327-kube-api-access-phdqq\") pod \"maas-api-revoke-sub-c3838e9d-pdzg7\" (UID: \"76d4505d-2bd3-491d-99c2-e4e3e6942327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3838e9d-pdzg7"
Oct 30 13:00:51.247349 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:51.247315    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76d4505d-2bd3-491d-99c2-e4e3e6942327-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3838e9d-pdzg7\" (UID: \"76d4505d-2bd3-491d-99c2-e4e3e6942327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3838e9d-pdzg7"
Oct 30 13:00:51.249751 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:51.249722    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76d4505d-2bd3-491d-99c2-e4e3e6942327-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3838e9d-pdzg7\" (UID: \"76d4505d-2bd3-491d-99c2-e4e3e6942327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3838e9d-pdzg7"
Oct 30 13:00:51.270354 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:51.270309    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phdqq\" (UniqueName: \"kubernetes.io/projected/76d4505d-2bd3-491d-99c2-e4e3e6942327-kube-api-access-phdqq\") pod \"maas-api-revoke-sub-c3838e9d-pdzg7\" (UID: \"76d4505d-2bd3-491d-99c2-e4e3e6942327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3838e9d-pdzg7"
Oct 30 13:00:51.353893 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:51.353855    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3838e9d-pdzg7"
Oct 30 13:00:51.598855 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:51.598692    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:51.602808 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:51.602414    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23911506-vdb5f"
Oct 30 13:00:51.681861 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:51.681827    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37d468c2-42fa-4cf6-903b-167b4780e5b2-kube-api-access-gxf7b\" (UniqueName: \"kubernetes.io/projected/37d468c2-42fa-4cf6-903b-167b4780e5b2-kube-api-access-gxf7b\") pod \"37d468c2-42fa-4cf6-903b-167b4780e5b2\" (UID: \"37d468c2-42fa-4cf6-903b-167b4780e5b2\") "
Oct 30 13:00:51.682091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:51.681975    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37d468c2-42fa-4cf6-903b-167b4780e5b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37d468c2-42fa-4cf6-903b-167b4780e5b2-maas-api-openshift-ca\") pod \"37d468c2-42fa-4cf6-903b-167b4780e5b2\" (UID: \"37d468c2-42fa-4cf6-903b-167b4780e5b2\") "
Oct 30 13:00:51.682169 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:51.682091    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37d468c2-42fa-4cf6-903b-167b4780e5b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37d468c2-42fa-4cf6-903b-167b4780e5b2-maas-api-xks-ca\") pod \"37d468c2-42fa-4cf6-903b-167b4780e5b2\" (UID: \"37d468c2-42fa-4cf6-903b-167b4780e5b2\") "
Oct 30 13:00:51.682852 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:51.682795    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37d468c2-42fa-4cf6-903b-167b4780e5b2-maas-api-openshift-ca" pod "37d468c2-42fa-4cf6-903b-167b4780e5b2" (UID: "37d468c2-42fa-4cf6-903b-167b4780e5b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:51.684732 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:51.684702    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37d468c2-42fa-4cf6-903b-167b4780e5b2-maas-api-xks-ca" pod "37d468c2-42fa-4cf6-903b-167b4780e5b2" (UID: "37d468c2-42fa-4cf6-903b-167b4780e5b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:00:51.685495 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:51.685467    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d468c2-42fa-4cf6-903b-167b4780e5b2-kube-api-access-gxf7b" pod "37d468c2-42fa-4cf6-903b-167b4780e5b2" (UID: "37d468c2-42fa-4cf6-903b-167b4780e5b2"). InnerVolumeSpecName "kube-api-access-gxf7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:51.733551 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:51.733516    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3838e9d-pdzg7"]
Oct 30 13:00:51.738264 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:00:51.738232    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76d4505d_2bd3_491d_99c2_e4e3e6942327.slice/crio-ff0ee3c6c7c46313da6232dfcad1dadd0117639503bd1b8ace08b30c98ef41a0 WatchSource:0}: Error finding container ff0ee3c6c7c46313da6232dfcad1dadd0117639503bd1b8ace08b30c98ef41a0: Status 404 returned error can't find the container with id ff0ee3c6c7c46313da6232dfcad1dadd0117639503bd1b8ace08b30c98ef41a0
Oct 30 13:00:51.784315 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:51.784265    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37d468c2-42fa-4cf6-903b-167b4780e5b2-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:00:51.784315 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:51.784316    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37d468c2-42fa-4cf6-903b-167b4780e5b2-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:00:51.784594 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:51.784332    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxf7b\" (UniqueName: \"kubernetes.io/projected/37d468c2-42fa-4cf6-903b-167b4780e5b2-kube-api-access-gxf7b\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:00:52.328246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:52.319264    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23911506-vdb5f" event={"ID":"37d468c2-42fa-4cf6-903b-167b4780e5b2","Type":"ContainerDied","Data":"334eaf570a350e4bbb96a2089d02a83598bf46d39e5c87866b7f777b1d1425e7"}
Oct 30 13:00:52.328246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:52.319332    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="334eaf570a350e4bbb96a2089d02a83598bf46d39e5c87866b7f777b1d1425e7"
Oct 30 13:00:52.328246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:52.319404    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23911506-vdb5f"
Oct 30 13:00:52.361244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:52.352983    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3838e9d-pdzg7" event={"ID":"76d4505d-2bd3-491d-99c2-e4e3e6942327","Type":"ContainerStarted","Data":"ff0ee3c6c7c46313da6232dfcad1dadd0117639503bd1b8ace08b30c98ef41a0"}
Oct 30 13:00:52.866780 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:52.864202    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:53.420245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:53.402831    4544 generic.go:356] "Generic (PLEG): container finished" podID="76d4505d-2bd3-491d-99c2-e4e3e6942327" containerID="a669b8befe9b5f1ceff6dc039e3571b920a2e9089326777d31092f346a90becf" exitCode=0
Oct 30 13:00:53.420245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:53.402925    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3838e9d-pdzg7" event={"ID":"76d4505d-2bd3-491d-99c2-e4e3e6942327","Type":"ContainerDied","Data":"a669b8befe9b5f1ceff6dc039e3571b920a2e9089326777d31092f346a90becf"}
Oct 30 13:00:54.086243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:54.086184    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:55.022246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.021083    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"]
Oct 30 13:00:55.022246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.022038    4544 cpu_manager.go:414] "containerMap: removing container" podUID="37d468c2-42fa-4cf6-903b-167b4780e5b2" containerName="revoke-keys"
Oct 30 13:00:55.022246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.022062    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37d468c2-42fa-4cf6-903b-167b4780e5b2" containerName="revoke-keys"
Oct 30 13:00:55.031252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.030682    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="37d468c2-42fa-4cf6-903b-167b4780e5b2" containerName="revoke-keys"
Oct 30 13:00:55.468078 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.461442    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:00:55.480538 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.480487    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-8b46c9-openshift-default-dockercfg-9flrx\""
Oct 30 13:00:55.541487 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.541442    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-workload-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:00:55.541734 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.541711    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtlr7\" (UniqueName: \"kubernetes.io/projected/ab79901f-7e31-4e95-a033-6eb1235af1fa-kube-api-access-mtlr7\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:00:55.541934 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.541914    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab79901f-7e31-4e95-a033-6eb1235af1fa-istio-token\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:00:55.542063 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.542047    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-workload-certs\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:00:55.542307 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.542287    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-credential-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:00:55.546267 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.545330    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-istio-envoy\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:00:55.546267 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.545408    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-istio-data\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:00:55.546267 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.545496    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab79901f-7e31-4e95-a033-6eb1235af1fa-istiod-ca-cert\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:00:55.546267 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.545629    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab79901f-7e31-4e95-a033-6eb1235af1fa-istio-podinfo\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:00:55.589097 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.589046    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"]
Oct 30 13:00:55.611509 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.611476    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3838e9d-pdzg7"
Oct 30 13:00:55.648131 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.648081    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab79901f-7e31-4e95-a033-6eb1235af1fa-istio-podinfo\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:00:55.649045 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.649016    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-workload-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:00:55.649687 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.649665    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtlr7\" (UniqueName: \"kubernetes.io/projected/ab79901f-7e31-4e95-a033-6eb1235af1fa-kube-api-access-mtlr7\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:00:55.650267 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.650246    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab79901f-7e31-4e95-a033-6eb1235af1fa-istio-token\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:00:55.650434 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.650412    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-workload-certs\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:00:55.651282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.651261    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-credential-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:00:55.651753 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.651732    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-istio-envoy\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:00:55.651918 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.651901    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-istio-data\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:00:55.652109 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.652093    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab79901f-7e31-4e95-a033-6eb1235af1fa-istiod-ca-cert\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:00:55.666805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.649593    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-workload-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:00:55.666805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.651098    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-workload-certs\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:00:55.666805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.654403    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab79901f-7e31-4e95-a033-6eb1235af1fa-istiod-ca-cert\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:00:55.666805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.654679    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-istio-data\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:00:55.666805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.651679    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-credential-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:00:55.666805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.655377    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab79901f-7e31-4e95-a033-6eb1235af1fa-istio-podinfo\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:00:55.666805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.659153    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-istio-envoy\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:00:55.678048 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.677979    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab79901f-7e31-4e95-a033-6eb1235af1fa-istio-token\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:00:55.682935 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.682864    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtlr7\" (UniqueName: \"kubernetes.io/projected/ab79901f-7e31-4e95-a033-6eb1235af1fa-kube-api-access-mtlr7\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:00:55.753283 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.753173    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76d4505d-2bd3-491d-99c2-e4e3e6942327-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76d4505d-2bd3-491d-99c2-e4e3e6942327-maas-api-xks-ca\") pod \"76d4505d-2bd3-491d-99c2-e4e3e6942327\" (UID: \"76d4505d-2bd3-491d-99c2-e4e3e6942327\") "
Oct 30 13:00:55.753439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.753317    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76d4505d-2bd3-491d-99c2-e4e3e6942327-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76d4505d-2bd3-491d-99c2-e4e3e6942327-maas-api-openshift-ca\") pod \"76d4505d-2bd3-491d-99c2-e4e3e6942327\" (UID: \"76d4505d-2bd3-491d-99c2-e4e3e6942327\") "
Oct 30 13:00:55.753495 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.753471    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76d4505d-2bd3-491d-99c2-e4e3e6942327-kube-api-access-phdqq\" (UniqueName: \"kubernetes.io/projected/76d4505d-2bd3-491d-99c2-e4e3e6942327-kube-api-access-phdqq\") pod \"76d4505d-2bd3-491d-99c2-e4e3e6942327\" (UID: \"76d4505d-2bd3-491d-99c2-e4e3e6942327\") "
Oct 30 13:00:55.753748 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.753721    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76d4505d-2bd3-491d-99c2-e4e3e6942327-maas-api-openshift-ca" pod "76d4505d-2bd3-491d-99c2-e4e3e6942327" (UID: "76d4505d-2bd3-491d-99c2-e4e3e6942327"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:55.754010 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.753988    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76d4505d-2bd3-491d-99c2-e4e3e6942327-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:00:55.755418 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.755392    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76d4505d-2bd3-491d-99c2-e4e3e6942327-maas-api-xks-ca" pod "76d4505d-2bd3-491d-99c2-e4e3e6942327" (UID: "76d4505d-2bd3-491d-99c2-e4e3e6942327"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:00:55.755586 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.755559    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76d4505d-2bd3-491d-99c2-e4e3e6942327-kube-api-access-phdqq" pod "76d4505d-2bd3-491d-99c2-e4e3e6942327" (UID: "76d4505d-2bd3-491d-99c2-e4e3e6942327"). InnerVolumeSpecName "kube-api-access-phdqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:55.854913 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.854872    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76d4505d-2bd3-491d-99c2-e4e3e6942327-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:00:55.854913 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.854912    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phdqq\" (UniqueName: \"kubernetes.io/projected/76d4505d-2bd3-491d-99c2-e4e3e6942327-kube-api-access-phdqq\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:00:55.858738 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:55.858703    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:00:56.185539 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:56.185507    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:56.192487 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:56.192455    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"]
Oct 30 13:00:56.196350 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:00:56.196318    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab79901f_7e31_4e95_a033_6eb1235af1fa.slice/crio-6ca0df542599bc3615c75f3d40d051ff4c112be70240688e2a519f92f9713ff2 WatchSource:0}: Error finding container 6ca0df542599bc3615c75f3d40d051ff4c112be70240688e2a519f92f9713ff2: Status 404 returned error can't find the container with id 6ca0df542599bc3615c75f3d40d051ff4c112be70240688e2a519f92f9713ff2
Oct 30 13:00:56.199148 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:56.199105    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:00:56.199318 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:56.199184    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:00:56.199318 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:56.199235    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:00:56.537064 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:56.537024    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x" event={"ID":"ab79901f-7e31-4e95-a033-6eb1235af1fa","Type":"ContainerStarted","Data":"cbd936f4a22fdec1ef4f436357f28313e0368980388cb7d4e7b9115c2ea239ac"}
Oct 30 13:00:56.537316 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:56.537088    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x" event={"ID":"ab79901f-7e31-4e95-a033-6eb1235af1fa","Type":"ContainerStarted","Data":"6ca0df542599bc3615c75f3d40d051ff4c112be70240688e2a519f92f9713ff2"}
Oct 30 13:00:56.539403 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:56.539360    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3838e9d-pdzg7" event={"ID":"76d4505d-2bd3-491d-99c2-e4e3e6942327","Type":"ContainerDied","Data":"ff0ee3c6c7c46313da6232dfcad1dadd0117639503bd1b8ace08b30c98ef41a0"}
Oct 30 13:00:56.539547 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:56.539429    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff0ee3c6c7c46313da6232dfcad1dadd0117639503bd1b8ace08b30c98ef41a0"
Oct 30 13:00:56.539547 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:56.539522    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3838e9d-pdzg7"
Oct 30 13:00:56.587699 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:56.587624    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x" podStartSLOduration=2.587601132 podStartE2EDuration="2.587601132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:56.584589849 +0000 UTC m=+951.631723802" watchObservedRunningTime="2026-10-30 13:00:56.587601132 +0000 UTC m=+951.634735088"
Oct 30 13:00:56.859266 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:56.859199    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:00:57.861626 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:57.861591    4544 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 13:00:57.862044 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:57.861650    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x" podUID="ab79901f-7e31-4e95-a033-6eb1235af1fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 13:00:58.280436 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:58.280406    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:00:58.860037 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:58.859999    4544 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 13:00:58.860204 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:58.860054    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x" podUID="ab79901f-7e31-4e95-a033-6eb1235af1fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 13:00:59.008666 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:00:59.008624    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:00:59.859813 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:59.859770    4544 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 13:00:59.860006 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:00:59.859834    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x" podUID="ab79901f-7e31-4e95-a033-6eb1235af1fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 13:01:00.438793 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:00.438749    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7cac70f-v85z2"]
Oct 30 13:01:00.439758 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:00.439730    4544 cpu_manager.go:414] "containerMap: removing container" podUID="76d4505d-2bd3-491d-99c2-e4e3e6942327" containerName="revoke-keys"
Oct 30 13:01:00.439880 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:00.439761    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76d4505d-2bd3-491d-99c2-e4e3e6942327" containerName="revoke-keys"
Oct 30 13:01:00.439986 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:00.439970    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="76d4505d-2bd3-491d-99c2-e4e3e6942327" containerName="revoke-keys"
Oct 30 13:01:00.445295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:00.445269    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7cac70f-v85z2"
Oct 30 13:01:00.460739 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:00.460713    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7cac70f-v85z2"]
Oct 30 13:01:00.466939 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:00.466900    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:00.469260 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:00.468235    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:01:00.469260 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:00.468272    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:01:00.510144 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:00.510069    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184b9e84-a59d-49f2-8e76-0af679c4f148-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7cac70f-v85z2\" (UID: \"184b9e84-a59d-49f2-8e76-0af679c4f148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7cac70f-v85z2"
Oct 30 13:01:00.510773 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:00.510750    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184b9e84-a59d-49f2-8e76-0af679c4f148-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7cac70f-v85z2\" (UID: \"184b9e84-a59d-49f2-8e76-0af679c4f148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7cac70f-v85z2"
Oct 30 13:01:00.510991 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:00.510978    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh69w\" (UniqueName: \"kubernetes.io/projected/184b9e84-a59d-49f2-8e76-0af679c4f148-kube-api-access-bh69w\") pod \"maas-api-revoke-sub-f7cac70f-v85z2\" (UID: \"184b9e84-a59d-49f2-8e76-0af679c4f148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7cac70f-v85z2"
Oct 30 13:01:00.636248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:00.612652    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184b9e84-a59d-49f2-8e76-0af679c4f148-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7cac70f-v85z2\" (UID: \"184b9e84-a59d-49f2-8e76-0af679c4f148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7cac70f-v85z2"
Oct 30 13:01:00.638732 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:00.638579    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184b9e84-a59d-49f2-8e76-0af679c4f148-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7cac70f-v85z2\" (UID: \"184b9e84-a59d-49f2-8e76-0af679c4f148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7cac70f-v85z2"
Oct 30 13:01:00.638732 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:00.638660    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh69w\" (UniqueName: \"kubernetes.io/projected/184b9e84-a59d-49f2-8e76-0af679c4f148-kube-api-access-bh69w\") pod \"maas-api-revoke-sub-f7cac70f-v85z2\" (UID: \"184b9e84-a59d-49f2-8e76-0af679c4f148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7cac70f-v85z2"
Oct 30 13:01:00.639534 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:00.639471    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184b9e84-a59d-49f2-8e76-0af679c4f148-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7cac70f-v85z2\" (UID: \"184b9e84-a59d-49f2-8e76-0af679c4f148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7cac70f-v85z2"
Oct 30 13:01:00.640588 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:00.640562    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184b9e84-a59d-49f2-8e76-0af679c4f148-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7cac70f-v85z2\" (UID: \"184b9e84-a59d-49f2-8e76-0af679c4f148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7cac70f-v85z2"
Oct 30 13:01:00.662561 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:00.662530    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh69w\" (UniqueName: \"kubernetes.io/projected/184b9e84-a59d-49f2-8e76-0af679c4f148-kube-api-access-bh69w\") pod \"maas-api-revoke-sub-f7cac70f-v85z2\" (UID: \"184b9e84-a59d-49f2-8e76-0af679c4f148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7cac70f-v85z2"
Oct 30 13:01:00.760351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:00.760249    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7cac70f-v85z2"
Oct 30 13:01:00.859126 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:00.859084    4544 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 13:01:00.859440 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:00.859411    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x" podUID="ab79901f-7e31-4e95-a033-6eb1235af1fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 13:01:01.098368 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:01.098330    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7cac70f-v85z2"]
Oct 30 13:01:01.101609 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:01:01.101578    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod184b9e84_a59d_49f2_8e76_0af679c4f148.slice/crio-1bb5ede02c4d9beda4eb0dc16a663c1c857d2e81e66923c37edad53606c4ff69 WatchSource:0}: Error finding container 1bb5ede02c4d9beda4eb0dc16a663c1c857d2e81e66923c37edad53606c4ff69: Status 404 returned error can't find the container with id 1bb5ede02c4d9beda4eb0dc16a663c1c857d2e81e66923c37edad53606c4ff69
Oct 30 13:01:01.460160 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:01.460115    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6347ffc7-tzb98"]
Oct 30 13:01:01.465698 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:01.465668    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6347ffc7-tzb98"
Oct 30 13:01:01.485702 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:01.485672    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6347ffc7-tzb98"]
Oct 30 13:01:01.553837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:01.553785    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a0441eb-217b-49ff-a9f5-33c5372c0442-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6347ffc7-tzb98\" (UID: \"2a0441eb-217b-49ff-a9f5-33c5372c0442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6347ffc7-tzb98"
Oct 30 13:01:01.554108 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:01.554087    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z59m\" (UniqueName: \"kubernetes.io/projected/2a0441eb-217b-49ff-a9f5-33c5372c0442-kube-api-access-7z59m\") pod \"maas-api-revoke-sub-6347ffc7-tzb98\" (UID: \"2a0441eb-217b-49ff-a9f5-33c5372c0442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6347ffc7-tzb98"
Oct 30 13:01:01.554186 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:01.554125    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a0441eb-217b-49ff-a9f5-33c5372c0442-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6347ffc7-tzb98\" (UID: \"2a0441eb-217b-49ff-a9f5-33c5372c0442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6347ffc7-tzb98"
Oct 30 13:01:01.600308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:01.600273    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7cac70f-v85z2" event={"ID":"184b9e84-a59d-49f2-8e76-0af679c4f148","Type":"ContainerStarted","Data":"474cbafe2ed3cb7d2eb894b353d88336b2321c3971603625df5a9751e916623f"}
Oct 30 13:01:01.600494 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:01.600325    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7cac70f-v85z2" event={"ID":"184b9e84-a59d-49f2-8e76-0af679c4f148","Type":"ContainerStarted","Data":"1bb5ede02c4d9beda4eb0dc16a663c1c857d2e81e66923c37edad53606c4ff69"}
Oct 30 13:01:01.656008 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:01.655975    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z59m\" (UniqueName: \"kubernetes.io/projected/2a0441eb-217b-49ff-a9f5-33c5372c0442-kube-api-access-7z59m\") pod \"maas-api-revoke-sub-6347ffc7-tzb98\" (UID: \"2a0441eb-217b-49ff-a9f5-33c5372c0442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6347ffc7-tzb98"
Oct 30 13:01:01.656175 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:01.656014    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a0441eb-217b-49ff-a9f5-33c5372c0442-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6347ffc7-tzb98\" (UID: \"2a0441eb-217b-49ff-a9f5-33c5372c0442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6347ffc7-tzb98"
Oct 30 13:01:01.656175 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:01.656108    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a0441eb-217b-49ff-a9f5-33c5372c0442-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6347ffc7-tzb98\" (UID: \"2a0441eb-217b-49ff-a9f5-33c5372c0442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6347ffc7-tzb98"
Oct 30 13:01:01.657067 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:01.657046    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a0441eb-217b-49ff-a9f5-33c5372c0442-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6347ffc7-tzb98\" (UID: \"2a0441eb-217b-49ff-a9f5-33c5372c0442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6347ffc7-tzb98"
Oct 30 13:01:01.658604 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:01.658581    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a0441eb-217b-49ff-a9f5-33c5372c0442-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6347ffc7-tzb98\" (UID: \"2a0441eb-217b-49ff-a9f5-33c5372c0442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6347ffc7-tzb98"
Oct 30 13:01:01.663372 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:01.663326    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7cac70f-v85z2" podStartSLOduration=1.663312339 podStartE2EDuration="1.663312339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:01.66266346 +0000 UTC m=+956.709797415" watchObservedRunningTime="2026-10-30 13:01:01.663312339 +0000 UTC m=+956.710446341"
Oct 30 13:01:01.679634 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:01.679604    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z59m\" (UniqueName: \"kubernetes.io/projected/2a0441eb-217b-49ff-a9f5-33c5372c0442-kube-api-access-7z59m\") pod \"maas-api-revoke-sub-6347ffc7-tzb98\" (UID: \"2a0441eb-217b-49ff-a9f5-33c5372c0442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6347ffc7-tzb98"
Oct 30 13:01:01.779896 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:01.779804    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6347ffc7-tzb98"
Oct 30 13:01:01.860930 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:01.860897    4544 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 13:01:01.861052 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:01.860955    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x" podUID="ab79901f-7e31-4e95-a033-6eb1235af1fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 13:01:02.068195 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:02.068110    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-332f4078-zltrf"]
Oct 30 13:01:02.073427 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:02.073397    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-332f4078-zltrf"
Oct 30 13:01:02.098486 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:02.098441    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-332f4078-zltrf"]
Oct 30 13:01:02.143986 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:02.141384    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6347ffc7-tzb98"]
Oct 30 13:01:02.166976 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:02.166896    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bca9a20-c06b-4047-bfa1-accd6355ae87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-332f4078-zltrf\" (UID: \"0bca9a20-c06b-4047-bfa1-accd6355ae87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-332f4078-zltrf"
Oct 30 13:01:02.167114 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:02.167010    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6cdc\" (UniqueName: \"kubernetes.io/projected/0bca9a20-c06b-4047-bfa1-accd6355ae87-kube-api-access-l6cdc\") pod \"maas-api-revoke-sub-332f4078-zltrf\" (UID: \"0bca9a20-c06b-4047-bfa1-accd6355ae87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-332f4078-zltrf"
Oct 30 13:01:02.167114 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:02.167109    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bca9a20-c06b-4047-bfa1-accd6355ae87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-332f4078-zltrf\" (UID: \"0bca9a20-c06b-4047-bfa1-accd6355ae87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-332f4078-zltrf"
Oct 30 13:01:02.268262 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:02.268206    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bca9a20-c06b-4047-bfa1-accd6355ae87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-332f4078-zltrf\" (UID: \"0bca9a20-c06b-4047-bfa1-accd6355ae87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-332f4078-zltrf"
Oct 30 13:01:02.268446 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:02.268405    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bca9a20-c06b-4047-bfa1-accd6355ae87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-332f4078-zltrf\" (UID: \"0bca9a20-c06b-4047-bfa1-accd6355ae87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-332f4078-zltrf"
Oct 30 13:01:02.268446 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:02.268436    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6cdc\" (UniqueName: \"kubernetes.io/projected/0bca9a20-c06b-4047-bfa1-accd6355ae87-kube-api-access-l6cdc\") pod \"maas-api-revoke-sub-332f4078-zltrf\" (UID: \"0bca9a20-c06b-4047-bfa1-accd6355ae87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-332f4078-zltrf"
Oct 30 13:01:02.269088 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:02.269061    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bca9a20-c06b-4047-bfa1-accd6355ae87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-332f4078-zltrf\" (UID: \"0bca9a20-c06b-4047-bfa1-accd6355ae87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-332f4078-zltrf"
Oct 30 13:01:02.270848 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:02.270827    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bca9a20-c06b-4047-bfa1-accd6355ae87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-332f4078-zltrf\" (UID: \"0bca9a20-c06b-4047-bfa1-accd6355ae87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-332f4078-zltrf"
Oct 30 13:01:02.292030 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:02.291994    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6cdc\" (UniqueName: \"kubernetes.io/projected/0bca9a20-c06b-4047-bfa1-accd6355ae87-kube-api-access-l6cdc\") pod \"maas-api-revoke-sub-332f4078-zltrf\" (UID: \"0bca9a20-c06b-4047-bfa1-accd6355ae87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-332f4078-zltrf"
Oct 30 13:01:02.388159 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:02.388108    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:02.390012 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:02.389988    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-332f4078-zltrf"
Oct 30 13:01:02.610047 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:02.609950    4544 generic.go:356] "Generic (PLEG): container finished" podID="184b9e84-a59d-49f2-8e76-0af679c4f148" containerID="474cbafe2ed3cb7d2eb894b353d88336b2321c3971603625df5a9751e916623f" exitCode=0
Oct 30 13:01:02.610047 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:02.610012    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7cac70f-v85z2" event={"ID":"184b9e84-a59d-49f2-8e76-0af679c4f148","Type":"ContainerDied","Data":"474cbafe2ed3cb7d2eb894b353d88336b2321c3971603625df5a9751e916623f"}
Oct 30 13:01:02.612129 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:02.612096    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6347ffc7-tzb98" event={"ID":"2a0441eb-217b-49ff-a9f5-33c5372c0442","Type":"ContainerStarted","Data":"d0561c8547ebad47d60986824a6c7087f70156fd49918053202e85120f5b55dc"}
Oct 30 13:01:02.612301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:02.612156    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6347ffc7-tzb98" event={"ID":"2a0441eb-217b-49ff-a9f5-33c5372c0442","Type":"ContainerStarted","Data":"947065778c7f31e6b80a994c354b96ea4fc82367b70e8847fd0e0407f10c18f4"}
Oct 30 13:01:02.692025 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:02.691969    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6347ffc7-tzb98" podStartSLOduration=1.691952127 podStartE2EDuration="1.691952127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:02.68918569 +0000 UTC m=+957.736319643" watchObservedRunningTime="2026-10-30 13:01:02.691952127 +0000 UTC m=+957.739086080"
Oct 30 13:01:02.765509 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:02.765373    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-332f4078-zltrf"]
Oct 30 13:01:02.768689 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:01:02.768655    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bca9a20_c06b_4047_bfa1_accd6355ae87.slice/crio-0c11babf124b727769082e8fd1d1ef9885f70ee8541905ca55ceb0bb797ac5c1 WatchSource:0}: Error finding container 0c11babf124b727769082e8fd1d1ef9885f70ee8541905ca55ceb0bb797ac5c1: Status 404 returned error can't find the container with id 0c11babf124b727769082e8fd1d1ef9885f70ee8541905ca55ceb0bb797ac5c1
Oct 30 13:01:02.859957 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:02.859907    4544 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 13:01:02.860183 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:02.859973    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x" podUID="ab79901f-7e31-4e95-a033-6eb1235af1fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 13:01:03.620628 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:03.620585    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-332f4078-zltrf" event={"ID":"0bca9a20-c06b-4047-bfa1-accd6355ae87","Type":"ContainerStarted","Data":"d1aaef764be4e317b873b1e5171ba6c5d048f1048ecb8e54f013dc601cb926e4"}
Oct 30 13:01:03.621114 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:03.620653    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-332f4078-zltrf" event={"ID":"0bca9a20-c06b-4047-bfa1-accd6355ae87","Type":"ContainerStarted","Data":"0c11babf124b727769082e8fd1d1ef9885f70ee8541905ca55ceb0bb797ac5c1"}
Oct 30 13:01:03.622315 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:03.622290    4544 generic.go:356] "Generic (PLEG): container finished" podID="2a0441eb-217b-49ff-a9f5-33c5372c0442" containerID="d0561c8547ebad47d60986824a6c7087f70156fd49918053202e85120f5b55dc" exitCode=0
Oct 30 13:01:03.622450 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:03.622371    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6347ffc7-tzb98" event={"ID":"2a0441eb-217b-49ff-a9f5-33c5372c0442","Type":"ContainerDied","Data":"d0561c8547ebad47d60986824a6c7087f70156fd49918053202e85120f5b55dc"}
Oct 30 13:01:03.664154 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:03.664078    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-332f4078-zltrf" podStartSLOduration=1.664058137 podStartE2EDuration="1.664058137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:03.663634814 +0000 UTC m=+958.710768768" watchObservedRunningTime="2026-10-30 13:01:03.664058137 +0000 UTC m=+958.711192191"
Oct 30 13:01:03.778692 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:03.778659    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7cac70f-v85z2"
Oct 30 13:01:03.864407 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:03.864373    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:01:03.864863 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:03.864835    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:01:03.865974 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:03.865951    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:01:03.896827 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:03.896747    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/184b9e84-a59d-49f2-8e76-0af679c4f148-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184b9e84-a59d-49f2-8e76-0af679c4f148-maas-api-xks-ca\") pod \"184b9e84-a59d-49f2-8e76-0af679c4f148\" (UID: \"184b9e84-a59d-49f2-8e76-0af679c4f148\") "
Oct 30 13:01:03.896827 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:03.896790    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/184b9e84-a59d-49f2-8e76-0af679c4f148-kube-api-access-bh69w\" (UniqueName: \"kubernetes.io/projected/184b9e84-a59d-49f2-8e76-0af679c4f148-kube-api-access-bh69w\") pod \"184b9e84-a59d-49f2-8e76-0af679c4f148\" (UID: \"184b9e84-a59d-49f2-8e76-0af679c4f148\") "
Oct 30 13:01:03.897002 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:03.896951    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/184b9e84-a59d-49f2-8e76-0af679c4f148-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184b9e84-a59d-49f2-8e76-0af679c4f148-maas-api-openshift-ca\") pod \"184b9e84-a59d-49f2-8e76-0af679c4f148\" (UID: \"184b9e84-a59d-49f2-8e76-0af679c4f148\") "
Oct 30 13:01:03.897328 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:03.897298    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/184b9e84-a59d-49f2-8e76-0af679c4f148-maas-api-openshift-ca" pod "184b9e84-a59d-49f2-8e76-0af679c4f148" (UID: "184b9e84-a59d-49f2-8e76-0af679c4f148"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:01:03.897446 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:03.897428    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184b9e84-a59d-49f2-8e76-0af679c4f148-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:01:03.898896 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:03.898868    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/184b9e84-a59d-49f2-8e76-0af679c4f148-maas-api-xks-ca" pod "184b9e84-a59d-49f2-8e76-0af679c4f148" (UID: "184b9e84-a59d-49f2-8e76-0af679c4f148"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:01:03.899021 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:03.898997    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/184b9e84-a59d-49f2-8e76-0af679c4f148-kube-api-access-bh69w" pod "184b9e84-a59d-49f2-8e76-0af679c4f148" (UID: "184b9e84-a59d-49f2-8e76-0af679c4f148"). InnerVolumeSpecName "kube-api-access-bh69w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:03.999198 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:03.999165    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184b9e84-a59d-49f2-8e76-0af679c4f148-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:01:03.999441 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:03.999359    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh69w\" (UniqueName: \"kubernetes.io/projected/184b9e84-a59d-49f2-8e76-0af679c4f148-kube-api-access-bh69w\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:01:04.179840 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:04.179750    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:04.631471 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:04.631446    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7cac70f-v85z2"
Oct 30 13:01:04.631968 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:04.631471    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7cac70f-v85z2" event={"ID":"184b9e84-a59d-49f2-8e76-0af679c4f148","Type":"ContainerDied","Data":"1bb5ede02c4d9beda4eb0dc16a663c1c857d2e81e66923c37edad53606c4ff69"}
Oct 30 13:01:04.631968 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:04.631518    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bb5ede02c4d9beda4eb0dc16a663c1c857d2e81e66923c37edad53606c4ff69"
Oct 30 13:01:04.633261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:04.633232    4544 generic.go:356] "Generic (PLEG): container finished" podID="0bca9a20-c06b-4047-bfa1-accd6355ae87" containerID="d1aaef764be4e317b873b1e5171ba6c5d048f1048ecb8e54f013dc601cb926e4" exitCode=0
Oct 30 13:01:04.633421 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:04.633310    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-332f4078-zltrf" event={"ID":"0bca9a20-c06b-4047-bfa1-accd6355ae87","Type":"ContainerDied","Data":"d1aaef764be4e317b873b1e5171ba6c5d048f1048ecb8e54f013dc601cb926e4"}
Oct 30 13:01:04.811443 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:04.811414    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6347ffc7-tzb98"
Oct 30 13:01:04.908448 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:04.908353    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a0441eb-217b-49ff-a9f5-33c5372c0442-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a0441eb-217b-49ff-a9f5-33c5372c0442-maas-api-xks-ca\") pod \"2a0441eb-217b-49ff-a9f5-33c5372c0442\" (UID: \"2a0441eb-217b-49ff-a9f5-33c5372c0442\") "
Oct 30 13:01:04.908607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:04.908467    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a0441eb-217b-49ff-a9f5-33c5372c0442-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a0441eb-217b-49ff-a9f5-33c5372c0442-maas-api-openshift-ca\") pod \"2a0441eb-217b-49ff-a9f5-33c5372c0442\" (UID: \"2a0441eb-217b-49ff-a9f5-33c5372c0442\") "
Oct 30 13:01:04.908684 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:04.908643    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a0441eb-217b-49ff-a9f5-33c5372c0442-kube-api-access-7z59m\" (UniqueName: \"kubernetes.io/projected/2a0441eb-217b-49ff-a9f5-33c5372c0442-kube-api-access-7z59m\") pod \"2a0441eb-217b-49ff-a9f5-33c5372c0442\" (UID: \"2a0441eb-217b-49ff-a9f5-33c5372c0442\") "
Oct 30 13:01:04.908781 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:04.908754    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a0441eb-217b-49ff-a9f5-33c5372c0442-maas-api-openshift-ca" pod "2a0441eb-217b-49ff-a9f5-33c5372c0442" (UID: "2a0441eb-217b-49ff-a9f5-33c5372c0442"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:01:04.909148 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:04.909128    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a0441eb-217b-49ff-a9f5-33c5372c0442-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:01:04.910671 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:04.910646    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a0441eb-217b-49ff-a9f5-33c5372c0442-maas-api-xks-ca" pod "2a0441eb-217b-49ff-a9f5-33c5372c0442" (UID: "2a0441eb-217b-49ff-a9f5-33c5372c0442"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:01:04.910671 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:04.910654    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a0441eb-217b-49ff-a9f5-33c5372c0442-kube-api-access-7z59m" pod "2a0441eb-217b-49ff-a9f5-33c5372c0442" (UID: "2a0441eb-217b-49ff-a9f5-33c5372c0442"). InnerVolumeSpecName "kube-api-access-7z59m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:05.009955 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:05.009914    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z59m\" (UniqueName: \"kubernetes.io/projected/2a0441eb-217b-49ff-a9f5-33c5372c0442-kube-api-access-7z59m\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:01:05.009955 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:05.009947    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a0441eb-217b-49ff-a9f5-33c5372c0442-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:01:05.641934 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:05.641890    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6347ffc7-tzb98" event={"ID":"2a0441eb-217b-49ff-a9f5-33c5372c0442","Type":"ContainerDied","Data":"947065778c7f31e6b80a994c354b96ea4fc82367b70e8847fd0e0407f10c18f4"}
Oct 30 13:01:05.642412 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:05.641945    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6347ffc7-tzb98"
Oct 30 13:01:05.642412 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:05.641976    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="947065778c7f31e6b80a994c354b96ea4fc82367b70e8847fd0e0407f10c18f4"
Oct 30 13:01:05.796790 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:05.796754    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-332f4078-zltrf"
Oct 30 13:01:05.920366 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:05.920255    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bca9a20-c06b-4047-bfa1-accd6355ae87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bca9a20-c06b-4047-bfa1-accd6355ae87-maas-api-openshift-ca\") pod \"0bca9a20-c06b-4047-bfa1-accd6355ae87\" (UID: \"0bca9a20-c06b-4047-bfa1-accd6355ae87\") "
Oct 30 13:01:05.920366 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:05.920307    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bca9a20-c06b-4047-bfa1-accd6355ae87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bca9a20-c06b-4047-bfa1-accd6355ae87-maas-api-xks-ca\") pod \"0bca9a20-c06b-4047-bfa1-accd6355ae87\" (UID: \"0bca9a20-c06b-4047-bfa1-accd6355ae87\") "
Oct 30 13:01:05.920593 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:05.920463    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bca9a20-c06b-4047-bfa1-accd6355ae87-kube-api-access-l6cdc\" (UniqueName: \"kubernetes.io/projected/0bca9a20-c06b-4047-bfa1-accd6355ae87-kube-api-access-l6cdc\") pod \"0bca9a20-c06b-4047-bfa1-accd6355ae87\" (UID: \"0bca9a20-c06b-4047-bfa1-accd6355ae87\") "
Oct 30 13:01:05.920724 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:05.920702    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bca9a20-c06b-4047-bfa1-accd6355ae87-maas-api-openshift-ca" pod "0bca9a20-c06b-4047-bfa1-accd6355ae87" (UID: "0bca9a20-c06b-4047-bfa1-accd6355ae87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:01:05.921060 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:05.921034    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bca9a20-c06b-4047-bfa1-accd6355ae87-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:01:05.922685 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:05.922650    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bca9a20-c06b-4047-bfa1-accd6355ae87-maas-api-xks-ca" pod "0bca9a20-c06b-4047-bfa1-accd6355ae87" (UID: "0bca9a20-c06b-4047-bfa1-accd6355ae87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:01:05.922783 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:05.922723    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bca9a20-c06b-4047-bfa1-accd6355ae87-kube-api-access-l6cdc" pod "0bca9a20-c06b-4047-bfa1-accd6355ae87" (UID: "0bca9a20-c06b-4047-bfa1-accd6355ae87"). InnerVolumeSpecName "kube-api-access-l6cdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:05.974955 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:05.974923    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:06.022504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:06.022468    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6cdc\" (UniqueName: \"kubernetes.io/projected/0bca9a20-c06b-4047-bfa1-accd6355ae87-kube-api-access-l6cdc\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:01:06.022504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:06.022498    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bca9a20-c06b-4047-bfa1-accd6355ae87-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:01:06.650012 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:06.649980    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-332f4078-zltrf"
Oct 30 13:01:06.650012 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:06.649998    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-332f4078-zltrf" event={"ID":"0bca9a20-c06b-4047-bfa1-accd6355ae87","Type":"ContainerDied","Data":"0c11babf124b727769082e8fd1d1ef9885f70ee8541905ca55ceb0bb797ac5c1"}
Oct 30 13:01:06.650616 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:06.650041    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c11babf124b727769082e8fd1d1ef9885f70ee8541905ca55ceb0bb797ac5c1"
Oct 30 13:01:07.662508 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:07.662453    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:07.727013 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:07.726961    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c46fc70e-wwlhp"]
Oct 30 13:01:07.728203 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:07.728168    4544 cpu_manager.go:414] "containerMap: removing container" podUID="0bca9a20-c06b-4047-bfa1-accd6355ae87" containerName="revoke-keys"
Oct 30 13:01:07.728524 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:07.728503    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bca9a20-c06b-4047-bfa1-accd6355ae87" containerName="revoke-keys"
Oct 30 13:01:07.728677 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:07.728663    4544 cpu_manager.go:414] "containerMap: removing container" podUID="184b9e84-a59d-49f2-8e76-0af679c4f148" containerName="revoke-keys"
Oct 30 13:01:07.728759 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:07.728748    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="184b9e84-a59d-49f2-8e76-0af679c4f148" containerName="revoke-keys"
Oct 30 13:01:07.728834 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:07.728825    4544 cpu_manager.go:414] "containerMap: removing container" podUID="2a0441eb-217b-49ff-a9f5-33c5372c0442" containerName="revoke-keys"
Oct 30 13:01:07.728905 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:07.728894    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a0441eb-217b-49ff-a9f5-33c5372c0442" containerName="revoke-keys"
Oct 30 13:01:07.729195 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:07.729181    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bca9a20-c06b-4047-bfa1-accd6355ae87" containerName="revoke-keys"
Oct 30 13:01:07.729340 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:07.729328    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="184b9e84-a59d-49f2-8e76-0af679c4f148" containerName="revoke-keys"
Oct 30 13:01:07.729429 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:07.729420    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a0441eb-217b-49ff-a9f5-33c5372c0442" containerName="revoke-keys"
Oct 30 13:01:07.739186 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:07.739152    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46fc70e-wwlhp"
Oct 30 13:01:07.754742 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:07.754710    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:01:07.757383 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:07.757296    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c46fc70e-wwlhp"]
Oct 30 13:01:07.770817 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:07.770782    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:01:07.848833 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:07.848779    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d750941a-ed7c-4a24-bfd1-de926f1b09e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c46fc70e-wwlhp\" (UID: \"d750941a-ed7c-4a24-bfd1-de926f1b09e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46fc70e-wwlhp"
Oct 30 13:01:07.849015 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:07.848952    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z5mz\" (UniqueName: \"kubernetes.io/projected/d750941a-ed7c-4a24-bfd1-de926f1b09e0-kube-api-access-6z5mz\") pod \"maas-api-revoke-sub-c46fc70e-wwlhp\" (UID: \"d750941a-ed7c-4a24-bfd1-de926f1b09e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46fc70e-wwlhp"
Oct 30 13:01:07.849118 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:07.849038    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d750941a-ed7c-4a24-bfd1-de926f1b09e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c46fc70e-wwlhp\" (UID: \"d750941a-ed7c-4a24-bfd1-de926f1b09e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46fc70e-wwlhp"
Oct 30 13:01:07.950829 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:07.950733    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d750941a-ed7c-4a24-bfd1-de926f1b09e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c46fc70e-wwlhp\" (UID: \"d750941a-ed7c-4a24-bfd1-de926f1b09e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46fc70e-wwlhp"
Oct 30 13:01:07.951034 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:07.950842    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z5mz\" (UniqueName: \"kubernetes.io/projected/d750941a-ed7c-4a24-bfd1-de926f1b09e0-kube-api-access-6z5mz\") pod \"maas-api-revoke-sub-c46fc70e-wwlhp\" (UID: \"d750941a-ed7c-4a24-bfd1-de926f1b09e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46fc70e-wwlhp"
Oct 30 13:01:07.951034 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:07.950888    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d750941a-ed7c-4a24-bfd1-de926f1b09e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c46fc70e-wwlhp\" (UID: \"d750941a-ed7c-4a24-bfd1-de926f1b09e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46fc70e-wwlhp"
Oct 30 13:01:07.951586 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:07.951558    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d750941a-ed7c-4a24-bfd1-de926f1b09e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c46fc70e-wwlhp\" (UID: \"d750941a-ed7c-4a24-bfd1-de926f1b09e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46fc70e-wwlhp"
Oct 30 13:01:07.953379 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:07.953347    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d750941a-ed7c-4a24-bfd1-de926f1b09e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c46fc70e-wwlhp\" (UID: \"d750941a-ed7c-4a24-bfd1-de926f1b09e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46fc70e-wwlhp"
Oct 30 13:01:07.974081 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:07.974052    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z5mz\" (UniqueName: \"kubernetes.io/projected/d750941a-ed7c-4a24-bfd1-de926f1b09e0-kube-api-access-6z5mz\") pod \"maas-api-revoke-sub-c46fc70e-wwlhp\" (UID: \"d750941a-ed7c-4a24-bfd1-de926f1b09e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46fc70e-wwlhp"
Oct 30 13:01:08.069615 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:08.069571    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46fc70e-wwlhp"
Oct 30 13:01:08.375580 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:08.375543    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c46fc70e-wwlhp"]
Oct 30 13:01:08.379040 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:01:08.379011    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd750941a_ed7c_4a24_bfd1_de926f1b09e0.slice/crio-b67fe59067de46d84a9a493fc2adc540cfc84a09727cb3a4abd2f8ba80234dd1 WatchSource:0}: Error finding container b67fe59067de46d84a9a493fc2adc540cfc84a09727cb3a4abd2f8ba80234dd1: Status 404 returned error can't find the container with id b67fe59067de46d84a9a493fc2adc540cfc84a09727cb3a4abd2f8ba80234dd1
Oct 30 13:01:08.664749 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:08.664656    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46fc70e-wwlhp" event={"ID":"d750941a-ed7c-4a24-bfd1-de926f1b09e0","Type":"ContainerStarted","Data":"b5fe2a3154b88fa34b8b96a317061d1ebb777983d29b4d718cf9b816013b09d0"}
Oct 30 13:01:08.664749 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:08.664721    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46fc70e-wwlhp" event={"ID":"d750941a-ed7c-4a24-bfd1-de926f1b09e0","Type":"ContainerStarted","Data":"b67fe59067de46d84a9a493fc2adc540cfc84a09727cb3a4abd2f8ba80234dd1"}
Oct 30 13:01:08.717108 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:08.717054    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46fc70e-wwlhp" podStartSLOduration=1.71704083 podStartE2EDuration="1.71704083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:08.705016933 +0000 UTC m=+963.752150887" watchObservedRunningTime="2026-10-30 13:01:08.71704083 +0000 UTC m=+963.764174782"
Oct 30 13:01:09.971255 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:09.962603    4544 generic.go:356] "Generic (PLEG): container finished" podID="d750941a-ed7c-4a24-bfd1-de926f1b09e0" containerID="b5fe2a3154b88fa34b8b96a317061d1ebb777983d29b4d718cf9b816013b09d0" exitCode=0
Oct 30 13:01:10.001375 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:09.977193    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46fc70e-wwlhp" event={"ID":"d750941a-ed7c-4a24-bfd1-de926f1b09e0","Type":"ContainerDied","Data":"b5fe2a3154b88fa34b8b96a317061d1ebb777983d29b4d718cf9b816013b09d0"}
Oct 30 13:01:10.230299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:10.230163    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:11.229209 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:11.229169    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46fc70e-wwlhp"
Oct 30 13:01:11.273766 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:11.273623    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d750941a-ed7c-4a24-bfd1-de926f1b09e0-kube-api-access-6z5mz\" (UniqueName: \"kubernetes.io/projected/d750941a-ed7c-4a24-bfd1-de926f1b09e0-kube-api-access-6z5mz\") pod \"d750941a-ed7c-4a24-bfd1-de926f1b09e0\" (UID: \"d750941a-ed7c-4a24-bfd1-de926f1b09e0\") "
Oct 30 13:01:11.280600 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:11.280565    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d750941a-ed7c-4a24-bfd1-de926f1b09e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d750941a-ed7c-4a24-bfd1-de926f1b09e0-maas-api-xks-ca\") pod \"d750941a-ed7c-4a24-bfd1-de926f1b09e0\" (UID: \"d750941a-ed7c-4a24-bfd1-de926f1b09e0\") "
Oct 30 13:01:11.282420 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:11.282370    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d750941a-ed7c-4a24-bfd1-de926f1b09e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d750941a-ed7c-4a24-bfd1-de926f1b09e0-maas-api-openshift-ca\") pod \"d750941a-ed7c-4a24-bfd1-de926f1b09e0\" (UID: \"d750941a-ed7c-4a24-bfd1-de926f1b09e0\") "
Oct 30 13:01:11.283067 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:11.283035    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d750941a-ed7c-4a24-bfd1-de926f1b09e0-maas-api-openshift-ca" pod "d750941a-ed7c-4a24-bfd1-de926f1b09e0" (UID: "d750941a-ed7c-4a24-bfd1-de926f1b09e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:01:11.283639 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:11.283601    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d750941a-ed7c-4a24-bfd1-de926f1b09e0-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:01:11.283835 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:11.283688    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d750941a-ed7c-4a24-bfd1-de926f1b09e0-kube-api-access-6z5mz" pod "d750941a-ed7c-4a24-bfd1-de926f1b09e0" (UID: "d750941a-ed7c-4a24-bfd1-de926f1b09e0"). InnerVolumeSpecName "kube-api-access-6z5mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:11.284091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:11.283688    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d750941a-ed7c-4a24-bfd1-de926f1b09e0-maas-api-xks-ca" pod "d750941a-ed7c-4a24-bfd1-de926f1b09e0" (UID: "d750941a-ed7c-4a24-bfd1-de926f1b09e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:01:11.385307 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:11.385272    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z5mz\" (UniqueName: \"kubernetes.io/projected/d750941a-ed7c-4a24-bfd1-de926f1b09e0-kube-api-access-6z5mz\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:01:11.385307 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:11.385302    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d750941a-ed7c-4a24-bfd1-de926f1b09e0-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:01:11.993245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:11.993204    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46fc70e-wwlhp"
Oct 30 13:01:11.993432 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:11.993261    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c46fc70e-wwlhp" event={"ID":"d750941a-ed7c-4a24-bfd1-de926f1b09e0","Type":"ContainerDied","Data":"b67fe59067de46d84a9a493fc2adc540cfc84a09727cb3a4abd2f8ba80234dd1"}
Oct 30 13:01:11.993432 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:11.993309    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b67fe59067de46d84a9a493fc2adc540cfc84a09727cb3a4abd2f8ba80234dd1"
Oct 30 13:01:13.101642 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:13.101599    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:13.841803 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:13.841762    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ec1adf9-zql4m"]
Oct 30 13:01:13.842399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:13.842380    4544 cpu_manager.go:414] "containerMap: removing container" podUID="d750941a-ed7c-4a24-bfd1-de926f1b09e0" containerName="revoke-keys"
Oct 30 13:01:13.842399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:13.842399    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d750941a-ed7c-4a24-bfd1-de926f1b09e0" containerName="revoke-keys"
Oct 30 13:01:13.842518 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:13.842475    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="d750941a-ed7c-4a24-bfd1-de926f1b09e0" containerName="revoke-keys"
Oct 30 13:01:13.846325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:13.846297    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec1adf9-zql4m"
Oct 30 13:01:13.860826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:13.860795    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:01:13.872477 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:13.872444    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:01:13.876694 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:13.876646    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ec1adf9-zql4m"]
Oct 30 13:01:13.948107 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:13.948053    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f370f314-d33a-4ae4-b2b2-498533ee66b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ec1adf9-zql4m\" (UID: \"f370f314-d33a-4ae4-b2b2-498533ee66b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec1adf9-zql4m"
Oct 30 13:01:13.948394 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:13.948158    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrfgf\" (UniqueName: \"kubernetes.io/projected/f370f314-d33a-4ae4-b2b2-498533ee66b4-kube-api-access-lrfgf\") pod \"maas-api-revoke-sub-1ec1adf9-zql4m\" (UID: \"f370f314-d33a-4ae4-b2b2-498533ee66b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec1adf9-zql4m"
Oct 30 13:01:13.948394 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:13.948299    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f370f314-d33a-4ae4-b2b2-498533ee66b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ec1adf9-zql4m\" (UID: \"f370f314-d33a-4ae4-b2b2-498533ee66b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec1adf9-zql4m"
Oct 30 13:01:14.049679 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:14.049634    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f370f314-d33a-4ae4-b2b2-498533ee66b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ec1adf9-zql4m\" (UID: \"f370f314-d33a-4ae4-b2b2-498533ee66b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec1adf9-zql4m"
Oct 30 13:01:14.049871 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:14.049714    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrfgf\" (UniqueName: \"kubernetes.io/projected/f370f314-d33a-4ae4-b2b2-498533ee66b4-kube-api-access-lrfgf\") pod \"maas-api-revoke-sub-1ec1adf9-zql4m\" (UID: \"f370f314-d33a-4ae4-b2b2-498533ee66b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec1adf9-zql4m"
Oct 30 13:01:14.049871 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:14.049738    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f370f314-d33a-4ae4-b2b2-498533ee66b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ec1adf9-zql4m\" (UID: \"f370f314-d33a-4ae4-b2b2-498533ee66b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec1adf9-zql4m"
Oct 30 13:01:14.050477 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:14.050451    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f370f314-d33a-4ae4-b2b2-498533ee66b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ec1adf9-zql4m\" (UID: \"f370f314-d33a-4ae4-b2b2-498533ee66b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec1adf9-zql4m"
Oct 30 13:01:14.052083 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:14.052063    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f370f314-d33a-4ae4-b2b2-498533ee66b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ec1adf9-zql4m\" (UID: \"f370f314-d33a-4ae4-b2b2-498533ee66b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec1adf9-zql4m"
Oct 30 13:01:14.073254 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:14.073191    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrfgf\" (UniqueName: \"kubernetes.io/projected/f370f314-d33a-4ae4-b2b2-498533ee66b4-kube-api-access-lrfgf\") pod \"maas-api-revoke-sub-1ec1adf9-zql4m\" (UID: \"f370f314-d33a-4ae4-b2b2-498533ee66b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec1adf9-zql4m"
Oct 30 13:01:14.161088 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:14.160993    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec1adf9-zql4m"
Oct 30 13:01:14.470057 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:14.469971    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ec1adf9-zql4m"]
Oct 30 13:01:14.474032 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:01:14.474004    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf370f314_d33a_4ae4_b2b2_498533ee66b4.slice/crio-80dbf4f19f4e9e09062c944d0207cf6f229230608aa5c37231006160c04a465c WatchSource:0}: Error finding container 80dbf4f19f4e9e09062c944d0207cf6f229230608aa5c37231006160c04a465c: Status 404 returned error can't find the container with id 80dbf4f19f4e9e09062c944d0207cf6f229230608aa5c37231006160c04a465c
Oct 30 13:01:14.880725 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:14.880690    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:15.016136 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:15.016095    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec1adf9-zql4m" event={"ID":"f370f314-d33a-4ae4-b2b2-498533ee66b4","Type":"ContainerStarted","Data":"4bfcf27bf6e968cd62b1f148ad88b712ce418ed82499a5ece8489faa9079eec5"}
Oct 30 13:01:15.016329 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:15.016163    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec1adf9-zql4m" event={"ID":"f370f314-d33a-4ae4-b2b2-498533ee66b4","Type":"ContainerStarted","Data":"80dbf4f19f4e9e09062c944d0207cf6f229230608aa5c37231006160c04a465c"}
Oct 30 13:01:15.059858 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:15.059800    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec1adf9-zql4m" podStartSLOduration=2.059782398 podStartE2EDuration="2.059782398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:15.056424583 +0000 UTC m=+970.103558539" watchObservedRunningTime="2026-10-30 13:01:15.059782398 +0000 UTC m=+970.106916351"
Oct 30 13:01:15.366865 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:15.366826    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:16.030998 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:16.030952    4544 generic.go:356] "Generic (PLEG): container finished" podID="f370f314-d33a-4ae4-b2b2-498533ee66b4" containerID="4bfcf27bf6e968cd62b1f148ad88b712ce418ed82499a5ece8489faa9079eec5" exitCode=0
Oct 30 13:01:16.031313 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:16.031265    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec1adf9-zql4m" event={"ID":"f370f314-d33a-4ae4-b2b2-498533ee66b4","Type":"ContainerDied","Data":"4bfcf27bf6e968cd62b1f148ad88b712ce418ed82499a5ece8489faa9079eec5"}
Oct 30 13:01:16.984679 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:16.984641    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:17.239150 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:17.239080    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec1adf9-zql4m"
Oct 30 13:01:17.389401 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:17.389366    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f370f314-d33a-4ae4-b2b2-498533ee66b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f370f314-d33a-4ae4-b2b2-498533ee66b4-maas-api-openshift-ca\") pod \"f370f314-d33a-4ae4-b2b2-498533ee66b4\" (UID: \"f370f314-d33a-4ae4-b2b2-498533ee66b4\") "
Oct 30 13:01:17.389630 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:17.389436    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f370f314-d33a-4ae4-b2b2-498533ee66b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f370f314-d33a-4ae4-b2b2-498533ee66b4-maas-api-xks-ca\") pod \"f370f314-d33a-4ae4-b2b2-498533ee66b4\" (UID: \"f370f314-d33a-4ae4-b2b2-498533ee66b4\") "
Oct 30 13:01:17.389630 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:17.389573    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f370f314-d33a-4ae4-b2b2-498533ee66b4-kube-api-access-lrfgf\" (UniqueName: \"kubernetes.io/projected/f370f314-d33a-4ae4-b2b2-498533ee66b4-kube-api-access-lrfgf\") pod \"f370f314-d33a-4ae4-b2b2-498533ee66b4\" (UID: \"f370f314-d33a-4ae4-b2b2-498533ee66b4\") "
Oct 30 13:01:17.390171 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:17.390134    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f370f314-d33a-4ae4-b2b2-498533ee66b4-maas-api-openshift-ca" pod "f370f314-d33a-4ae4-b2b2-498533ee66b4" (UID: "f370f314-d33a-4ae4-b2b2-498533ee66b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:01:17.392053 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:17.392027    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f370f314-d33a-4ae4-b2b2-498533ee66b4-maas-api-xks-ca" pod "f370f314-d33a-4ae4-b2b2-498533ee66b4" (UID: "f370f314-d33a-4ae4-b2b2-498533ee66b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:01:17.392160 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:17.392135    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f370f314-d33a-4ae4-b2b2-498533ee66b4-kube-api-access-lrfgf" pod "f370f314-d33a-4ae4-b2b2-498533ee66b4" (UID: "f370f314-d33a-4ae4-b2b2-498533ee66b4"). InnerVolumeSpecName "kube-api-access-lrfgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:17.491310 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:17.491173    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrfgf\" (UniqueName: \"kubernetes.io/projected/f370f314-d33a-4ae4-b2b2-498533ee66b4-kube-api-access-lrfgf\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:01:17.491310 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:17.491211    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f370f314-d33a-4ae4-b2b2-498533ee66b4-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:01:17.491310 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:17.491242    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f370f314-d33a-4ae4-b2b2-498533ee66b4-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:01:18.069622 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:18.069592    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec1adf9-zql4m"
Oct 30 13:01:18.070040 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:18.069623    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ec1adf9-zql4m" event={"ID":"f370f314-d33a-4ae4-b2b2-498533ee66b4","Type":"ContainerDied","Data":"80dbf4f19f4e9e09062c944d0207cf6f229230608aa5c37231006160c04a465c"}
Oct 30 13:01:18.070040 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:18.069672    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80dbf4f19f4e9e09062c944d0207cf6f229230608aa5c37231006160c04a465c"
Oct 30 13:01:18.786294 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:18.786259    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:20.768808 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:20.768773    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:22.608108 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:22.608063    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:24.200015 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:24.199976    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55463289-2mj2g"]
Oct 30 13:01:24.200592 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:24.200577    4544 cpu_manager.go:414] "containerMap: removing container" podUID="f370f314-d33a-4ae4-b2b2-498533ee66b4" containerName="revoke-keys"
Oct 30 13:01:24.200639 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:24.200593    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f370f314-d33a-4ae4-b2b2-498533ee66b4" containerName="revoke-keys"
Oct 30 13:01:24.200698 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:24.200689    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="f370f314-d33a-4ae4-b2b2-498533ee66b4" containerName="revoke-keys"
Oct 30 13:01:24.204545 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:24.204517    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55463289-2mj2g"
Oct 30 13:01:24.231338 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:24.231299    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:01:24.237662 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:24.237627    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55463289-2mj2g"]
Oct 30 13:01:24.237860 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:24.237785    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:01:24.286439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:24.286400    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfxqm\" (UniqueName: \"kubernetes.io/projected/3d1d21c0-897a-4e80-9fd2-b774bda9b260-kube-api-access-mfxqm\") pod \"maas-api-revoke-sub-55463289-2mj2g\" (UID: \"3d1d21c0-897a-4e80-9fd2-b774bda9b260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55463289-2mj2g"
Oct 30 13:01:24.286439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:24.286449    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d1d21c0-897a-4e80-9fd2-b774bda9b260-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55463289-2mj2g\" (UID: \"3d1d21c0-897a-4e80-9fd2-b774bda9b260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55463289-2mj2g"
Oct 30 13:01:24.286659 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:24.286481    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d1d21c0-897a-4e80-9fd2-b774bda9b260-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55463289-2mj2g\" (UID: \"3d1d21c0-897a-4e80-9fd2-b774bda9b260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55463289-2mj2g"
Oct 30 13:01:24.387977 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:24.387936    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfxqm\" (UniqueName: \"kubernetes.io/projected/3d1d21c0-897a-4e80-9fd2-b774bda9b260-kube-api-access-mfxqm\") pod \"maas-api-revoke-sub-55463289-2mj2g\" (UID: \"3d1d21c0-897a-4e80-9fd2-b774bda9b260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55463289-2mj2g"
Oct 30 13:01:24.387977 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:24.387983    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d1d21c0-897a-4e80-9fd2-b774bda9b260-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55463289-2mj2g\" (UID: \"3d1d21c0-897a-4e80-9fd2-b774bda9b260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55463289-2mj2g"
Oct 30 13:01:24.388198 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:24.388010    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d1d21c0-897a-4e80-9fd2-b774bda9b260-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55463289-2mj2g\" (UID: \"3d1d21c0-897a-4e80-9fd2-b774bda9b260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55463289-2mj2g"
Oct 30 13:01:24.388696 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:24.388673    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d1d21c0-897a-4e80-9fd2-b774bda9b260-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55463289-2mj2g\" (UID: \"3d1d21c0-897a-4e80-9fd2-b774bda9b260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55463289-2mj2g"
Oct 30 13:01:24.390602 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:24.390579    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d1d21c0-897a-4e80-9fd2-b774bda9b260-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55463289-2mj2g\" (UID: \"3d1d21c0-897a-4e80-9fd2-b774bda9b260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55463289-2mj2g"
Oct 30 13:01:24.413360 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:24.413326    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfxqm\" (UniqueName: \"kubernetes.io/projected/3d1d21c0-897a-4e80-9fd2-b774bda9b260-kube-api-access-mfxqm\") pod \"maas-api-revoke-sub-55463289-2mj2g\" (UID: \"3d1d21c0-897a-4e80-9fd2-b774bda9b260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55463289-2mj2g"
Oct 30 13:01:24.518918 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:24.518824    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55463289-2mj2g"
Oct 30 13:01:24.746243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:24.744610    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:24.952710 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:24.952667    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55463289-2mj2g"]
Oct 30 13:01:25.163796 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:25.163758    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55463289-2mj2g" event={"ID":"3d1d21c0-897a-4e80-9fd2-b774bda9b260","Type":"ContainerStarted","Data":"c1969050b306d3be44bec8a4916da1cd9b2e371530fd43665579ba60b0391c4e"}
Oct 30 13:01:25.329635 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:25.329576    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"]
Oct 30 13:01:25.339732 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:25.339692    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:01:25.366835 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:25.366768    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"]
Oct 30 13:01:25.471878 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:25.471842    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-istio-data\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g\" (UID: \"75a99162-5892-4899-8487-071967579551\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:01:25.472296 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:25.472267    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-credential-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g\" (UID: \"75a99162-5892-4899-8487-071967579551\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:01:25.472503 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:25.472485    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75a99162-5892-4899-8487-071967579551-istio-token\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g\" (UID: \"75a99162-5892-4899-8487-071967579551\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:01:25.472665 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:25.472651    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-workload-certs\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g\" (UID: \"75a99162-5892-4899-8487-071967579551\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:01:25.472791 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:25.472778    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a99162-5892-4899-8487-071967579551-istiod-ca-cert\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g\" (UID: \"75a99162-5892-4899-8487-071967579551\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:01:25.472928 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:25.472914    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-workload-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g\" (UID: \"75a99162-5892-4899-8487-071967579551\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:01:25.473064 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:25.473049    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-istio-envoy\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g\" (UID: \"75a99162-5892-4899-8487-071967579551\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:01:25.473170 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:25.473156    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cztw2\" (UniqueName: \"kubernetes.io/projected/75a99162-5892-4899-8487-071967579551-kube-api-access-cztw2\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g\" (UID: \"75a99162-5892-4899-8487-071967579551\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:01:25.473305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:25.473288    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a99162-5892-4899-8487-071967579551-istio-podinfo\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g\" (UID: \"75a99162-5892-4899-8487-071967579551\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:01:25.611448 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:25.585979    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-workload-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g\" (UID: \"75a99162-5892-4899-8487-071967579551\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:01:25.611448 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:25.586046    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-istio-envoy\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g\" (UID: \"75a99162-5892-4899-8487-071967579551\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:01:25.611448 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:25.586077    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cztw2\" (UniqueName: \"kubernetes.io/projected/75a99162-5892-4899-8487-071967579551-kube-api-access-cztw2\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g\" (UID: \"75a99162-5892-4899-8487-071967579551\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:01:25.611448 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:25.586113    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a99162-5892-4899-8487-071967579551-istio-podinfo\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g\" (UID: \"75a99162-5892-4899-8487-071967579551\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:01:25.611448 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:25.586193    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-istio-data\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g\" (UID: \"75a99162-5892-4899-8487-071967579551\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:01:25.611448 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:25.586292    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-credential-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g\" (UID: \"75a99162-5892-4899-8487-071967579551\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:01:25.611448 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:25.586365    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75a99162-5892-4899-8487-071967579551-istio-token\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g\" (UID: \"75a99162-5892-4899-8487-071967579551\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:01:25.611448 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:25.586425    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-workload-certs\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g\" (UID: \"75a99162-5892-4899-8487-071967579551\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:01:25.611448 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:25.586465    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a99162-5892-4899-8487-071967579551-istiod-ca-cert\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g\" (UID: \"75a99162-5892-4899-8487-071967579551\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:01:25.611448 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:25.587313    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a99162-5892-4899-8487-071967579551-istiod-ca-cert\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g\" (UID: \"75a99162-5892-4899-8487-071967579551\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:01:25.611448 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:25.587626    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-workload-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g\" (UID: \"75a99162-5892-4899-8487-071967579551\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:01:25.628544 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:25.612594    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-istio-data\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g\" (UID: \"75a99162-5892-4899-8487-071967579551\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:01:25.628544 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:25.613915    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-credential-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g\" (UID: \"75a99162-5892-4899-8487-071967579551\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:01:25.628544 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:25.614153    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-workload-certs\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g\" (UID: \"75a99162-5892-4899-8487-071967579551\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:01:25.631441 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:25.631409    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-istio-envoy\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g\" (UID: \"75a99162-5892-4899-8487-071967579551\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:01:25.694475 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:25.694241    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75a99162-5892-4899-8487-071967579551-istio-token\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g\" (UID: \"75a99162-5892-4899-8487-071967579551\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:01:25.696886 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:25.696858    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a99162-5892-4899-8487-071967579551-istio-podinfo\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g\" (UID: \"75a99162-5892-4899-8487-071967579551\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:01:25.697857 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:25.697834    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cztw2\" (UniqueName: \"kubernetes.io/projected/75a99162-5892-4899-8487-071967579551-kube-api-access-cztw2\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g\" (UID: \"75a99162-5892-4899-8487-071967579551\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:01:25.720871 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:25.720568    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:01:26.174255 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:26.172405    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"]
Oct 30 13:01:26.236564 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:26.236033    4544 generic.go:356] "Generic (PLEG): container finished" podID="3d1d21c0-897a-4e80-9fd2-b774bda9b260" containerID="3ee9945efefdf83761fc35608def7457367fccb932cde9cbf68e666bf9b9080b" exitCode=0
Oct 30 13:01:26.236564 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:26.236105    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55463289-2mj2g" event={"ID":"3d1d21c0-897a-4e80-9fd2-b774bda9b260","Type":"ContainerDied","Data":"3ee9945efefdf83761fc35608def7457367fccb932cde9cbf68e666bf9b9080b"}
Oct 30 13:01:26.260451 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:26.260345    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:01:26.265800 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:26.265595    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:01:26.266231 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:26.266184    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:01:26.810916 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:26.810872    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:27.246789 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:27.246753    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g" event={"ID":"75a99162-5892-4899-8487-071967579551","Type":"ContainerStarted","Data":"4bed83bb47acdb31c9bb102d0c03712b21a56e3c53d8d02e8ada1cd9b229dffe"}
Oct 30 13:01:27.247013 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:27.246815    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g" event={"ID":"75a99162-5892-4899-8487-071967579551","Type":"ContainerStarted","Data":"f1f59d9690993a1a31a88e07c9584c3eec2a4658e24a40230c9e561c7b2993e5"}
Oct 30 13:01:27.305866 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:27.305787    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g" podStartSLOduration=2.305766531 podStartE2EDuration="2.305766531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:27.30280683 +0000 UTC m=+982.349940783" watchObservedRunningTime="2026-10-30 13:01:27.305766531 +0000 UTC m=+982.352900485"
Oct 30 13:01:27.445919 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:27.445883    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55463289-2mj2g"
Oct 30 13:01:27.604169 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:27.604134    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d1d21c0-897a-4e80-9fd2-b774bda9b260-kube-api-access-mfxqm\" (UniqueName: \"kubernetes.io/projected/3d1d21c0-897a-4e80-9fd2-b774bda9b260-kube-api-access-mfxqm\") pod \"3d1d21c0-897a-4e80-9fd2-b774bda9b260\" (UID: \"3d1d21c0-897a-4e80-9fd2-b774bda9b260\") "
Oct 30 13:01:27.604405 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:27.604256    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d1d21c0-897a-4e80-9fd2-b774bda9b260-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d1d21c0-897a-4e80-9fd2-b774bda9b260-maas-api-openshift-ca\") pod \"3d1d21c0-897a-4e80-9fd2-b774bda9b260\" (UID: \"3d1d21c0-897a-4e80-9fd2-b774bda9b260\") "
Oct 30 13:01:27.604405 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:27.604312    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d1d21c0-897a-4e80-9fd2-b774bda9b260-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d1d21c0-897a-4e80-9fd2-b774bda9b260-maas-api-xks-ca\") pod \"3d1d21c0-897a-4e80-9fd2-b774bda9b260\" (UID: \"3d1d21c0-897a-4e80-9fd2-b774bda9b260\") "
Oct 30 13:01:27.604693 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:27.604667    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d1d21c0-897a-4e80-9fd2-b774bda9b260-maas-api-openshift-ca" pod "3d1d21c0-897a-4e80-9fd2-b774bda9b260" (UID: "3d1d21c0-897a-4e80-9fd2-b774bda9b260"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:01:27.604857 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:27.604841    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d1d21c0-897a-4e80-9fd2-b774bda9b260-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:01:27.606399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:27.606371    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d1d21c0-897a-4e80-9fd2-b774bda9b260-maas-api-xks-ca" pod "3d1d21c0-897a-4e80-9fd2-b774bda9b260" (UID: "3d1d21c0-897a-4e80-9fd2-b774bda9b260"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:01:27.606499 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:27.606460    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d1d21c0-897a-4e80-9fd2-b774bda9b260-kube-api-access-mfxqm" pod "3d1d21c0-897a-4e80-9fd2-b774bda9b260" (UID: "3d1d21c0-897a-4e80-9fd2-b774bda9b260"). InnerVolumeSpecName "kube-api-access-mfxqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:27.706500 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:27.706462    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfxqm\" (UniqueName: \"kubernetes.io/projected/3d1d21c0-897a-4e80-9fd2-b774bda9b260-kube-api-access-mfxqm\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:01:27.706500 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:27.706496    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d1d21c0-897a-4e80-9fd2-b774bda9b260-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:01:27.737329 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:27.737293    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:01:27.738502 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:27.738470    4544 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 13:01:27.738625 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:27.738536    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g" podUID="75a99162-5892-4899-8487-071967579551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 13:01:28.254907 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:28.254872    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55463289-2mj2g"
Oct 30 13:01:28.254907 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:28.254893    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55463289-2mj2g" event={"ID":"3d1d21c0-897a-4e80-9fd2-b774bda9b260","Type":"ContainerDied","Data":"c1969050b306d3be44bec8a4916da1cd9b2e371530fd43665579ba60b0391c4e"}
Oct 30 13:01:28.255400 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:28.254966    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1969050b306d3be44bec8a4916da1cd9b2e371530fd43665579ba60b0391c4e"
Oct 30 13:01:28.731246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:28.727935    4544 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 13:01:28.731246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:28.728002    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g" podUID="75a99162-5892-4899-8487-071967579551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 13:01:28.797412 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:28.797377    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:29.010001 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:01:29.009911    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:01:29.727254 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:29.726787    4544 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 13:01:29.727254 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:29.726865    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g" podUID="75a99162-5892-4899-8487-071967579551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 13:01:30.704212 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:30.704180    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:30.731244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:30.730687    4544 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 13:01:30.731244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:30.730760    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g" podUID="75a99162-5892-4899-8487-071967579551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 13:01:31.736245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:31.735629    4544 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 13:01:31.736245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:31.735708    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g" podUID="75a99162-5892-4899-8487-071967579551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 13:01:32.690706 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:32.690674    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:32.722652 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:32.722608    4544 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 13:01:32.722821 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:32.722668    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g" podUID="75a99162-5892-4899-8487-071967579551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 13:01:33.726414 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:33.726377    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:01:33.726907 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:33.726757    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:01:33.727537 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:33.727517    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:01:34.462317 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:34.462277    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"]
Oct 30 13:01:34.463189 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:34.463164    4544 cpu_manager.go:414] "containerMap: removing container" podUID="3d1d21c0-897a-4e80-9fd2-b774bda9b260" containerName="revoke-keys"
Oct 30 13:01:34.463189 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:34.463189    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d1d21c0-897a-4e80-9fd2-b774bda9b260" containerName="revoke-keys"
Oct 30 13:01:34.463381 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:34.463367    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d1d21c0-897a-4e80-9fd2-b774bda9b260" containerName="revoke-keys"
Oct 30 13:01:34.468714 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:34.468688    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:01:34.494626 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:34.494596    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"]
Oct 30 13:01:34.501017 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:34.500989    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8b46c9\"/\"distinct-w1-8b46c9-kserve-self-signed-certs\""
Oct 30 13:01:34.501181 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:34.501146    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8b46c9\"/\"default-dockercfg-kqjkl\""
Oct 30 13:01:34.501714 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:34.501691    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-8b46c9\"/\"openshift-service-ca.crt\""
Oct 30 13:01:34.501768 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:34.501743    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-8b46c9\"/\"kube-root-ca.crt\""
Oct 30 13:01:34.504483 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:34.504456    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-tmp-dir\") pod \"distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj\" (UID: \"0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:01:34.504627 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:34.504518    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-model-cache\") pod \"distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj\" (UID: \"0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:01:34.504627 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:34.504578    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-tls-certs\") pod \"distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj\" (UID: \"0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:01:34.504627 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:34.504610    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67cc7\" (UniqueName: \"kubernetes.io/projected/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-kube-api-access-67cc7\") pod \"distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj\" (UID: \"0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:01:34.504743 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:34.504647    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-home\") pod \"distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj\" (UID: \"0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:01:34.504743 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:34.504712    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-dshm\") pod \"distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj\" (UID: \"0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:01:34.533488 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:34.533453    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:34.606258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:34.606209    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-tmp-dir\") pod \"distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj\" (UID: \"0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:01:34.606486 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:34.606300    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-model-cache\") pod \"distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj\" (UID: \"0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:01:34.606486 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:34.606332    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-tls-certs\") pod \"distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj\" (UID: \"0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:01:34.606486 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:34.606373    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67cc7\" (UniqueName: \"kubernetes.io/projected/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-kube-api-access-67cc7\") pod \"distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj\" (UID: \"0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:01:34.606486 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:34.606433    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-home\") pod \"distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj\" (UID: \"0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:01:34.606486 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:34.606468    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-dshm\") pod \"distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj\" (UID: \"0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:01:34.606701 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:34.606674    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-tmp-dir\") pod \"distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj\" (UID: \"0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:01:34.606759 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:34.606719    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-model-cache\") pod \"distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj\" (UID: \"0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:01:34.606816 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:34.606795    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-home\") pod \"distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj\" (UID: \"0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:01:34.608838 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:34.608807    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-dshm\") pod \"distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj\" (UID: \"0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:01:34.609101 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:34.609079    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-tls-certs\") pod \"distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj\" (UID: \"0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:01:34.709789 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:34.709749    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67cc7\" (UniqueName: \"kubernetes.io/projected/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-kube-api-access-67cc7\") pod \"distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj\" (UID: \"0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:01:34.722058 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:34.721940    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:34.782884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:34.782473    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:01:35.118450 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:35.118423    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"]
Oct 30 13:01:35.121340 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:01:35.121313    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d6204f2_be22_4e1a_8aa6_8cf7aab8a78a.slice/crio-c7450ad6f9b41fbeff83d41687d9a75478de24eba17476ab0271c2b2d0e7f908 WatchSource:0}: Error finding container c7450ad6f9b41fbeff83d41687d9a75478de24eba17476ab0271c2b2d0e7f908: Status 404 returned error can't find the container with id c7450ad6f9b41fbeff83d41687d9a75478de24eba17476ab0271c2b2d0e7f908
Oct 30 13:01:35.303128 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:35.303084    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" event={"ID":"0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a","Type":"ContainerStarted","Data":"b5c593ff77e5a4795dd276ffcb5056cd8706ff649fe625f0a070735eae9004b7"}
Oct 30 13:01:35.303300 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:35.303137    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" event={"ID":"0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a","Type":"ContainerStarted","Data":"c7450ad6f9b41fbeff83d41687d9a75478de24eba17476ab0271c2b2d0e7f908"}
Oct 30 13:01:35.345641 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:35.345576    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podStartSLOduration=1.345557126 podStartE2EDuration="1.345557126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:35.343810532 +0000 UTC m=+990.390944487" watchObservedRunningTime="2026-10-30 13:01:35.345557126 +0000 UTC m=+990.392691083"
Oct 30 13:01:35.784575 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:35.784539    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:01:36.294461 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:36.294424    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:36.588303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:36.588274    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:37.636422 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:37.636383    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4521c558-fjdcg"]
Oct 30 13:01:37.640734 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:37.640709    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4521c558-fjdcg"
Oct 30 13:01:37.669815 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:37.669752    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:01:37.670021 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:37.669903    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:01:37.673748 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:37.673710    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4521c558-fjdcg"]
Oct 30 13:01:37.765596 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:37.765556    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpm9d\" (UniqueName: \"kubernetes.io/projected/e4e25784-2cf6-432d-8924-e855d1c88155-kube-api-access-tpm9d\") pod \"maas-api-revoke-sub-4521c558-fjdcg\" (UID: \"e4e25784-2cf6-432d-8924-e855d1c88155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4521c558-fjdcg"
Oct 30 13:01:37.765596 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:37.765601    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4e25784-2cf6-432d-8924-e855d1c88155-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4521c558-fjdcg\" (UID: \"e4e25784-2cf6-432d-8924-e855d1c88155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4521c558-fjdcg"
Oct 30 13:01:37.765888 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:37.765767    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4e25784-2cf6-432d-8924-e855d1c88155-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4521c558-fjdcg\" (UID: \"e4e25784-2cf6-432d-8924-e855d1c88155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4521c558-fjdcg"
Oct 30 13:01:37.867031 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:37.866990    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4e25784-2cf6-432d-8924-e855d1c88155-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4521c558-fjdcg\" (UID: \"e4e25784-2cf6-432d-8924-e855d1c88155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4521c558-fjdcg"
Oct 30 13:01:37.867258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:37.867157    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpm9d\" (UniqueName: \"kubernetes.io/projected/e4e25784-2cf6-432d-8924-e855d1c88155-kube-api-access-tpm9d\") pod \"maas-api-revoke-sub-4521c558-fjdcg\" (UID: \"e4e25784-2cf6-432d-8924-e855d1c88155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4521c558-fjdcg"
Oct 30 13:01:37.867258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:37.867198    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4e25784-2cf6-432d-8924-e855d1c88155-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4521c558-fjdcg\" (UID: \"e4e25784-2cf6-432d-8924-e855d1c88155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4521c558-fjdcg"
Oct 30 13:01:37.867771 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:37.867748    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4e25784-2cf6-432d-8924-e855d1c88155-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4521c558-fjdcg\" (UID: \"e4e25784-2cf6-432d-8924-e855d1c88155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4521c558-fjdcg"
Oct 30 13:01:37.869710 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:37.869679    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4e25784-2cf6-432d-8924-e855d1c88155-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4521c558-fjdcg\" (UID: \"e4e25784-2cf6-432d-8924-e855d1c88155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4521c558-fjdcg"
Oct 30 13:01:37.890830 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:37.890761    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpm9d\" (UniqueName: \"kubernetes.io/projected/e4e25784-2cf6-432d-8924-e855d1c88155-kube-api-access-tpm9d\") pod \"maas-api-revoke-sub-4521c558-fjdcg\" (UID: \"e4e25784-2cf6-432d-8924-e855d1c88155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4521c558-fjdcg"
Oct 30 13:01:37.957435 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:37.957245    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4521c558-fjdcg"
Oct 30 13:01:38.413138 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:38.413103    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4521c558-fjdcg"]
Oct 30 13:01:38.416937 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:01:38.416901    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e25784_2cf6_432d_8924_e855d1c88155.slice/crio-2397586baadd5dfdb5090c5202c8699ed81621f72dbe6bb770827fb9795f62d9 WatchSource:0}: Error finding container 2397586baadd5dfdb5090c5202c8699ed81621f72dbe6bb770827fb9795f62d9: Status 404 returned error can't find the container with id 2397586baadd5dfdb5090c5202c8699ed81621f72dbe6bb770827fb9795f62d9
Oct 30 13:01:38.693311 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:38.693179    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:39.332095 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:39.332063    4544 generic.go:356] "Generic (PLEG): container finished" podID="e4e25784-2cf6-432d-8924-e855d1c88155" containerID="355b209fed7301cb190c5b9a492114a7e5b65075a0385d3cb6cf443047e23c97" exitCode=0
Oct 30 13:01:39.332289 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:39.332150    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4521c558-fjdcg" event={"ID":"e4e25784-2cf6-432d-8924-e855d1c88155","Type":"ContainerDied","Data":"355b209fed7301cb190c5b9a492114a7e5b65075a0385d3cb6cf443047e23c97"}
Oct 30 13:01:39.332289 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:39.332213    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4521c558-fjdcg" event={"ID":"e4e25784-2cf6-432d-8924-e855d1c88155","Type":"ContainerStarted","Data":"2397586baadd5dfdb5090c5202c8699ed81621f72dbe6bb770827fb9795f62d9"}
Oct 30 13:01:40.498101 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:40.498072    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4521c558-fjdcg"
Oct 30 13:01:40.598540 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:40.598457    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4e25784-2cf6-432d-8924-e855d1c88155-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4e25784-2cf6-432d-8924-e855d1c88155-maas-api-openshift-ca\") pod \"e4e25784-2cf6-432d-8924-e855d1c88155\" (UID: \"e4e25784-2cf6-432d-8924-e855d1c88155\") "
Oct 30 13:01:40.598540 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:40.598502    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4e25784-2cf6-432d-8924-e855d1c88155-kube-api-access-tpm9d\" (UniqueName: \"kubernetes.io/projected/e4e25784-2cf6-432d-8924-e855d1c88155-kube-api-access-tpm9d\") pod \"e4e25784-2cf6-432d-8924-e855d1c88155\" (UID: \"e4e25784-2cf6-432d-8924-e855d1c88155\") "
Oct 30 13:01:40.598737 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:40.598665    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4e25784-2cf6-432d-8924-e855d1c88155-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4e25784-2cf6-432d-8924-e855d1c88155-maas-api-xks-ca\") pod \"e4e25784-2cf6-432d-8924-e855d1c88155\" (UID: \"e4e25784-2cf6-432d-8924-e855d1c88155\") "
Oct 30 13:01:40.598949 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:40.598879    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4e25784-2cf6-432d-8924-e855d1c88155-maas-api-openshift-ca" pod "e4e25784-2cf6-432d-8924-e855d1c88155" (UID: "e4e25784-2cf6-432d-8924-e855d1c88155"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:01:40.599081 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:40.599064    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4e25784-2cf6-432d-8924-e855d1c88155-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:01:40.600671 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:40.600643    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e25784-2cf6-432d-8924-e855d1c88155-kube-api-access-tpm9d" pod "e4e25784-2cf6-432d-8924-e855d1c88155" (UID: "e4e25784-2cf6-432d-8924-e855d1c88155"). InnerVolumeSpecName "kube-api-access-tpm9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:40.600791 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:40.600705    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4e25784-2cf6-432d-8924-e855d1c88155-maas-api-xks-ca" pod "e4e25784-2cf6-432d-8924-e855d1c88155" (UID: "e4e25784-2cf6-432d-8924-e855d1c88155"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:01:40.699900 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:40.699863    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpm9d\" (UniqueName: \"kubernetes.io/projected/e4e25784-2cf6-432d-8924-e855d1c88155-kube-api-access-tpm9d\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:01:40.699900 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:40.699891    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4e25784-2cf6-432d-8924-e855d1c88155-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:01:40.711103 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:40.711075    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:41.346632 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:41.346597    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4521c558-fjdcg"
Oct 30 13:01:41.346810 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:41.346634    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4521c558-fjdcg" event={"ID":"e4e25784-2cf6-432d-8924-e855d1c88155","Type":"ContainerDied","Data":"2397586baadd5dfdb5090c5202c8699ed81621f72dbe6bb770827fb9795f62d9"}
Oct 30 13:01:41.346810 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:41.346695    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2397586baadd5dfdb5090c5202c8699ed81621f72dbe6bb770827fb9795f62d9"
Oct 30 13:01:43.157246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:43.155246    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:44.784488 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:44.784440    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:01:44.797925 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:44.797891    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:01:44.926551 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:44.926514    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:45.388801 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:45.388771    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:01:46.699491 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:46.699429    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:47.986300 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:47.986260    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:48.676254 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:48.676166    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:48.789735 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:48.789700    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:50.708485 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:50.708401    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:52.064059 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:52.064021    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fedd835-c2x48"]
Oct 30 13:01:52.064613 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:52.064599    4544 cpu_manager.go:414] "containerMap: removing container" podUID="e4e25784-2cf6-432d-8924-e855d1c88155" containerName="revoke-keys"
Oct 30 13:01:52.064661 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:52.064615    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4e25784-2cf6-432d-8924-e855d1c88155" containerName="revoke-keys"
Oct 30 13:01:52.064731 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:52.064722    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4e25784-2cf6-432d-8924-e855d1c88155" containerName="revoke-keys"
Oct 30 13:01:52.068159 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:52.068142    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fedd835-c2x48"
Oct 30 13:01:52.097408 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:52.097368    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:01:52.097408 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:52.097381    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:01:52.103316 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:52.103286    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fedd835-c2x48"]
Oct 30 13:01:52.221413 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:52.221377    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28879ce9-05ee-4ba5-8cd2-98d514e651e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fedd835-c2x48\" (UID: \"28879ce9-05ee-4ba5-8cd2-98d514e651e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fedd835-c2x48"
Oct 30 13:01:52.221594 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:52.221539    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8d48\" (UniqueName: \"kubernetes.io/projected/28879ce9-05ee-4ba5-8cd2-98d514e651e1-kube-api-access-w8d48\") pod \"maas-api-revoke-sub-4fedd835-c2x48\" (UID: \"28879ce9-05ee-4ba5-8cd2-98d514e651e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fedd835-c2x48"
Oct 30 13:01:52.221594 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:52.221573    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28879ce9-05ee-4ba5-8cd2-98d514e651e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fedd835-c2x48\" (UID: \"28879ce9-05ee-4ba5-8cd2-98d514e651e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fedd835-c2x48"
Oct 30 13:01:52.322852 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:52.322750    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8d48\" (UniqueName: \"kubernetes.io/projected/28879ce9-05ee-4ba5-8cd2-98d514e651e1-kube-api-access-w8d48\") pod \"maas-api-revoke-sub-4fedd835-c2x48\" (UID: \"28879ce9-05ee-4ba5-8cd2-98d514e651e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fedd835-c2x48"
Oct 30 13:01:52.322852 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:52.322796    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28879ce9-05ee-4ba5-8cd2-98d514e651e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fedd835-c2x48\" (UID: \"28879ce9-05ee-4ba5-8cd2-98d514e651e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fedd835-c2x48"
Oct 30 13:01:52.323109 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:52.322924    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28879ce9-05ee-4ba5-8cd2-98d514e651e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fedd835-c2x48\" (UID: \"28879ce9-05ee-4ba5-8cd2-98d514e651e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fedd835-c2x48"
Oct 30 13:01:52.323603 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:52.323573    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28879ce9-05ee-4ba5-8cd2-98d514e651e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fedd835-c2x48\" (UID: \"28879ce9-05ee-4ba5-8cd2-98d514e651e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fedd835-c2x48"
Oct 30 13:01:52.325375 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:52.325355    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28879ce9-05ee-4ba5-8cd2-98d514e651e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fedd835-c2x48\" (UID: \"28879ce9-05ee-4ba5-8cd2-98d514e651e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fedd835-c2x48"
Oct 30 13:01:52.346072 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:52.346028    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8d48\" (UniqueName: \"kubernetes.io/projected/28879ce9-05ee-4ba5-8cd2-98d514e651e1-kube-api-access-w8d48\") pod \"maas-api-revoke-sub-4fedd835-c2x48\" (UID: \"28879ce9-05ee-4ba5-8cd2-98d514e651e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fedd835-c2x48"
Oct 30 13:01:52.382682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:52.382647    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fedd835-c2x48"
Oct 30 13:01:52.765466 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:01:52.762719    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28879ce9_05ee_4ba5_8cd2_98d514e651e1.slice/crio-9423b3fd8e3ebe08fd94254530b250e39519d8936180acfe3d736864c5dd7f8d WatchSource:0}: Error finding container 9423b3fd8e3ebe08fd94254530b250e39519d8936180acfe3d736864c5dd7f8d: Status 404 returned error can't find the container with id 9423b3fd8e3ebe08fd94254530b250e39519d8936180acfe3d736864c5dd7f8d
Oct 30 13:01:52.779049 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:52.778991    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fedd835-c2x48"]
Oct 30 13:01:52.850021 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:52.849985    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:52.995091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:52.995059    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:53.442322 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:53.442283    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fedd835-c2x48" event={"ID":"28879ce9-05ee-4ba5-8cd2-98d514e651e1","Type":"ContainerStarted","Data":"5bcceaf30c078cd17436b2de6eac5be36c038837ba56b7384464c46a45330cef"}
Oct 30 13:01:53.442770 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:53.442349    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fedd835-c2x48" event={"ID":"28879ce9-05ee-4ba5-8cd2-98d514e651e1","Type":"ContainerStarted","Data":"9423b3fd8e3ebe08fd94254530b250e39519d8936180acfe3d736864c5dd7f8d"}
Oct 30 13:01:53.485892 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:53.485832    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fedd835-c2x48" podStartSLOduration=1.48580357 podStartE2EDuration="1.48580357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:53.482200674 +0000 UTC m=+1008.529334629" watchObservedRunningTime="2026-10-30 13:01:53.48580357 +0000 UTC m=+1008.532937524"
Oct 30 13:01:54.450321 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:54.450281    4544 generic.go:356] "Generic (PLEG): container finished" podID="28879ce9-05ee-4ba5-8cd2-98d514e651e1" containerID="5bcceaf30c078cd17436b2de6eac5be36c038837ba56b7384464c46a45330cef" exitCode=0
Oct 30 13:01:54.450750 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:54.450364    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fedd835-c2x48" event={"ID":"28879ce9-05ee-4ba5-8cd2-98d514e651e1","Type":"ContainerDied","Data":"5bcceaf30c078cd17436b2de6eac5be36c038837ba56b7384464c46a45330cef"}
Oct 30 13:01:54.579359 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:54.579322    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:55.053555 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.053519    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"]
Oct 30 13:01:55.057782 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.057757    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:01:55.094959 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.094927    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"]
Oct 30 13:01:55.165995 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.165890    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtp64\" (UniqueName: \"kubernetes.io/projected/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-kube-api-access-dtp64\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:01:55.165995 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.165981    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istiod-ca-cert\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:01:55.166370 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.166023    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istio-data\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:01:55.167343 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.167322    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-workload-certs\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:01:55.167428 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.167359    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-credential-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:01:55.167428 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.167391    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-workload-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:01:55.167518 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.167422    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istio-token\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:01:55.167622 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.167599    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istio-envoy\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:01:55.167763 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.167740    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istio-podinfo\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:01:55.269425 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.269381    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istiod-ca-cert\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:01:55.269637 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.269439    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istio-data\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:01:55.269637 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.269482    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-workload-certs\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:01:55.269637 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.269504    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-credential-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:01:55.269637 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.269583    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-workload-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:01:55.269637 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.269623    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istio-token\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:01:55.270141 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.269744    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istio-envoy\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:01:55.270141 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.269853    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istio-podinfo\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:01:55.270141 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.269909    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtp64\" (UniqueName: \"kubernetes.io/projected/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-kube-api-access-dtp64\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:01:55.270141 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.269991    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istio-data\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:01:55.270668 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.270184    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-credential-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:01:55.270668 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.270212    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-workload-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:01:55.270668 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.270485    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-workload-certs\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:01:55.270955 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.270934    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istiod-ca-cert\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:01:55.272422 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.272397    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istio-envoy\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:01:55.272650 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.272633    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istio-podinfo\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:01:55.293456 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.293421    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istio-token\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:01:55.293631 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.293485    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtp64\" (UniqueName: \"kubernetes.io/projected/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-kube-api-access-dtp64\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:01:55.374952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.374914    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:01:55.612153 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.612120    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fedd835-c2x48"
Oct 30 13:01:55.698554 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.698518    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"]
Oct 30 13:01:55.699914 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:01:55.699889    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f93aa79_27fa_4f24_9b20_b6da0dee9ec2.slice/crio-44c7f42a34037c1cb22659354bcb216a04f11b9cbe0b6048d04e7f2ce8da90fd WatchSource:0}: Error finding container 44c7f42a34037c1cb22659354bcb216a04f11b9cbe0b6048d04e7f2ce8da90fd: Status 404 returned error can't find the container with id 44c7f42a34037c1cb22659354bcb216a04f11b9cbe0b6048d04e7f2ce8da90fd
Oct 30 13:01:55.702932 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.702896    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:01:55.703011 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.702966    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:01:55.703011 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.702992    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:01:55.776515 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.776483    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28879ce9-05ee-4ba5-8cd2-98d514e651e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28879ce9-05ee-4ba5-8cd2-98d514e651e1-maas-api-xks-ca\") pod \"28879ce9-05ee-4ba5-8cd2-98d514e651e1\" (UID: \"28879ce9-05ee-4ba5-8cd2-98d514e651e1\") "
Oct 30 13:01:55.776722 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.776537    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28879ce9-05ee-4ba5-8cd2-98d514e651e1-kube-api-access-w8d48\" (UniqueName: \"kubernetes.io/projected/28879ce9-05ee-4ba5-8cd2-98d514e651e1-kube-api-access-w8d48\") pod \"28879ce9-05ee-4ba5-8cd2-98d514e651e1\" (UID: \"28879ce9-05ee-4ba5-8cd2-98d514e651e1\") "
Oct 30 13:01:55.776722 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.776578    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28879ce9-05ee-4ba5-8cd2-98d514e651e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28879ce9-05ee-4ba5-8cd2-98d514e651e1-maas-api-openshift-ca\") pod \"28879ce9-05ee-4ba5-8cd2-98d514e651e1\" (UID: \"28879ce9-05ee-4ba5-8cd2-98d514e651e1\") "
Oct 30 13:01:55.777099 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.777062    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28879ce9-05ee-4ba5-8cd2-98d514e651e1-maas-api-openshift-ca" pod "28879ce9-05ee-4ba5-8cd2-98d514e651e1" (UID: "28879ce9-05ee-4ba5-8cd2-98d514e651e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:01:55.778747 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.778716    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28879ce9-05ee-4ba5-8cd2-98d514e651e1-kube-api-access-w8d48" pod "28879ce9-05ee-4ba5-8cd2-98d514e651e1" (UID: "28879ce9-05ee-4ba5-8cd2-98d514e651e1"). InnerVolumeSpecName "kube-api-access-w8d48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:55.778747 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.778724    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28879ce9-05ee-4ba5-8cd2-98d514e651e1-maas-api-xks-ca" pod "28879ce9-05ee-4ba5-8cd2-98d514e651e1" (UID: "28879ce9-05ee-4ba5-8cd2-98d514e651e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:01:55.878802 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.878767    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28879ce9-05ee-4ba5-8cd2-98d514e651e1-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:01:55.878802 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.878797    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8d48\" (UniqueName: \"kubernetes.io/projected/28879ce9-05ee-4ba5-8cd2-98d514e651e1-kube-api-access-w8d48\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:01:55.878802 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:55.878807    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28879ce9-05ee-4ba5-8cd2-98d514e651e1-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:01:56.403855 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:56.403813    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:56.488280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:56.488239    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn" event={"ID":"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2","Type":"ContainerStarted","Data":"91258d98de4e7bf522bcc4e337e678335fb6d5c3a18aa8cb6f2629fb36e640a2"}
Oct 30 13:01:56.488565 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:56.488541    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn" event={"ID":"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2","Type":"ContainerStarted","Data":"44c7f42a34037c1cb22659354bcb216a04f11b9cbe0b6048d04e7f2ce8da90fd"}
Oct 30 13:01:56.492117 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:56.492087    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fedd835-c2x48" event={"ID":"28879ce9-05ee-4ba5-8cd2-98d514e651e1","Type":"ContainerDied","Data":"9423b3fd8e3ebe08fd94254530b250e39519d8936180acfe3d736864c5dd7f8d"}
Oct 30 13:01:56.492342 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:56.492325    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9423b3fd8e3ebe08fd94254530b250e39519d8936180acfe3d736864c5dd7f8d"
Oct 30 13:01:56.492547 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:56.492533    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fedd835-c2x48"
Oct 30 13:01:56.544611 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:56.539768    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn" podStartSLOduration=1.539734028 podStartE2EDuration="1.539734028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:56.536958855 +0000 UTC m=+1011.584092820" watchObservedRunningTime="2026-10-30 13:01:56.539734028 +0000 UTC m=+1011.586867982"
Oct 30 13:01:57.380246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:57.375390    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:01:57.384455 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:57.384416    4544 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 13:01:57.384866 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:57.384813    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn" podUID="7f93aa79-27fa-4f24-9b20-b6da0dee9ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 13:01:58.375928 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:58.375886    4544 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 13:01:58.376146 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:58.375961    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn" podUID="7f93aa79-27fa-4f24-9b20-b6da0dee9ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 13:01:58.490535 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:58.490401    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:01:59.044904 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:01:59.044834    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:01:59.375952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:59.375911    4544 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 13:01:59.376138 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:01:59.375977    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn" podUID="7f93aa79-27fa-4f24-9b20-b6da0dee9ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 13:02:00.375732 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:00.375694    4544 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 13:02:00.376149 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:00.375745    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn" podUID="7f93aa79-27fa-4f24-9b20-b6da0dee9ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 13:02:00.438239 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:00.438188    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:01.375873 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:01.375840    4544 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 13:02:01.376305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:01.375908    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn" podUID="7f93aa79-27fa-4f24-9b20-b6da0dee9ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 13:02:02.369951 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:02.369898    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:02.375555 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:02.375524    4544 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 13:02:02.375698 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:02.375579    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn" podUID="7f93aa79-27fa-4f24-9b20-b6da0dee9ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 13:02:04.254560 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:04.254520    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:04.353439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:04.353405    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:02:04.353740 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:04.353718    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:02:04.354669 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:04.354649    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:02:06.144114 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:06.144079    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:06.334615 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:06.334580    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a106d20d-hqnr5"]
Oct 30 13:02:06.335150 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:06.335137    4544 cpu_manager.go:414] "containerMap: removing container" podUID="28879ce9-05ee-4ba5-8cd2-98d514e651e1" containerName="revoke-keys"
Oct 30 13:02:06.335193 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:06.335152    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28879ce9-05ee-4ba5-8cd2-98d514e651e1" containerName="revoke-keys"
Oct 30 13:02:06.335267 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:06.335262    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="28879ce9-05ee-4ba5-8cd2-98d514e651e1" containerName="revoke-keys"
Oct 30 13:02:06.340558 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:06.340537    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a106d20d-hqnr5"
Oct 30 13:02:06.367900 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:06.367865    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:02:06.368061 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:06.367933    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:02:06.374259 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:06.374204    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a106d20d-hqnr5"]
Oct 30 13:02:06.407590 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:06.407497    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d40b4984-06b7-4d0d-bc03-39491f39cf2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a106d20d-hqnr5\" (UID: \"d40b4984-06b7-4d0d-bc03-39491f39cf2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a106d20d-hqnr5"
Oct 30 13:02:06.407752 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:06.407662    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mzrn\" (UniqueName: \"kubernetes.io/projected/d40b4984-06b7-4d0d-bc03-39491f39cf2d-kube-api-access-2mzrn\") pod \"maas-api-revoke-sub-a106d20d-hqnr5\" (UID: \"d40b4984-06b7-4d0d-bc03-39491f39cf2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a106d20d-hqnr5"
Oct 30 13:02:06.407752 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:06.407724    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d40b4984-06b7-4d0d-bc03-39491f39cf2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a106d20d-hqnr5\" (UID: \"d40b4984-06b7-4d0d-bc03-39491f39cf2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a106d20d-hqnr5"
Oct 30 13:02:06.509079 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:06.509046    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d40b4984-06b7-4d0d-bc03-39491f39cf2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a106d20d-hqnr5\" (UID: \"d40b4984-06b7-4d0d-bc03-39491f39cf2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a106d20d-hqnr5"
Oct 30 13:02:06.509316 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:06.509152    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mzrn\" (UniqueName: \"kubernetes.io/projected/d40b4984-06b7-4d0d-bc03-39491f39cf2d-kube-api-access-2mzrn\") pod \"maas-api-revoke-sub-a106d20d-hqnr5\" (UID: \"d40b4984-06b7-4d0d-bc03-39491f39cf2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a106d20d-hqnr5"
Oct 30 13:02:06.509316 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:06.509191    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d40b4984-06b7-4d0d-bc03-39491f39cf2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a106d20d-hqnr5\" (UID: \"d40b4984-06b7-4d0d-bc03-39491f39cf2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a106d20d-hqnr5"
Oct 30 13:02:06.509824 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:06.509796    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d40b4984-06b7-4d0d-bc03-39491f39cf2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a106d20d-hqnr5\" (UID: \"d40b4984-06b7-4d0d-bc03-39491f39cf2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a106d20d-hqnr5"
Oct 30 13:02:06.511480 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:06.511459    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d40b4984-06b7-4d0d-bc03-39491f39cf2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a106d20d-hqnr5\" (UID: \"d40b4984-06b7-4d0d-bc03-39491f39cf2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a106d20d-hqnr5"
Oct 30 13:02:06.533938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:06.533913    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mzrn\" (UniqueName: \"kubernetes.io/projected/d40b4984-06b7-4d0d-bc03-39491f39cf2d-kube-api-access-2mzrn\") pod \"maas-api-revoke-sub-a106d20d-hqnr5\" (UID: \"d40b4984-06b7-4d0d-bc03-39491f39cf2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a106d20d-hqnr5"
Oct 30 13:02:06.653770 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:06.653731    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a106d20d-hqnr5"
Oct 30 13:02:06.837202 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:06.837166    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:07.006115 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:07.005923    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a106d20d-hqnr5"]
Oct 30 13:02:07.009046 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:02:07.009016    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd40b4984_06b7_4d0d_bc03_39491f39cf2d.slice/crio-4044041eea6dc93440c03f82d9e421e2a7875f51a9c22b11c7d901092ab1623b WatchSource:0}: Error finding container 4044041eea6dc93440c03f82d9e421e2a7875f51a9c22b11c7d901092ab1623b: Status 404 returned error can't find the container with id 4044041eea6dc93440c03f82d9e421e2a7875f51a9c22b11c7d901092ab1623b
Oct 30 13:02:07.621513 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:07.621425    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a106d20d-hqnr5" event={"ID":"d40b4984-06b7-4d0d-bc03-39491f39cf2d","Type":"ContainerStarted","Data":"b17e27dc77390b0928855b70c377080f1e71655b394ee947531449fcfe2efc6c"}
Oct 30 13:02:07.621513 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:07.621494    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a106d20d-hqnr5" event={"ID":"d40b4984-06b7-4d0d-bc03-39491f39cf2d","Type":"ContainerStarted","Data":"4044041eea6dc93440c03f82d9e421e2a7875f51a9c22b11c7d901092ab1623b"}
Oct 30 13:02:07.665148 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:07.665090    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a106d20d-hqnr5" podStartSLOduration=1.665077804 podStartE2EDuration="1.665077804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:07.661657391 +0000 UTC m=+1022.708791345" watchObservedRunningTime="2026-10-30 13:02:07.665077804 +0000 UTC m=+1022.712211756"
Oct 30 13:02:08.187111 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:08.187028    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:08.631523 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:08.631484    4544 generic.go:356] "Generic (PLEG): container finished" podID="d40b4984-06b7-4d0d-bc03-39491f39cf2d" containerID="b17e27dc77390b0928855b70c377080f1e71655b394ee947531449fcfe2efc6c" exitCode=0
Oct 30 13:02:08.632018 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:08.631578    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a106d20d-hqnr5" event={"ID":"d40b4984-06b7-4d0d-bc03-39491f39cf2d","Type":"ContainerDied","Data":"b17e27dc77390b0928855b70c377080f1e71655b394ee947531449fcfe2efc6c"}
Oct 30 13:02:09.514489 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.514450    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"]
Oct 30 13:02:09.518799 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.518777    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:02:09.548052 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.548025    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istiod-ca-cert\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:02:09.548202 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.548100    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-workload-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:02:09.548293 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.548247    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istio-data\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:02:09.548356 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.548320    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-credential-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:02:09.548412 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.548353    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istio-podinfo\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:02:09.548465 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.548425    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istio-envoy\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:02:09.548465 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.548454    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-workload-certs\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:02:09.548557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.548503    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr79h\" (UniqueName: \"kubernetes.io/projected/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-kube-api-access-gr79h\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:02:09.548636 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.548613    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istio-token\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:02:09.568043 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.568011    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"]
Oct 30 13:02:09.650010 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.649976    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istiod-ca-cert\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:02:09.650474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.650076    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-workload-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:02:09.650474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.650111    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istio-data\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:02:09.650474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.650171    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-credential-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:02:09.650474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.650263    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istio-podinfo\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:02:09.650474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.650328    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istio-envoy\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:02:09.650474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.650357    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-workload-certs\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:02:09.650474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.650384    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr79h\" (UniqueName: \"kubernetes.io/projected/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-kube-api-access-gr79h\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:02:09.650474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.650445    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istio-token\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:02:09.650811 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.650483    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-workload-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:02:09.650811 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.650569    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istio-data\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:02:09.650811 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.650607    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-credential-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:02:09.650811 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.650785    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istiod-ca-cert\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:02:09.651019 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.650995    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-workload-certs\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:02:09.652794 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.652771    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istio-envoy\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:02:09.653103 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.653085    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istio-podinfo\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:02:09.674865 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.674834    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istio-token\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:02:09.675375 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.675351    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr79h\" (UniqueName: \"kubernetes.io/projected/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-kube-api-access-gr79h\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:02:09.782193 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.782169    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a106d20d-hqnr5"
Oct 30 13:02:09.835950 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.835663    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:02:09.854085 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.854057    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d40b4984-06b7-4d0d-bc03-39491f39cf2d-kube-api-access-2mzrn\" (UniqueName: \"kubernetes.io/projected/d40b4984-06b7-4d0d-bc03-39491f39cf2d-kube-api-access-2mzrn\") pod \"d40b4984-06b7-4d0d-bc03-39491f39cf2d\" (UID: \"d40b4984-06b7-4d0d-bc03-39491f39cf2d\") "
Oct 30 13:02:09.854252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.854207    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d40b4984-06b7-4d0d-bc03-39491f39cf2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d40b4984-06b7-4d0d-bc03-39491f39cf2d-maas-api-xks-ca\") pod \"d40b4984-06b7-4d0d-bc03-39491f39cf2d\" (UID: \"d40b4984-06b7-4d0d-bc03-39491f39cf2d\") "
Oct 30 13:02:09.854344 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.854270    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d40b4984-06b7-4d0d-bc03-39491f39cf2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d40b4984-06b7-4d0d-bc03-39491f39cf2d-maas-api-openshift-ca\") pod \"d40b4984-06b7-4d0d-bc03-39491f39cf2d\" (UID: \"d40b4984-06b7-4d0d-bc03-39491f39cf2d\") "
Oct 30 13:02:09.854802 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.854776    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d40b4984-06b7-4d0d-bc03-39491f39cf2d-maas-api-openshift-ca" pod "d40b4984-06b7-4d0d-bc03-39491f39cf2d" (UID: "d40b4984-06b7-4d0d-bc03-39491f39cf2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:02:09.856545 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.856519    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d40b4984-06b7-4d0d-bc03-39491f39cf2d-kube-api-access-2mzrn" pod "d40b4984-06b7-4d0d-bc03-39491f39cf2d" (UID: "d40b4984-06b7-4d0d-bc03-39491f39cf2d"). InnerVolumeSpecName "kube-api-access-2mzrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:02:09.857042 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.857010    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d40b4984-06b7-4d0d-bc03-39491f39cf2d-maas-api-xks-ca" pod "d40b4984-06b7-4d0d-bc03-39491f39cf2d" (UID: "d40b4984-06b7-4d0d-bc03-39491f39cf2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:02:09.956565 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.956526    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mzrn\" (UniqueName: \"kubernetes.io/projected/d40b4984-06b7-4d0d-bc03-39491f39cf2d-kube-api-access-2mzrn\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:02:09.956565 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.956565    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d40b4984-06b7-4d0d-bc03-39491f39cf2d-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:02:09.956790 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:09.956582    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d40b4984-06b7-4d0d-bc03-39491f39cf2d-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:02:10.194954 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:10.194916    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"]
Oct 30 13:02:10.198820 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:02:10.198791    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b2c1ca_75ce_45bb_93bc_d1da88552cf5.slice/crio-8b55b923aa981535ce9961283c94667a73833e123af62404bdc1fcc10358c842 WatchSource:0}: Error finding container 8b55b923aa981535ce9961283c94667a73833e123af62404bdc1fcc10358c842: Status 404 returned error can't find the container with id 8b55b923aa981535ce9961283c94667a73833e123af62404bdc1fcc10358c842
Oct 30 13:02:10.201402 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:10.201369    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:02:10.201478 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:10.201440    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:02:10.201478 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:10.201468    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:02:10.319520 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:10.319481    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:10.647552 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:10.647516    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl" event={"ID":"89b2c1ca-75ce-45bb-93bc-d1da88552cf5","Type":"ContainerStarted","Data":"aabf38611462548a1a9eccbca9a5352f764aac7b6afc6de687d730ea0818bfd8"}
Oct 30 13:02:10.647737 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:10.647582    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl" event={"ID":"89b2c1ca-75ce-45bb-93bc-d1da88552cf5","Type":"ContainerStarted","Data":"8b55b923aa981535ce9961283c94667a73833e123af62404bdc1fcc10358c842"}
Oct 30 13:02:10.648968 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:10.648943    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a106d20d-hqnr5"
Oct 30 13:02:10.648968 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:10.648954    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a106d20d-hqnr5" event={"ID":"d40b4984-06b7-4d0d-bc03-39491f39cf2d","Type":"ContainerDied","Data":"4044041eea6dc93440c03f82d9e421e2a7875f51a9c22b11c7d901092ab1623b"}
Oct 30 13:02:10.649119 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:10.648990    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4044041eea6dc93440c03f82d9e421e2a7875f51a9c22b11c7d901092ab1623b"
Oct 30 13:02:10.709808 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:10.709753    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl" podStartSLOduration=1.709740079 podStartE2EDuration="1.709740079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:10.70908774 +0000 UTC m=+1025.756221693" watchObservedRunningTime="2026-10-30 13:02:10.709740079 +0000 UTC m=+1025.756874031"
Oct 30 13:02:10.836614 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:10.836502    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:02:11.839135 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:11.839099    4544 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 13:02:11.839567 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:11.839155    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl" podUID="89b2c1ca-75ce-45bb-93bc-d1da88552cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 13:02:12.197304 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:12.193369    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:12.837900 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:12.837857    4544 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 13:02:12.838077 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:12.837944    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl" podUID="89b2c1ca-75ce-45bb-93bc-d1da88552cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 13:02:13.839245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:13.838668    4544 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 13:02:13.839245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:13.838731    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl" podUID="89b2c1ca-75ce-45bb-93bc-d1da88552cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 13:02:14.171413 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:14.171329    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:14.837248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:14.836684    4544 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 13:02:14.837248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:14.836748    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl" podUID="89b2c1ca-75ce-45bb-93bc-d1da88552cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 13:02:15.837782 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:15.837736    4544 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 13:02:15.838332 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:15.837835    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl" podUID="89b2c1ca-75ce-45bb-93bc-d1da88552cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 13:02:16.415877 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:16.415839    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:16.857248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:16.850361    4544 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 13:02:16.857248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:16.850444    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl" podUID="89b2c1ca-75ce-45bb-93bc-d1da88552cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 13:02:18.636246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:18.635672    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:02:18.640242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:18.637549    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:02:18.640242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:18.639490    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:02:18.814925 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:18.814894    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:19.604400 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:19.604359    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"]
Oct 30 13:02:19.605286 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:19.605263    4544 cpu_manager.go:414] "containerMap: removing container" podUID="d40b4984-06b7-4d0d-bc03-39491f39cf2d" containerName="revoke-keys"
Oct 30 13:02:19.605286 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:19.605286    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d40b4984-06b7-4d0d-bc03-39491f39cf2d" containerName="revoke-keys"
Oct 30 13:02:19.605491 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:19.605473    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="d40b4984-06b7-4d0d-bc03-39491f39cf2d" containerName="revoke-keys"
Oct 30 13:02:19.611380 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:19.611358    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:02:19.642542 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:19.642509    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c90b9e\"/\"openshift-service-ca.crt\""
Oct 30 13:02:19.642908 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:19.642621    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c90b9e\"/\"kube-root-ca.crt\""
Oct 30 13:02:19.648969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:19.648938    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"]
Oct 30 13:02:19.656975 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:19.656943    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c90b9e\"/\"default-dockercfg-v2t7t\""
Oct 30 13:02:19.657111 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:19.657047    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c90b9e\"/\"distinct-2-w0-c90b9e-kserve-self-signed-certs\""
Oct 30 13:02:19.710438 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:19.710396    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24nvl\" (UniqueName: \"kubernetes.io/projected/b66ef925-6b4b-4859-a706-960515034d64-kube-api-access-24nvl\") pod \"distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz\" (UID: \"b66ef925-6b4b-4859-a706-960515034d64\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:02:19.710609 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:19.710443    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b66ef925-6b4b-4859-a706-960515034d64-home\") pod \"distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz\" (UID: \"b66ef925-6b4b-4859-a706-960515034d64\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:02:19.710609 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:19.710583    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b66ef925-6b4b-4859-a706-960515034d64-tls-certs\") pod \"distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz\" (UID: \"b66ef925-6b4b-4859-a706-960515034d64\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:02:19.710730 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:19.710636    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b66ef925-6b4b-4859-a706-960515034d64-model-cache\") pod \"distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz\" (UID: \"b66ef925-6b4b-4859-a706-960515034d64\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:02:19.710817 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:19.710716    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b66ef925-6b4b-4859-a706-960515034d64-tmp-dir\") pod \"distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz\" (UID: \"b66ef925-6b4b-4859-a706-960515034d64\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:02:19.710913 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:19.710818    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b66ef925-6b4b-4859-a706-960515034d64-dshm\") pod \"distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz\" (UID: \"b66ef925-6b4b-4859-a706-960515034d64\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:02:19.717382 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:19.717355    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:19.836242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:19.835109    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24nvl\" (UniqueName: \"kubernetes.io/projected/b66ef925-6b4b-4859-a706-960515034d64-kube-api-access-24nvl\") pod \"distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz\" (UID: \"b66ef925-6b4b-4859-a706-960515034d64\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:02:19.836242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:19.835193    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b66ef925-6b4b-4859-a706-960515034d64-home\") pod \"distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz\" (UID: \"b66ef925-6b4b-4859-a706-960515034d64\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:02:19.836242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:19.835350    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b66ef925-6b4b-4859-a706-960515034d64-tls-certs\") pod \"distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz\" (UID: \"b66ef925-6b4b-4859-a706-960515034d64\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:02:19.836242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:19.835418    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b66ef925-6b4b-4859-a706-960515034d64-model-cache\") pod \"distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz\" (UID: \"b66ef925-6b4b-4859-a706-960515034d64\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:02:19.836242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:19.835462    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b66ef925-6b4b-4859-a706-960515034d64-tmp-dir\") pod \"distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz\" (UID: \"b66ef925-6b4b-4859-a706-960515034d64\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:02:19.836242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:19.835501    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b66ef925-6b4b-4859-a706-960515034d64-dshm\") pod \"distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz\" (UID: \"b66ef925-6b4b-4859-a706-960515034d64\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:02:19.836684 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:19.836531    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b66ef925-6b4b-4859-a706-960515034d64-home\") pod \"distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz\" (UID: \"b66ef925-6b4b-4859-a706-960515034d64\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:02:19.840248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:19.837978    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b66ef925-6b4b-4859-a706-960515034d64-tmp-dir\") pod \"distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz\" (UID: \"b66ef925-6b4b-4859-a706-960515034d64\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:02:19.840248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:19.838285    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b66ef925-6b4b-4859-a706-960515034d64-model-cache\") pod \"distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz\" (UID: \"b66ef925-6b4b-4859-a706-960515034d64\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:02:19.878297 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:19.871832    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b66ef925-6b4b-4859-a706-960515034d64-tls-certs\") pod \"distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz\" (UID: \"b66ef925-6b4b-4859-a706-960515034d64\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:02:19.880647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:19.879960    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24nvl\" (UniqueName: \"kubernetes.io/projected/b66ef925-6b4b-4859-a706-960515034d64-kube-api-access-24nvl\") pod \"distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz\" (UID: \"b66ef925-6b4b-4859-a706-960515034d64\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:02:19.904245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:19.897619    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b66ef925-6b4b-4859-a706-960515034d64-dshm\") pod \"distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz\" (UID: \"b66ef925-6b4b-4859-a706-960515034d64\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:02:19.926079 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:19.924774    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:02:20.260685 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:20.260595    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"]
Oct 30 13:02:20.265132 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:02:20.265100    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb66ef925_6b4b_4859_a706_960515034d64.slice/crio-6a3f0772ca3f206601178af4bb13856635a192231ea2c2d14b5ba222c640511b WatchSource:0}: Error finding container 6a3f0772ca3f206601178af4bb13856635a192231ea2c2d14b5ba222c640511b: Status 404 returned error can't find the container with id 6a3f0772ca3f206601178af4bb13856635a192231ea2c2d14b5ba222c640511b
Oct 30 13:02:20.469231 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:20.469180    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e24e8752-2npsq"]
Oct 30 13:02:20.473353 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:20.473329    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24e8752-2npsq"
Oct 30 13:02:20.493770 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:20.493745    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:02:20.497884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:20.497852    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e24e8752-2npsq"]
Oct 30 13:02:20.499358 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:20.499335    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:02:20.542614 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:20.542506    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a52feac-644f-4fe3-9599-8f6af1172191-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e24e8752-2npsq\" (UID: \"3a52feac-644f-4fe3-9599-8f6af1172191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24e8752-2npsq"
Oct 30 13:02:20.542787 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:20.542615    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j22kf\" (UniqueName: \"kubernetes.io/projected/3a52feac-644f-4fe3-9599-8f6af1172191-kube-api-access-j22kf\") pod \"maas-api-revoke-sub-e24e8752-2npsq\" (UID: \"3a52feac-644f-4fe3-9599-8f6af1172191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24e8752-2npsq"
Oct 30 13:02:20.542787 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:20.542668    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a52feac-644f-4fe3-9599-8f6af1172191-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e24e8752-2npsq\" (UID: \"3a52feac-644f-4fe3-9599-8f6af1172191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24e8752-2npsq"
Oct 30 13:02:20.643935 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:20.643901    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a52feac-644f-4fe3-9599-8f6af1172191-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e24e8752-2npsq\" (UID: \"3a52feac-644f-4fe3-9599-8f6af1172191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24e8752-2npsq"
Oct 30 13:02:20.644468 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:20.643964    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j22kf\" (UniqueName: \"kubernetes.io/projected/3a52feac-644f-4fe3-9599-8f6af1172191-kube-api-access-j22kf\") pod \"maas-api-revoke-sub-e24e8752-2npsq\" (UID: \"3a52feac-644f-4fe3-9599-8f6af1172191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24e8752-2npsq"
Oct 30 13:02:20.644468 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:20.643999    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a52feac-644f-4fe3-9599-8f6af1172191-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e24e8752-2npsq\" (UID: \"3a52feac-644f-4fe3-9599-8f6af1172191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24e8752-2npsq"
Oct 30 13:02:20.644701 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:20.644681    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a52feac-644f-4fe3-9599-8f6af1172191-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e24e8752-2npsq\" (UID: \"3a52feac-644f-4fe3-9599-8f6af1172191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24e8752-2npsq"
Oct 30 13:02:20.646424 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:20.646406    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a52feac-644f-4fe3-9599-8f6af1172191-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e24e8752-2npsq\" (UID: \"3a52feac-644f-4fe3-9599-8f6af1172191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24e8752-2npsq"
Oct 30 13:02:20.670391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:20.670362    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j22kf\" (UniqueName: \"kubernetes.io/projected/3a52feac-644f-4fe3-9599-8f6af1172191-kube-api-access-j22kf\") pod \"maas-api-revoke-sub-e24e8752-2npsq\" (UID: \"3a52feac-644f-4fe3-9599-8f6af1172191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24e8752-2npsq"
Oct 30 13:02:20.724804 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:20.724773    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" event={"ID":"b66ef925-6b4b-4859-a706-960515034d64","Type":"ContainerStarted","Data":"29e718e88749e1646a66da0a7a1865e8ada6d73917ba134a98ae4d2b242a797a"}
Oct 30 13:02:20.724978 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:20.724813    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" event={"ID":"b66ef925-6b4b-4859-a706-960515034d64","Type":"ContainerStarted","Data":"6a3f0772ca3f206601178af4bb13856635a192231ea2c2d14b5ba222c640511b"}
Oct 30 13:02:20.780421 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:20.780361    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podStartSLOduration=1.780348698 podStartE2EDuration="1.780348698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:20.778331074 +0000 UTC m=+1035.825465028" watchObservedRunningTime="2026-10-30 13:02:20.780348698 +0000 UTC m=+1035.827482651"
Oct 30 13:02:20.786946 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:20.786921    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24e8752-2npsq"
Oct 30 13:02:20.959793 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:20.954670    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:02:21.021211 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:21.021170    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:21.113635 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:21.113608    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e24e8752-2npsq"]
Oct 30 13:02:21.117002 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:02:21.116969    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a52feac_644f_4fe3_9599_8f6af1172191.slice/crio-6bbc164f7cd9c90696771b43f01b313914051a7303636793f4c3079c4765c604 WatchSource:0}: Error finding container 6bbc164f7cd9c90696771b43f01b313914051a7303636793f4c3079c4765c604: Status 404 returned error can't find the container with id 6bbc164f7cd9c90696771b43f01b313914051a7303636793f4c3079c4765c604
Oct 30 13:02:21.733273 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:21.733160    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24e8752-2npsq" event={"ID":"3a52feac-644f-4fe3-9599-8f6af1172191","Type":"ContainerStarted","Data":"168d03a3ebd722e60fecfac7c7daa1e4f15ba656c5b69f7fcf5a78972f11f26b"}
Oct 30 13:02:21.733273 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:21.733229    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24e8752-2npsq" event={"ID":"3a52feac-644f-4fe3-9599-8f6af1172191","Type":"ContainerStarted","Data":"6bbc164f7cd9c90696771b43f01b313914051a7303636793f4c3079c4765c604"}
Oct 30 13:02:21.778613 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:21.778563    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24e8752-2npsq" podStartSLOduration=1.778548413 podStartE2EDuration="1.778548413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:21.777273987 +0000 UTC m=+1036.824407942" watchObservedRunningTime="2026-10-30 13:02:21.778548413 +0000 UTC m=+1036.825682365"
Oct 30 13:02:22.740873 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:22.740839    4544 generic.go:356] "Generic (PLEG): container finished" podID="3a52feac-644f-4fe3-9599-8f6af1172191" containerID="168d03a3ebd722e60fecfac7c7daa1e4f15ba656c5b69f7fcf5a78972f11f26b" exitCode=0
Oct 30 13:02:22.741322 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:22.740927    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24e8752-2npsq" event={"ID":"3a52feac-644f-4fe3-9599-8f6af1172191","Type":"ContainerDied","Data":"168d03a3ebd722e60fecfac7c7daa1e4f15ba656c5b69f7fcf5a78972f11f26b"}
Oct 30 13:02:22.908510 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:22.908474    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:23.914415 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:23.914388    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24e8752-2npsq"
Oct 30 13:02:24.002097 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:24.001997    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a52feac-644f-4fe3-9599-8f6af1172191-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a52feac-644f-4fe3-9599-8f6af1172191-maas-api-xks-ca\") pod \"3a52feac-644f-4fe3-9599-8f6af1172191\" (UID: \"3a52feac-644f-4fe3-9599-8f6af1172191\") "
Oct 30 13:02:24.002308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:24.002107    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a52feac-644f-4fe3-9599-8f6af1172191-kube-api-access-j22kf\" (UniqueName: \"kubernetes.io/projected/3a52feac-644f-4fe3-9599-8f6af1172191-kube-api-access-j22kf\") pod \"3a52feac-644f-4fe3-9599-8f6af1172191\" (UID: \"3a52feac-644f-4fe3-9599-8f6af1172191\") "
Oct 30 13:02:24.002392 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:24.002366    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a52feac-644f-4fe3-9599-8f6af1172191-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a52feac-644f-4fe3-9599-8f6af1172191-maas-api-openshift-ca\") pod \"3a52feac-644f-4fe3-9599-8f6af1172191\" (UID: \"3a52feac-644f-4fe3-9599-8f6af1172191\") "
Oct 30 13:02:24.002883 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:24.002852    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a52feac-644f-4fe3-9599-8f6af1172191-maas-api-openshift-ca" pod "3a52feac-644f-4fe3-9599-8f6af1172191" (UID: "3a52feac-644f-4fe3-9599-8f6af1172191"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:02:24.004791 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:24.004768    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a52feac-644f-4fe3-9599-8f6af1172191-maas-api-xks-ca" pod "3a52feac-644f-4fe3-9599-8f6af1172191" (UID: "3a52feac-644f-4fe3-9599-8f6af1172191"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:02:24.012360 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:24.012334    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a52feac-644f-4fe3-9599-8f6af1172191-kube-api-access-j22kf" pod "3a52feac-644f-4fe3-9599-8f6af1172191" (UID: "3a52feac-644f-4fe3-9599-8f6af1172191"). InnerVolumeSpecName "kube-api-access-j22kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:02:24.104113 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:24.104067    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a52feac-644f-4fe3-9599-8f6af1172191-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:02:24.104113 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:24.104106    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a52feac-644f-4fe3-9599-8f6af1172191-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:02:24.104113 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:24.104116    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j22kf\" (UniqueName: \"kubernetes.io/projected/3a52feac-644f-4fe3-9599-8f6af1172191-kube-api-access-j22kf\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:02:24.694802 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:24.693920    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:24.766458 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:24.766425    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24e8752-2npsq"
Oct 30 13:02:24.766628 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:24.766433    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24e8752-2npsq" event={"ID":"3a52feac-644f-4fe3-9599-8f6af1172191","Type":"ContainerDied","Data":"6bbc164f7cd9c90696771b43f01b313914051a7303636793f4c3079c4765c604"}
Oct 30 13:02:24.766628 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:24.766579    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bbc164f7cd9c90696771b43f01b313914051a7303636793f4c3079c4765c604"
Oct 30 13:02:26.796564 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:26.796523    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:28.899685 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:28.899650    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:29.069106 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:02:29.069069    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:02:29.925721 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:29.925681    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:02:29.938567 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:29.938540    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:02:30.716922 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:30.716884    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:30.877173 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:30.877137    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:02:32.576893 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:32.576862    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:33.493754 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:33.493711    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:34.382041 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:34.381988    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:36.474841 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:36.474809    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:37.427005 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:37.426960    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a63d1b22-trb28"]
Oct 30 13:02:37.427665 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:37.427644    4544 cpu_manager.go:414] "containerMap: removing container" podUID="3a52feac-644f-4fe3-9599-8f6af1172191" containerName="revoke-keys"
Oct 30 13:02:37.427665 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:37.427665    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a52feac-644f-4fe3-9599-8f6af1172191" containerName="revoke-keys"
Oct 30 13:02:37.427846 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:37.427834    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a52feac-644f-4fe3-9599-8f6af1172191" containerName="revoke-keys"
Oct 30 13:02:37.432658 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:37.432636    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63d1b22-trb28"
Oct 30 13:02:37.451572 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:37.451549    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:02:37.467716 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:37.463165    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:02:37.467953 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:37.467928    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a63d1b22-trb28"]
Oct 30 13:02:37.515090 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:37.515038    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d7b6fa9-cc8f-45a8-b463-2b240666966f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a63d1b22-trb28\" (UID: \"8d7b6fa9-cc8f-45a8-b463-2b240666966f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63d1b22-trb28"
Oct 30 13:02:37.515590 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:37.515305    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d7b6fa9-cc8f-45a8-b463-2b240666966f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a63d1b22-trb28\" (UID: \"8d7b6fa9-cc8f-45a8-b463-2b240666966f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63d1b22-trb28"
Oct 30 13:02:37.515590 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:37.515422    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn4c8\" (UniqueName: \"kubernetes.io/projected/8d7b6fa9-cc8f-45a8-b463-2b240666966f-kube-api-access-vn4c8\") pod \"maas-api-revoke-sub-a63d1b22-trb28\" (UID: \"8d7b6fa9-cc8f-45a8-b463-2b240666966f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63d1b22-trb28"
Oct 30 13:02:37.616129 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:37.616095    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn4c8\" (UniqueName: \"kubernetes.io/projected/8d7b6fa9-cc8f-45a8-b463-2b240666966f-kube-api-access-vn4c8\") pod \"maas-api-revoke-sub-a63d1b22-trb28\" (UID: \"8d7b6fa9-cc8f-45a8-b463-2b240666966f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63d1b22-trb28"
Oct 30 13:02:37.616337 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:37.616181    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d7b6fa9-cc8f-45a8-b463-2b240666966f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a63d1b22-trb28\" (UID: \"8d7b6fa9-cc8f-45a8-b463-2b240666966f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63d1b22-trb28"
Oct 30 13:02:37.616337 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:37.616305    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d7b6fa9-cc8f-45a8-b463-2b240666966f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a63d1b22-trb28\" (UID: \"8d7b6fa9-cc8f-45a8-b463-2b240666966f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63d1b22-trb28"
Oct 30 13:02:37.616942 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:37.616918    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d7b6fa9-cc8f-45a8-b463-2b240666966f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a63d1b22-trb28\" (UID: \"8d7b6fa9-cc8f-45a8-b463-2b240666966f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63d1b22-trb28"
Oct 30 13:02:37.618720 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:37.618694    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d7b6fa9-cc8f-45a8-b463-2b240666966f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a63d1b22-trb28\" (UID: \"8d7b6fa9-cc8f-45a8-b463-2b240666966f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63d1b22-trb28"
Oct 30 13:02:37.639035 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:37.639009    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn4c8\" (UniqueName: \"kubernetes.io/projected/8d7b6fa9-cc8f-45a8-b463-2b240666966f-kube-api-access-vn4c8\") pod \"maas-api-revoke-sub-a63d1b22-trb28\" (UID: \"8d7b6fa9-cc8f-45a8-b463-2b240666966f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63d1b22-trb28"
Oct 30 13:02:37.746526 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:37.746445    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63d1b22-trb28"
Oct 30 13:02:38.054281 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:38.054150    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a63d1b22-trb28"]
Oct 30 13:02:38.058713 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:02:38.058687    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d7b6fa9_cc8f_45a8_b463_2b240666966f.slice/crio-641617ccd0eb67cc83f8b9338f5765c99c51795086b3416a1772759615b6be69 WatchSource:0}: Error finding container 641617ccd0eb67cc83f8b9338f5765c99c51795086b3416a1772759615b6be69: Status 404 returned error can't find the container with id 641617ccd0eb67cc83f8b9338f5765c99c51795086b3416a1772759615b6be69
Oct 30 13:02:38.273364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:38.273325    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:38.930295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:38.930260    4544 generic.go:356] "Generic (PLEG): container finished" podID="8d7b6fa9-cc8f-45a8-b463-2b240666966f" containerID="f2b0c8724717e5f839c36c07a23fc988f3278c7cfe6553a7defc6280de70c228" exitCode=0
Oct 30 13:02:38.930654 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:38.930348    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63d1b22-trb28" event={"ID":"8d7b6fa9-cc8f-45a8-b463-2b240666966f","Type":"ContainerDied","Data":"f2b0c8724717e5f839c36c07a23fc988f3278c7cfe6553a7defc6280de70c228"}
Oct 30 13:02:38.930654 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:38.930404    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63d1b22-trb28" event={"ID":"8d7b6fa9-cc8f-45a8-b463-2b240666966f","Type":"ContainerStarted","Data":"641617ccd0eb67cc83f8b9338f5765c99c51795086b3416a1772759615b6be69"}
Oct 30 13:02:40.099058 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.099029    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63d1b22-trb28"
Oct 30 13:02:40.219726 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.219638    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:40.247105 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.239319    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"]
Oct 30 13:02:40.247105 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.240326    4544 cpu_manager.go:414] "containerMap: removing container" podUID="8d7b6fa9-cc8f-45a8-b463-2b240666966f" containerName="revoke-keys"
Oct 30 13:02:40.247105 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.240350    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d7b6fa9-cc8f-45a8-b463-2b240666966f" containerName="revoke-keys"
Oct 30 13:02:40.247105 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.240532    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d7b6fa9-cc8f-45a8-b463-2b240666966f" containerName="revoke-keys"
Oct 30 13:02:40.270254 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.252736    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d7b6fa9-cc8f-45a8-b463-2b240666966f-kube-api-access-vn4c8\" (UniqueName: \"kubernetes.io/projected/8d7b6fa9-cc8f-45a8-b463-2b240666966f-kube-api-access-vn4c8\") pod \"8d7b6fa9-cc8f-45a8-b463-2b240666966f\" (UID: \"8d7b6fa9-cc8f-45a8-b463-2b240666966f\") "
Oct 30 13:02:40.270254 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.252878    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d7b6fa9-cc8f-45a8-b463-2b240666966f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d7b6fa9-cc8f-45a8-b463-2b240666966f-maas-api-xks-ca\") pod \"8d7b6fa9-cc8f-45a8-b463-2b240666966f\" (UID: \"8d7b6fa9-cc8f-45a8-b463-2b240666966f\") "
Oct 30 13:02:40.270254 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.253035    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d7b6fa9-cc8f-45a8-b463-2b240666966f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d7b6fa9-cc8f-45a8-b463-2b240666966f-maas-api-openshift-ca\") pod \"8d7b6fa9-cc8f-45a8-b463-2b240666966f\" (UID: \"8d7b6fa9-cc8f-45a8-b463-2b240666966f\") "
Oct 30 13:02:40.280125 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.270787    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d7b6fa9-cc8f-45a8-b463-2b240666966f-maas-api-openshift-ca" pod "8d7b6fa9-cc8f-45a8-b463-2b240666966f" (UID: "8d7b6fa9-cc8f-45a8-b463-2b240666966f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:02:40.293086 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.293038    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d7b6fa9-cc8f-45a8-b463-2b240666966f-kube-api-access-vn4c8" pod "8d7b6fa9-cc8f-45a8-b463-2b240666966f" (UID: "8d7b6fa9-cc8f-45a8-b463-2b240666966f"). InnerVolumeSpecName "kube-api-access-vn4c8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:02:40.307562 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.307525    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"]
Oct 30 13:02:40.307734 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.307674    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:02:40.308911 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.308878    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d7b6fa9-cc8f-45a8-b463-2b240666966f-maas-api-xks-ca" pod "8d7b6fa9-cc8f-45a8-b463-2b240666966f" (UID: "8d7b6fa9-cc8f-45a8-b463-2b240666966f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:02:40.347648 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.347617    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"]
Oct 30 13:02:40.352377 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.352354    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:02:40.355792 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.355758    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d7b6fa9-cc8f-45a8-b463-2b240666966f-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:02:40.355792 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.355786    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn4c8\" (UniqueName: \"kubernetes.io/projected/8d7b6fa9-cc8f-45a8-b463-2b240666966f-kube-api-access-vn4c8\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:02:40.356035 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.355810    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d7b6fa9-cc8f-45a8-b463-2b240666966f-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:02:40.378940 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.378904    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"]
Oct 30 13:02:40.456527 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.456479    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c508330-60f7-4a7c-abf6-26bd9c811e53-istio-podinfo\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:02:40.456738 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.456533    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w9xl\" (UniqueName: \"kubernetes.io/projected/1bf8120f-fb85-451c-922f-7e1cafe165df-kube-api-access-8w9xl\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:02:40.456738 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.456629    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bf8120f-fb85-451c-922f-7e1cafe165df-istiod-ca-cert\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:02:40.456738 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.456664    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-credential-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:02:40.456969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.456739    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-istio-envoy\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:02:40.456969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.456791    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bf8120f-fb85-451c-922f-7e1cafe165df-istio-podinfo\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:02:40.456969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.456869    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-istio-data\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:02:40.456969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.456900    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bf8120f-fb85-451c-922f-7e1cafe165df-istio-token\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:02:40.456969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.456929    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-istio-data\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:02:40.457153 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.457044    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-workload-certs\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:02:40.457153 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.457102    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-credential-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:02:40.457275 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.457178    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-workload-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:02:40.457275 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.457266    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c508330-60f7-4a7c-abf6-26bd9c811e53-istio-token\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:02:40.457407 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.457385    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-workload-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:02:40.457466 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.457423    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-istio-envoy\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:02:40.457521 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.457502    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wlqs\" (UniqueName: \"kubernetes.io/projected/3c508330-60f7-4a7c-abf6-26bd9c811e53-kube-api-access-5wlqs\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:02:40.457557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.457536    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c508330-60f7-4a7c-abf6-26bd9c811e53-istiod-ca-cert\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:02:40.457606 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.457564    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-workload-certs\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:02:40.558445 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.558343    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c508330-60f7-4a7c-abf6-26bd9c811e53-istio-token\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:02:40.558445 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.558396    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-workload-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:02:40.558445 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.558414    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-istio-envoy\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:02:40.558445 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.558445    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wlqs\" (UniqueName: \"kubernetes.io/projected/3c508330-60f7-4a7c-abf6-26bd9c811e53-kube-api-access-5wlqs\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:02:40.558792 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.558469    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c508330-60f7-4a7c-abf6-26bd9c811e53-istiod-ca-cert\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:02:40.558792 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.558490    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-workload-certs\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:02:40.558792 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.558545    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c508330-60f7-4a7c-abf6-26bd9c811e53-istio-podinfo\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:02:40.558792 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.558672    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w9xl\" (UniqueName: \"kubernetes.io/projected/1bf8120f-fb85-451c-922f-7e1cafe165df-kube-api-access-8w9xl\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:02:40.558792 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.558737    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bf8120f-fb85-451c-922f-7e1cafe165df-istiod-ca-cert\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:02:40.559046 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.558807    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-credential-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:02:40.559046 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.558859    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-istio-envoy\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:02:40.559046 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.558862    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-workload-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:02:40.559046 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.558911    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bf8120f-fb85-451c-922f-7e1cafe165df-istio-podinfo\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:02:40.559046 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.558960    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-workload-certs\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:02:40.559046 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.559008    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-istio-data\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:02:40.559046 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.559039    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bf8120f-fb85-451c-922f-7e1cafe165df-istio-token\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:02:40.560023 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.559063    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-istio-data\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:02:40.560023 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.559107    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-workload-certs\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:02:40.560023 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.559196    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-credential-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:02:40.560023 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.559270    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-workload-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:02:40.560023 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.559268    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c508330-60f7-4a7c-abf6-26bd9c811e53-istiod-ca-cert\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:02:40.560023 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.559513    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-istio-data\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:02:40.560023 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.559271    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-credential-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:02:40.560023 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.559564    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-credential-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:02:40.560023 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.559601    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-istio-data\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:02:40.560023 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.559700    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-workload-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:02:40.560023 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.559774    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-workload-certs\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:02:40.560385 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.560296    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bf8120f-fb85-451c-922f-7e1cafe165df-istiod-ca-cert\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:02:40.561130 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.561104    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-istio-envoy\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:02:40.561497 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.561467    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-istio-envoy\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:02:40.561663 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.561641    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bf8120f-fb85-451c-922f-7e1cafe165df-istio-podinfo\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:02:40.561737 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.561704    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c508330-60f7-4a7c-abf6-26bd9c811e53-istio-podinfo\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:02:40.582572 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.582534    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wlqs\" (UniqueName: \"kubernetes.io/projected/3c508330-60f7-4a7c-abf6-26bd9c811e53-kube-api-access-5wlqs\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:02:40.583118 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.583096    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bf8120f-fb85-451c-922f-7e1cafe165df-istio-token\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:02:40.583734 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.583711    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c508330-60f7-4a7c-abf6-26bd9c811e53-istio-token\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:02:40.583860 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.583845    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w9xl\" (UniqueName: \"kubernetes.io/projected/1bf8120f-fb85-451c-922f-7e1cafe165df-kube-api-access-8w9xl\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:02:40.645138 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.645100    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:02:40.668813 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.668778    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:02:40.946151 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.946107    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63d1b22-trb28"
Oct 30 13:02:40.946383 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.946097    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63d1b22-trb28" event={"ID":"8d7b6fa9-cc8f-45a8-b463-2b240666966f","Type":"ContainerDied","Data":"641617ccd0eb67cc83f8b9338f5765c99c51795086b3416a1772759615b6be69"}
Oct 30 13:02:40.946383 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.946252    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="641617ccd0eb67cc83f8b9338f5765c99c51795086b3416a1772759615b6be69"
Oct 30 13:02:40.981547 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.981509    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"]
Oct 30 13:02:40.988810 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:02:40.988780    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf8120f_fb85_451c_922f_7e1cafe165df.slice/crio-ad8d41b9ea4fe1e61beac2eb88fdaf2830215ce3aa5e02970b15fd0572e12ab1 WatchSource:0}: Error finding container ad8d41b9ea4fe1e61beac2eb88fdaf2830215ce3aa5e02970b15fd0572e12ab1: Status 404 returned error can't find the container with id ad8d41b9ea4fe1e61beac2eb88fdaf2830215ce3aa5e02970b15fd0572e12ab1
Oct 30 13:02:40.991429 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.991390    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:02:40.991510 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.991472    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:02:40.991510 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:40.991502    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:02:41.003651 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:41.003617    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"]
Oct 30 13:02:41.005443 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:02:41.005406    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c508330_60f7_4a7c_abf6_26bd9c811e53.slice/crio-5dacab7425e384945c0ac6c9b7871bf95ffa9b1643665af57a6b5a9600363411 WatchSource:0}: Error finding container 5dacab7425e384945c0ac6c9b7871bf95ffa9b1643665af57a6b5a9600363411: Status 404 returned error can't find the container with id 5dacab7425e384945c0ac6c9b7871bf95ffa9b1643665af57a6b5a9600363411
Oct 30 13:02:41.008201 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:41.008159    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:02:41.008317 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:41.008294    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:02:41.008354 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:41.008346    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:02:41.971867 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:41.971829    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf" event={"ID":"3c508330-60f7-4a7c-abf6-26bd9c811e53","Type":"ContainerStarted","Data":"8e34f4759644a58fd0e623065192fc71f5e40fa721e4eacee4059ae33ca73884"}
Oct 30 13:02:41.972327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:41.971884    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf" event={"ID":"3c508330-60f7-4a7c-abf6-26bd9c811e53","Type":"ContainerStarted","Data":"5dacab7425e384945c0ac6c9b7871bf95ffa9b1643665af57a6b5a9600363411"}
Oct 30 13:02:41.973881 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:41.973852    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz" event={"ID":"1bf8120f-fb85-451c-922f-7e1cafe165df","Type":"ContainerStarted","Data":"fe312d96c23fe5970bde1e3b8ead4c545fcbd408773972722bc15d7cb23fbd46"}
Oct 30 13:02:41.974014 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:41.973905    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz" event={"ID":"1bf8120f-fb85-451c-922f-7e1cafe165df","Type":"ContainerStarted","Data":"ad8d41b9ea4fe1e61beac2eb88fdaf2830215ce3aa5e02970b15fd0572e12ab1"}
Oct 30 13:02:42.051245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:42.051160    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf" podStartSLOduration=2.051139416 podStartE2EDuration="2.051139416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:42.034186033 +0000 UTC m=+1057.081319990" watchObservedRunningTime="2026-10-30 13:02:42.051139416 +0000 UTC m=+1057.098273372"
Oct 30 13:02:42.090152 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:42.090092    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz" podStartSLOduration=2.090077493 podStartE2EDuration="2.090077493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:42.086930695 +0000 UTC m=+1057.134064650" watchObservedRunningTime="2026-10-30 13:02:42.090077493 +0000 UTC m=+1057.137211446"
Oct 30 13:02:42.145245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:42.141844    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:42.379243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:42.376611    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d6c1321-jcxvw"]
Oct 30 13:02:42.410243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:42.387571    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6c1321-jcxvw"
Oct 30 13:02:42.424398 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:42.424355    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d6c1321-jcxvw"]
Oct 30 13:02:42.484277 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:42.458833    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:02:42.484277 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:42.481989    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:02:42.492267 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:42.491902    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3031e4f-f22e-471e-aba2-4bee2f6b4a82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d6c1321-jcxvw\" (UID: \"f3031e4f-f22e-471e-aba2-4bee2f6b4a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6c1321-jcxvw"
Oct 30 13:02:42.492267 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:42.491936    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3031e4f-f22e-471e-aba2-4bee2f6b4a82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d6c1321-jcxvw\" (UID: \"f3031e4f-f22e-471e-aba2-4bee2f6b4a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6c1321-jcxvw"
Oct 30 13:02:42.492267 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:42.492078    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95fn5\" (UniqueName: \"kubernetes.io/projected/f3031e4f-f22e-471e-aba2-4bee2f6b4a82-kube-api-access-95fn5\") pod \"maas-api-revoke-sub-6d6c1321-jcxvw\" (UID: \"f3031e4f-f22e-471e-aba2-4bee2f6b4a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6c1321-jcxvw"
Oct 30 13:02:42.593767 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:42.593724    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3031e4f-f22e-471e-aba2-4bee2f6b4a82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d6c1321-jcxvw\" (UID: \"f3031e4f-f22e-471e-aba2-4bee2f6b4a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6c1321-jcxvw"
Oct 30 13:02:42.593954 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:42.593771    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3031e4f-f22e-471e-aba2-4bee2f6b4a82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d6c1321-jcxvw\" (UID: \"f3031e4f-f22e-471e-aba2-4bee2f6b4a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6c1321-jcxvw"
Oct 30 13:02:42.594037 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:42.594016    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95fn5\" (UniqueName: \"kubernetes.io/projected/f3031e4f-f22e-471e-aba2-4bee2f6b4a82-kube-api-access-95fn5\") pod \"maas-api-revoke-sub-6d6c1321-jcxvw\" (UID: \"f3031e4f-f22e-471e-aba2-4bee2f6b4a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6c1321-jcxvw"
Oct 30 13:02:42.594513 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:42.594486    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3031e4f-f22e-471e-aba2-4bee2f6b4a82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d6c1321-jcxvw\" (UID: \"f3031e4f-f22e-471e-aba2-4bee2f6b4a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6c1321-jcxvw"
Oct 30 13:02:42.596482 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:42.596465    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3031e4f-f22e-471e-aba2-4bee2f6b4a82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d6c1321-jcxvw\" (UID: \"f3031e4f-f22e-471e-aba2-4bee2f6b4a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6c1321-jcxvw"
Oct 30 13:02:42.645503 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:42.645391    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:02:42.651247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:42.648341    4544 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 13:02:42.651247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:42.648405    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz" podUID="1bf8120f-fb85-451c-922f-7e1cafe165df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 13:02:42.661254 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:42.661083    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95fn5\" (UniqueName: \"kubernetes.io/projected/f3031e4f-f22e-471e-aba2-4bee2f6b4a82-kube-api-access-95fn5\") pod \"maas-api-revoke-sub-6d6c1321-jcxvw\" (UID: \"f3031e4f-f22e-471e-aba2-4bee2f6b4a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6c1321-jcxvw"
Oct 30 13:02:42.669800 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:42.669768    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:02:42.671181 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:42.671156    4544 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 13:02:42.671257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:42.671205    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf" podUID="3c508330-60f7-4a7c-abf6-26bd9c811e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 13:02:42.799884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:42.799846    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6c1321-jcxvw"
Oct 30 13:02:43.268046 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:02:43.267992    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3031e4f_f22e_471e_aba2_4bee2f6b4a82.slice/crio-aabf99e755153ee5f3904c9afd5486e9524baa61fecb6d29f133b41c65bfb047 WatchSource:0}: Error finding container aabf99e755153ee5f3904c9afd5486e9524baa61fecb6d29f133b41c65bfb047: Status 404 returned error can't find the container with id aabf99e755153ee5f3904c9afd5486e9524baa61fecb6d29f133b41c65bfb047
Oct 30 13:02:43.268624 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:43.268187    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d6c1321-jcxvw"]
Oct 30 13:02:43.647245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:43.647048    4544 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 13:02:43.647245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:43.647115    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz" podUID="1bf8120f-fb85-451c-922f-7e1cafe165df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 13:02:43.669763 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:43.669727    4544 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 13:02:43.669894 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:43.669782    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf" podUID="3c508330-60f7-4a7c-abf6-26bd9c811e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 13:02:44.069012 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:44.067572    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6c1321-jcxvw" event={"ID":"f3031e4f-f22e-471e-aba2-4bee2f6b4a82","Type":"ContainerStarted","Data":"55bec12d02494bfbf68cb0b269d577e82e9b0ebdbba2846a2261ea0fafae3b47"}
Oct 30 13:02:44.069012 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:44.067645    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6c1321-jcxvw" event={"ID":"f3031e4f-f22e-471e-aba2-4bee2f6b4a82","Type":"ContainerStarted","Data":"aabf99e755153ee5f3904c9afd5486e9524baa61fecb6d29f133b41c65bfb047"}
Oct 30 13:02:44.108888 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:44.108836    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6c1321-jcxvw" podStartSLOduration=2.108821066 podStartE2EDuration="2.108821066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:44.107389922 +0000 UTC m=+1059.154523875" watchObservedRunningTime="2026-10-30 13:02:44.108821066 +0000 UTC m=+1059.155955057"
Oct 30 13:02:44.195886 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:44.195852    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:44.649247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:44.648682    4544 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 13:02:44.649247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:44.648748    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz" podUID="1bf8120f-fb85-451c-922f-7e1cafe165df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 13:02:44.669236 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:44.669185    4544 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 13:02:44.669407 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:44.669257    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf" podUID="3c508330-60f7-4a7c-abf6-26bd9c811e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 13:02:45.079022 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:45.078990    4544 generic.go:356] "Generic (PLEG): container finished" podID="f3031e4f-f22e-471e-aba2-4bee2f6b4a82" containerID="55bec12d02494bfbf68cb0b269d577e82e9b0ebdbba2846a2261ea0fafae3b47" exitCode=0
Oct 30 13:02:45.086737 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:45.086671    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6c1321-jcxvw" event={"ID":"f3031e4f-f22e-471e-aba2-4bee2f6b4a82","Type":"ContainerDied","Data":"55bec12d02494bfbf68cb0b269d577e82e9b0ebdbba2846a2261ea0fafae3b47"}
Oct 30 13:02:45.653861 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:45.653813    4544 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 13:02:45.659857 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:45.659811    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz" podUID="1bf8120f-fb85-451c-922f-7e1cafe165df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 13:02:45.669256 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:45.669188    4544 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 13:02:45.669407 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:45.669279    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf" podUID="3c508330-60f7-4a7c-abf6-26bd9c811e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 13:02:46.288890 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:46.288854    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:46.418295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:46.417747    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6c1321-jcxvw"
Oct 30 13:02:46.508964 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:46.508931    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3031e4f-f22e-471e-aba2-4bee2f6b4a82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3031e4f-f22e-471e-aba2-4bee2f6b4a82-maas-api-xks-ca\") pod \"f3031e4f-f22e-471e-aba2-4bee2f6b4a82\" (UID: \"f3031e4f-f22e-471e-aba2-4bee2f6b4a82\") "
Oct 30 13:02:46.509409 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:46.509376    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3031e4f-f22e-471e-aba2-4bee2f6b4a82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3031e4f-f22e-471e-aba2-4bee2f6b4a82-maas-api-openshift-ca\") pod \"f3031e4f-f22e-471e-aba2-4bee2f6b4a82\" (UID: \"f3031e4f-f22e-471e-aba2-4bee2f6b4a82\") "
Oct 30 13:02:46.509582 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:46.509562    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3031e4f-f22e-471e-aba2-4bee2f6b4a82-kube-api-access-95fn5\" (UniqueName: \"kubernetes.io/projected/f3031e4f-f22e-471e-aba2-4bee2f6b4a82-kube-api-access-95fn5\") pod \"f3031e4f-f22e-471e-aba2-4bee2f6b4a82\" (UID: \"f3031e4f-f22e-471e-aba2-4bee2f6b4a82\") "
Oct 30 13:02:46.509831 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:46.509790    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3031e4f-f22e-471e-aba2-4bee2f6b4a82-maas-api-openshift-ca" pod "f3031e4f-f22e-471e-aba2-4bee2f6b4a82" (UID: "f3031e4f-f22e-471e-aba2-4bee2f6b4a82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:02:46.510138 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:46.510118    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3031e4f-f22e-471e-aba2-4bee2f6b4a82-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:02:46.511745 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:46.511721    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3031e4f-f22e-471e-aba2-4bee2f6b4a82-maas-api-xks-ca" pod "f3031e4f-f22e-471e-aba2-4bee2f6b4a82" (UID: "f3031e4f-f22e-471e-aba2-4bee2f6b4a82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:02:46.512210 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:46.512189    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3031e4f-f22e-471e-aba2-4bee2f6b4a82-kube-api-access-95fn5" pod "f3031e4f-f22e-471e-aba2-4bee2f6b4a82" (UID: "f3031e4f-f22e-471e-aba2-4bee2f6b4a82"). InnerVolumeSpecName "kube-api-access-95fn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:02:46.611415 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:46.611381    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95fn5\" (UniqueName: \"kubernetes.io/projected/f3031e4f-f22e-471e-aba2-4bee2f6b4a82-kube-api-access-95fn5\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:02:46.611415 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:46.611411    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3031e4f-f22e-471e-aba2-4bee2f6b4a82-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:02:46.656984 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:46.656946    4544 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 13:02:46.658010 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:46.657589    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz" podUID="1bf8120f-fb85-451c-922f-7e1cafe165df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 13:02:46.669619 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:46.669404    4544 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 13:02:46.669619 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:46.669480    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf" podUID="3c508330-60f7-4a7c-abf6-26bd9c811e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 13:02:47.098608 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:47.098575    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6c1321-jcxvw"
Oct 30 13:02:47.101237 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:47.101183    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6c1321-jcxvw" event={"ID":"f3031e4f-f22e-471e-aba2-4bee2f6b4a82","Type":"ContainerDied","Data":"aabf99e755153ee5f3904c9afd5486e9524baa61fecb6d29f133b41c65bfb047"}
Oct 30 13:02:47.101465 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:47.101448    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aabf99e755153ee5f3904c9afd5486e9524baa61fecb6d29f133b41c65bfb047"
Oct 30 13:02:47.645816 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:47.645780    4544 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 13:02:47.646039 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:47.645843    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz" podUID="1bf8120f-fb85-451c-922f-7e1cafe165df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 13:02:47.669469 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:47.669421    4544 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 13:02:47.669992 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:47.669487    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf" podUID="3c508330-60f7-4a7c-abf6-26bd9c811e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 13:02:48.413847 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:48.413794    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:48.815527 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:48.815428    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:02:48.822881 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:48.822806    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:02:48.832125 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:48.832091    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:02:48.865342 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:48.865307    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:02:48.865985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:48.865961    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:02:48.870061 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:48.870039    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:02:50.155961 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:50.155934    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88bd852c-9kk2r"]
Oct 30 13:02:50.156540 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:50.156525    4544 cpu_manager.go:414] "containerMap: removing container" podUID="f3031e4f-f22e-471e-aba2-4bee2f6b4a82" containerName="revoke-keys"
Oct 30 13:02:50.156540 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:50.156541    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3031e4f-f22e-471e-aba2-4bee2f6b4a82" containerName="revoke-keys"
Oct 30 13:02:50.156636 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:50.156625    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3031e4f-f22e-471e-aba2-4bee2f6b4a82" containerName="revoke-keys"
Oct 30 13:02:50.160365 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:50.160350    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88bd852c-9kk2r"
Oct 30 13:02:50.179152 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:50.179128    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:02:50.191008 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:50.190984    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:02:50.195303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:50.195281    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88bd852c-9kk2r"]
Oct 30 13:02:50.336300 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:50.336265    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b89906d0-bc2c-48ca-bb49-4a8cd45d2f85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88bd852c-9kk2r\" (UID: \"b89906d0-bc2c-48ca-bb49-4a8cd45d2f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88bd852c-9kk2r"
Oct 30 13:02:50.336496 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:50.336387    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b89906d0-bc2c-48ca-bb49-4a8cd45d2f85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88bd852c-9kk2r\" (UID: \"b89906d0-bc2c-48ca-bb49-4a8cd45d2f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88bd852c-9kk2r"
Oct 30 13:02:50.336496 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:50.336447    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmhd5\" (UniqueName: \"kubernetes.io/projected/b89906d0-bc2c-48ca-bb49-4a8cd45d2f85-kube-api-access-jmhd5\") pod \"maas-api-revoke-sub-88bd852c-9kk2r\" (UID: \"b89906d0-bc2c-48ca-bb49-4a8cd45d2f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88bd852c-9kk2r"
Oct 30 13:02:50.336803 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:50.336770    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:50.438085 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:50.438000    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b89906d0-bc2c-48ca-bb49-4a8cd45d2f85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88bd852c-9kk2r\" (UID: \"b89906d0-bc2c-48ca-bb49-4a8cd45d2f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88bd852c-9kk2r"
Oct 30 13:02:50.438085 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:50.438041    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b89906d0-bc2c-48ca-bb49-4a8cd45d2f85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88bd852c-9kk2r\" (UID: \"b89906d0-bc2c-48ca-bb49-4a8cd45d2f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88bd852c-9kk2r"
Oct 30 13:02:50.438085 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:50.438078    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmhd5\" (UniqueName: \"kubernetes.io/projected/b89906d0-bc2c-48ca-bb49-4a8cd45d2f85-kube-api-access-jmhd5\") pod \"maas-api-revoke-sub-88bd852c-9kk2r\" (UID: \"b89906d0-bc2c-48ca-bb49-4a8cd45d2f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88bd852c-9kk2r"
Oct 30 13:02:50.438724 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:50.438704    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b89906d0-bc2c-48ca-bb49-4a8cd45d2f85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88bd852c-9kk2r\" (UID: \"b89906d0-bc2c-48ca-bb49-4a8cd45d2f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88bd852c-9kk2r"
Oct 30 13:02:50.440553 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:50.440531    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b89906d0-bc2c-48ca-bb49-4a8cd45d2f85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88bd852c-9kk2r\" (UID: \"b89906d0-bc2c-48ca-bb49-4a8cd45d2f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88bd852c-9kk2r"
Oct 30 13:02:50.463479 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:50.463143    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmhd5\" (UniqueName: \"kubernetes.io/projected/b89906d0-bc2c-48ca-bb49-4a8cd45d2f85-kube-api-access-jmhd5\") pod \"maas-api-revoke-sub-88bd852c-9kk2r\" (UID: \"b89906d0-bc2c-48ca-bb49-4a8cd45d2f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88bd852c-9kk2r"
Oct 30 13:02:50.474832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:50.474795    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88bd852c-9kk2r"
Oct 30 13:02:50.834662 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:50.834630    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88bd852c-9kk2r"]
Oct 30 13:02:51.243326 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:51.219318    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88bd852c-9kk2r" event={"ID":"b89906d0-bc2c-48ca-bb49-4a8cd45d2f85","Type":"ContainerStarted","Data":"db4e694d00ac962cf8657f958fdee5aac4660b5257f8ba99bd771829696a42b1"}
Oct 30 13:02:52.307240 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:52.306528    4544 generic.go:356] "Generic (PLEG): container finished" podID="b89906d0-bc2c-48ca-bb49-4a8cd45d2f85" containerID="c9b7a6f2d925ea2360a7142996ee4d14fac7b14d5881f4c5587e2fed5aeec1dc" exitCode=0
Oct 30 13:02:52.307240 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:52.306669    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88bd852c-9kk2r" event={"ID":"b89906d0-bc2c-48ca-bb49-4a8cd45d2f85","Type":"ContainerDied","Data":"c9b7a6f2d925ea2360a7142996ee4d14fac7b14d5881f4c5587e2fed5aeec1dc"}
Oct 30 13:02:53.225257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:53.221158    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:53.359441 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:53.359406    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:53.505911 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:53.505778    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:54.160782 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:54.160755    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88bd852c-9kk2r"
Oct 30 13:02:54.233845 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:54.233801    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b89906d0-bc2c-48ca-bb49-4a8cd45d2f85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b89906d0-bc2c-48ca-bb49-4a8cd45d2f85-maas-api-xks-ca\") pod \"b89906d0-bc2c-48ca-bb49-4a8cd45d2f85\" (UID: \"b89906d0-bc2c-48ca-bb49-4a8cd45d2f85\") "
Oct 30 13:02:54.234044 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:54.234023    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b89906d0-bc2c-48ca-bb49-4a8cd45d2f85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b89906d0-bc2c-48ca-bb49-4a8cd45d2f85-maas-api-openshift-ca\") pod \"b89906d0-bc2c-48ca-bb49-4a8cd45d2f85\" (UID: \"b89906d0-bc2c-48ca-bb49-4a8cd45d2f85\") "
Oct 30 13:02:54.234104 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:54.234089    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b89906d0-bc2c-48ca-bb49-4a8cd45d2f85-kube-api-access-jmhd5\" (UniqueName: \"kubernetes.io/projected/b89906d0-bc2c-48ca-bb49-4a8cd45d2f85-kube-api-access-jmhd5\") pod \"b89906d0-bc2c-48ca-bb49-4a8cd45d2f85\" (UID: \"b89906d0-bc2c-48ca-bb49-4a8cd45d2f85\") "
Oct 30 13:02:54.234461 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:54.234434    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b89906d0-bc2c-48ca-bb49-4a8cd45d2f85-maas-api-openshift-ca" pod "b89906d0-bc2c-48ca-bb49-4a8cd45d2f85" (UID: "b89906d0-bc2c-48ca-bb49-4a8cd45d2f85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:02:54.234691 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:54.234675    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b89906d0-bc2c-48ca-bb49-4a8cd45d2f85-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:02:54.235900 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:54.235873    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b89906d0-bc2c-48ca-bb49-4a8cd45d2f85-maas-api-xks-ca" pod "b89906d0-bc2c-48ca-bb49-4a8cd45d2f85" (UID: "b89906d0-bc2c-48ca-bb49-4a8cd45d2f85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:02:54.236424 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:54.236404    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b89906d0-bc2c-48ca-bb49-4a8cd45d2f85-kube-api-access-jmhd5" pod "b89906d0-bc2c-48ca-bb49-4a8cd45d2f85" (UID: "b89906d0-bc2c-48ca-bb49-4a8cd45d2f85"). InnerVolumeSpecName "kube-api-access-jmhd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:02:54.333556 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:54.333519    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88bd852c-9kk2r"
Oct 30 13:02:54.333811 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:54.333519    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88bd852c-9kk2r" event={"ID":"b89906d0-bc2c-48ca-bb49-4a8cd45d2f85","Type":"ContainerDied","Data":"db4e694d00ac962cf8657f958fdee5aac4660b5257f8ba99bd771829696a42b1"}
Oct 30 13:02:54.333811 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:54.333653    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db4e694d00ac962cf8657f958fdee5aac4660b5257f8ba99bd771829696a42b1"
Oct 30 13:02:54.336001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:54.335969    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b89906d0-bc2c-48ca-bb49-4a8cd45d2f85-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:02:54.336001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:54.335996    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmhd5\" (UniqueName: \"kubernetes.io/projected/b89906d0-bc2c-48ca-bb49-4a8cd45d2f85-kube-api-access-jmhd5\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:02:55.407195 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.407155    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"]
Oct 30 13:02:55.407993 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.407964    4544 cpu_manager.go:414] "containerMap: removing container" podUID="b89906d0-bc2c-48ca-bb49-4a8cd45d2f85" containerName="revoke-keys"
Oct 30 13:02:55.407993 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.407989    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b89906d0-bc2c-48ca-bb49-4a8cd45d2f85" containerName="revoke-keys"
Oct 30 13:02:55.408158 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.408134    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="b89906d0-bc2c-48ca-bb49-4a8cd45d2f85" containerName="revoke-keys"
Oct 30 13:02:55.412010 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.411988    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:02:55.446486 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.446402    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"]
Oct 30 13:02:55.450841 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.450804    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8eb30f8-aa71-4121-9542-4f65aff15344-istio-token\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:02:55.450989 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.450864    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8eb30f8-aa71-4121-9542-4f65aff15344-istiod-ca-cert\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:02:55.451041 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.450981    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2pdw\" (UniqueName: \"kubernetes.io/projected/f8eb30f8-aa71-4121-9542-4f65aff15344-kube-api-access-f2pdw\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:02:55.451098 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.451043    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-workload-certs\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:02:55.451154 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.451135    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8eb30f8-aa71-4121-9542-4f65aff15344-istio-podinfo\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:02:55.451215 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.451183    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-istio-data\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:02:55.451279 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.451261    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2bbc545-9zqtg"]
Oct 30 13:02:55.451315 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.451295    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-credential-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:02:55.451498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.451479    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-workload-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:02:55.451566 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.451544    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-istio-envoy\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:02:55.455630 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.455611    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2bbc545-9zqtg"
Oct 30 13:02:55.470089 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.470044    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:02:55.474185 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.474155    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2bbc545-9zqtg"]
Oct 30 13:02:55.474419 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.474401    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:02:55.474796 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.474771    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:55.552909 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.552858    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80094f69-bff5-4ccd-8849-4af37f390d86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2bbc545-9zqtg\" (UID: \"80094f69-bff5-4ccd-8849-4af37f390d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2bbc545-9zqtg"
Oct 30 13:02:55.553198 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.552919    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-workload-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:02:55.553198 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.552981    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-istio-envoy\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:02:55.553198 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.553020    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxddc\" (UniqueName: \"kubernetes.io/projected/80094f69-bff5-4ccd-8849-4af37f390d86-kube-api-access-fxddc\") pod \"maas-api-revoke-sub-f2bbc545-9zqtg\" (UID: \"80094f69-bff5-4ccd-8849-4af37f390d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2bbc545-9zqtg"
Oct 30 13:02:55.553198 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.553110    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8eb30f8-aa71-4121-9542-4f65aff15344-istio-token\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:02:55.553198 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.553145    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8eb30f8-aa71-4121-9542-4f65aff15344-istiod-ca-cert\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:02:55.553198 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.553196    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2pdw\" (UniqueName: \"kubernetes.io/projected/f8eb30f8-aa71-4121-9542-4f65aff15344-kube-api-access-f2pdw\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:02:55.553517 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.553264    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-workload-certs\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:02:55.553517 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.553299    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8eb30f8-aa71-4121-9542-4f65aff15344-istio-podinfo\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:02:55.553517 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.553337    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-istio-data\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:02:55.553517 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.553369    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-credential-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:02:55.553517 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.553428    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80094f69-bff5-4ccd-8849-4af37f390d86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2bbc545-9zqtg\" (UID: \"80094f69-bff5-4ccd-8849-4af37f390d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2bbc545-9zqtg"
Oct 30 13:02:55.554060 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.554026    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-workload-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:02:55.577012 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.558873    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8eb30f8-aa71-4121-9542-4f65aff15344-istiod-ca-cert\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:02:55.577012 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.559699    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-workload-certs\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:02:55.577012 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.559996    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-credential-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:02:55.577012 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.560533    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-istio-data\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:02:55.586593 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.585253    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-istio-envoy\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:02:55.601304 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.595828    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8eb30f8-aa71-4121-9542-4f65aff15344-istio-token\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:02:55.601304 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.601233    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8eb30f8-aa71-4121-9542-4f65aff15344-istio-podinfo\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:02:55.603952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.603745    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2pdw\" (UniqueName: \"kubernetes.io/projected/f8eb30f8-aa71-4121-9542-4f65aff15344-kube-api-access-f2pdw\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:02:55.609049 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.609006    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"]
Oct 30 13:02:55.615545 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.615506    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:02:55.639270 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.639165    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"]
Oct 30 13:02:55.650298 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.650269    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"]
Oct 30 13:02:55.654503 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.654471    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06d5354a-dc17-49c3-af9c-9b756ad02a00-istiod-ca-cert\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:02:55.654679 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.654522    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxddc\" (UniqueName: \"kubernetes.io/projected/80094f69-bff5-4ccd-8849-4af37f390d86-kube-api-access-fxddc\") pod \"maas-api-revoke-sub-f2bbc545-9zqtg\" (UID: \"80094f69-bff5-4ccd-8849-4af37f390d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2bbc545-9zqtg"
Oct 30 13:02:55.654679 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.654622    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-credential-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:02:55.654679 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.654657    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-istio-data\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:02:55.654876 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.654744    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-workload-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:02:55.654876 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.654793    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80094f69-bff5-4ccd-8849-4af37f390d86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2bbc545-9zqtg\" (UID: \"80094f69-bff5-4ccd-8849-4af37f390d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2bbc545-9zqtg"
Oct 30 13:02:55.654989 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.654902    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-istio-envoy\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:02:55.654989 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.654948    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80094f69-bff5-4ccd-8849-4af37f390d86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2bbc545-9zqtg\" (UID: \"80094f69-bff5-4ccd-8849-4af37f390d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2bbc545-9zqtg"
Oct 30 13:02:55.655268 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.655102    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-workload-certs\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:02:55.655268 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.655145    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06d5354a-dc17-49c3-af9c-9b756ad02a00-istio-podinfo\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:02:55.655268 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.655170    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06d5354a-dc17-49c3-af9c-9b756ad02a00-istio-token\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:02:55.655491 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.655371    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l72tg\" (UniqueName: \"kubernetes.io/projected/06d5354a-dc17-49c3-af9c-9b756ad02a00-kube-api-access-l72tg\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:02:55.655811 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.655774    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80094f69-bff5-4ccd-8849-4af37f390d86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2bbc545-9zqtg\" (UID: \"80094f69-bff5-4ccd-8849-4af37f390d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2bbc545-9zqtg"
Oct 30 13:02:55.658707 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.658649    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80094f69-bff5-4ccd-8849-4af37f390d86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2bbc545-9zqtg\" (UID: \"80094f69-bff5-4ccd-8849-4af37f390d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2bbc545-9zqtg"
Oct 30 13:02:55.658819 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.658717    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:02:55.682744 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.682706    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"]
Oct 30 13:02:55.683286 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.683261    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxddc\" (UniqueName: \"kubernetes.io/projected/80094f69-bff5-4ccd-8849-4af37f390d86-kube-api-access-fxddc\") pod \"maas-api-revoke-sub-f2bbc545-9zqtg\" (UID: \"80094f69-bff5-4ccd-8849-4af37f390d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2bbc545-9zqtg"
Oct 30 13:02:55.726841 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.726804    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:02:55.756421 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.756392    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-workload-certs\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:02:55.756579 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.756458    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-istio-envoy\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:02:55.756579 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.756547    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-workload-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:02:55.756699 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.756580    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istio-podinfo\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:02:55.756699 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.756603    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-workload-certs\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:02:55.756699 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.756641    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06d5354a-dc17-49c3-af9c-9b756ad02a00-istio-podinfo\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:02:55.756860 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.756736    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06d5354a-dc17-49c3-af9c-9b756ad02a00-istio-token\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:02:55.756860 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.756802    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l72tg\" (UniqueName: \"kubernetes.io/projected/06d5354a-dc17-49c3-af9c-9b756ad02a00-kube-api-access-l72tg\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:02:55.756860 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.756842    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istiod-ca-cert\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:02:55.757039 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.756908    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istio-envoy\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:02:55.757039 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.756934    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istio-token\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:02:55.757162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.757059    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06d5354a-dc17-49c3-af9c-9b756ad02a00-istiod-ca-cert\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:02:55.757162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.757092    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sskqq\" (UniqueName: \"kubernetes.io/projected/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-kube-api-access-sskqq\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:02:55.757162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.757109    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-workload-certs\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:02:55.757351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.757320    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istio-data\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:02:55.757416 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.757405    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-credential-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:02:55.757475 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.757438    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-credential-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:02:55.757534 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.757477    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-istio-data\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:02:55.757593 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.757579    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-workload-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:02:55.757794 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.757763    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06d5354a-dc17-49c3-af9c-9b756ad02a00-istiod-ca-cert\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:02:55.758009 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.757984    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-workload-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:02:55.758130 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.758110    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-credential-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:02:55.758201 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.758185    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-istio-data\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:02:55.758971 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.758955    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-istio-envoy\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:02:55.759467 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.759451    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06d5354a-dc17-49c3-af9c-9b756ad02a00-istio-podinfo\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:02:55.777146 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.777122    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2bbc545-9zqtg"
Oct 30 13:02:55.780383 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.780355    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06d5354a-dc17-49c3-af9c-9b756ad02a00-istio-token\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:02:55.780540 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.780504    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l72tg\" (UniqueName: \"kubernetes.io/projected/06d5354a-dc17-49c3-af9c-9b756ad02a00-kube-api-access-l72tg\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:02:55.860427 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.860399    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-credential-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:02:55.860589 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.860569    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-workload-certs\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:02:55.860690 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.860675    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-workload-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:02:55.860751 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.860707    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istio-podinfo\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:02:55.860751 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.860743    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istiod-ca-cert\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:02:55.860856 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.860782    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istio-envoy\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:02:55.860856 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.860810    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istio-token\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:02:55.860961 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.860873    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sskqq\" (UniqueName: \"kubernetes.io/projected/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-kube-api-access-sskqq\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:02:55.860961 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.860940    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istio-data\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:02:55.862266 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.861383    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istio-data\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:02:55.862266 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.862031    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istiod-ca-cert\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:02:55.862894 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.862574    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-workload-certs\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:02:55.862894 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.862609    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-credential-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:02:55.862894 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.862850    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-workload-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:02:55.866978 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.865485    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istio-podinfo\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:02:55.866978 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.865592    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istio-envoy\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:02:55.884963 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.884928    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sskqq\" (UniqueName: \"kubernetes.io/projected/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-kube-api-access-sskqq\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:02:55.884963 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.884951    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istio-token\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:02:55.946566 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.946468    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:02:55.976156 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:55.976121    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:02:56.052185 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:56.052075    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"]
Oct 30 13:02:56.055297 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:02:56.055266    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8eb30f8_aa71_4121_9542_4f65aff15344.slice/crio-f792af01042fe795300890784ffaad82a0a9ecb96436fa07a16c85de4458a0a3 WatchSource:0}: Error finding container f792af01042fe795300890784ffaad82a0a9ecb96436fa07a16c85de4458a0a3: Status 404 returned error can't find the container with id f792af01042fe795300890784ffaad82a0a9ecb96436fa07a16c85de4458a0a3
Oct 30 13:02:56.059377 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:56.058613    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:02:56.059377 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:56.058687    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:02:56.059377 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:56.058717    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:02:56.111277 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:56.111253    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2bbc545-9zqtg"]
Oct 30 13:02:56.115190 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:02:56.115164    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80094f69_bff5_4ccd_8849_4af37f390d86.slice/crio-a996b6dfffdca38e9b4efcda39a85f52f332fe3c39b76fe4bfe42b416ffb5d1e WatchSource:0}: Error finding container a996b6dfffdca38e9b4efcda39a85f52f332fe3c39b76fe4bfe42b416ffb5d1e: Status 404 returned error can't find the container with id a996b6dfffdca38e9b4efcda39a85f52f332fe3c39b76fe4bfe42b416ffb5d1e
Oct 30 13:02:56.260024 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:02:56.259982    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d5354a_dc17_49c3_af9c_9b756ad02a00.slice/crio-aba5ad3dd5de99a627092c9ab63304d0daef614784b2d124c475b26638c70432 WatchSource:0}: Error finding container aba5ad3dd5de99a627092c9ab63304d0daef614784b2d124c475b26638c70432: Status 404 returned error can't find the container with id aba5ad3dd5de99a627092c9ab63304d0daef614784b2d124c475b26638c70432
Oct 30 13:02:56.260294 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:56.260257    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"]
Oct 30 13:02:56.262945 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:56.262905    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:02:56.263058 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:56.262988    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:02:56.263121 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:56.263054    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:02:56.299631 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:56.299594    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"]
Oct 30 13:02:56.303503 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:02:56.303455    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc07f44cd_a58d_44ce_ad1c_0ce81ca98f3a.slice/crio-b3fbb5e091e3cc537460ad9a52a8efbbcd0c6952d4b829a71ecd715609a4dbf2 WatchSource:0}: Error finding container b3fbb5e091e3cc537460ad9a52a8efbbcd0c6952d4b829a71ecd715609a4dbf2: Status 404 returned error can't find the container with id b3fbb5e091e3cc537460ad9a52a8efbbcd0c6952d4b829a71ecd715609a4dbf2
Oct 30 13:02:56.307370 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:56.307315    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:02:56.307611 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:56.307408    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:02:56.307611 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:56.307446    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:02:56.360782 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:56.360740    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq" event={"ID":"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a","Type":"ContainerStarted","Data":"b3fbb5e091e3cc537460ad9a52a8efbbcd0c6952d4b829a71ecd715609a4dbf2"}
Oct 30 13:02:56.362805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:56.362773    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8" event={"ID":"06d5354a-dc17-49c3-af9c-9b756ad02a00","Type":"ContainerStarted","Data":"aba5ad3dd5de99a627092c9ab63304d0daef614784b2d124c475b26638c70432"}
Oct 30 13:02:56.365172 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:56.365142    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2bbc545-9zqtg" event={"ID":"80094f69-bff5-4ccd-8849-4af37f390d86","Type":"ContainerStarted","Data":"a996b6dfffdca38e9b4efcda39a85f52f332fe3c39b76fe4bfe42b416ffb5d1e"}
Oct 30 13:02:56.367269 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:56.367246    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt" event={"ID":"f8eb30f8-aa71-4121-9542-4f65aff15344","Type":"ContainerStarted","Data":"31a2df703c050cfa05e2788849a61fd6ec015897fc2a483ad33723fdac46d76e"}
Oct 30 13:02:56.367380 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:56.367288    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt" event={"ID":"f8eb30f8-aa71-4121-9542-4f65aff15344","Type":"ContainerStarted","Data":"f792af01042fe795300890784ffaad82a0a9ecb96436fa07a16c85de4458a0a3"}
Oct 30 13:02:56.436855 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:56.436790    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt" podStartSLOduration=1.436769625 podStartE2EDuration="1.436769625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:56.431004801 +0000 UTC m=+1071.478138756" watchObservedRunningTime="2026-10-30 13:02:56.436769625 +0000 UTC m=+1071.483903579"
Oct 30 13:02:56.727455 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:56.727408    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:02:57.278141 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:57.278103    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:57.451099 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:57.447789    4544 generic.go:356] "Generic (PLEG): container finished" podID="80094f69-bff5-4ccd-8849-4af37f390d86" containerID="e4e479758007a580acb0d94e67e670901af2fab5ed7c83ab2c59ecb8985ee70d" exitCode=0
Oct 30 13:02:57.451099 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:57.448029    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2bbc545-9zqtg" event={"ID":"80094f69-bff5-4ccd-8849-4af37f390d86","Type":"ContainerDied","Data":"e4e479758007a580acb0d94e67e670901af2fab5ed7c83ab2c59ecb8985ee70d"}
Oct 30 13:02:57.502544 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:57.496818    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq" event={"ID":"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a","Type":"ContainerStarted","Data":"280156e3fa2812ed77f449e8367238112d9336d818b926a3b960626144c3198f"}
Oct 30 13:02:57.511567 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:57.511528    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8" event={"ID":"06d5354a-dc17-49c3-af9c-9b756ad02a00","Type":"ContainerStarted","Data":"9c4b8f133ff65b8b1f93ca72c3b8c55cb0cd97c69941ae3b7dd25da39f6ceaf8"}
Oct 30 13:02:57.604127 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:57.604067    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq" podStartSLOduration=2.604047984 podStartE2EDuration="2.604047984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:57.567287615 +0000 UTC m=+1072.614421569" watchObservedRunningTime="2026-10-30 13:02:57.604047984 +0000 UTC m=+1072.651181938"
Oct 30 13:02:57.688276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:57.676035    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8" podStartSLOduration=2.676013668 podStartE2EDuration="2.676013668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:57.633896817 +0000 UTC m=+1072.681030771" watchObservedRunningTime="2026-10-30 13:02:57.676013668 +0000 UTC m=+1072.723147622"
Oct 30 13:02:57.735698 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:57.735601    4544 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 13:02:57.735948 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:57.735921    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt" podUID="f8eb30f8-aa71-4121-9542-4f65aff15344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 13:02:57.952247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:57.951601    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:02:57.970322 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:57.956356    4544 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 13:02:57.970322 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:57.956426    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8" podUID="06d5354a-dc17-49c3-af9c-9b756ad02a00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 13:02:57.986316 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:57.983991    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:02:57.986316 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:57.984130    4544 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 13:02:57.986316 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:57.984175    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq" podUID="c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 13:02:58.730426 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:58.727659    4544 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 13:02:58.730426 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:58.727723    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt" podUID="f8eb30f8-aa71-4121-9542-4f65aff15344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 13:02:58.976476 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:58.959607    4544 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 13:02:58.976476 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:58.959666    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8" podUID="06d5354a-dc17-49c3-af9c-9b756ad02a00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 13:02:59.052245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:59.045725    4544 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 13:02:59.052245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:59.045790    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq" podUID="c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 13:02:59.108250 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:59.105382    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2bbc545-9zqtg"
Oct 30 13:02:59.211309 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:02:59.210675    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:02:59.277755 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:59.277720    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80094f69-bff5-4ccd-8849-4af37f390d86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80094f69-bff5-4ccd-8849-4af37f390d86-maas-api-openshift-ca\") pod \"80094f69-bff5-4ccd-8849-4af37f390d86\" (UID: \"80094f69-bff5-4ccd-8849-4af37f390d86\") "
Oct 30 13:02:59.278094 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:59.278071    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80094f69-bff5-4ccd-8849-4af37f390d86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80094f69-bff5-4ccd-8849-4af37f390d86-maas-api-xks-ca\") pod \"80094f69-bff5-4ccd-8849-4af37f390d86\" (UID: \"80094f69-bff5-4ccd-8849-4af37f390d86\") "
Oct 30 13:02:59.278478 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:59.278445    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80094f69-bff5-4ccd-8849-4af37f390d86-kube-api-access-fxddc\" (UniqueName: \"kubernetes.io/projected/80094f69-bff5-4ccd-8849-4af37f390d86-kube-api-access-fxddc\") pod \"80094f69-bff5-4ccd-8849-4af37f390d86\" (UID: \"80094f69-bff5-4ccd-8849-4af37f390d86\") "
Oct 30 13:02:59.299511 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:59.296428    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80094f69-bff5-4ccd-8849-4af37f390d86-maas-api-openshift-ca" pod "80094f69-bff5-4ccd-8849-4af37f390d86" (UID: "80094f69-bff5-4ccd-8849-4af37f390d86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:02:59.302348 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:59.301676    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80094f69-bff5-4ccd-8849-4af37f390d86-maas-api-xks-ca" pod "80094f69-bff5-4ccd-8849-4af37f390d86" (UID: "80094f69-bff5-4ccd-8849-4af37f390d86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:02:59.302348 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:59.301901    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80094f69-bff5-4ccd-8849-4af37f390d86-kube-api-access-fxddc" pod "80094f69-bff5-4ccd-8849-4af37f390d86" (UID: "80094f69-bff5-4ccd-8849-4af37f390d86"). InnerVolumeSpecName "kube-api-access-fxddc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:02:59.306484 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:59.306455    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:02:59.493331 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:59.489462    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80094f69-bff5-4ccd-8849-4af37f390d86-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:02:59.493331 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:59.489514    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80094f69-bff5-4ccd-8849-4af37f390d86-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:02:59.493331 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:59.489534    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxddc\" (UniqueName: \"kubernetes.io/projected/80094f69-bff5-4ccd-8849-4af37f390d86-kube-api-access-fxddc\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:02:59.697874 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:59.697835    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2bbc545-9zqtg"
Oct 30 13:02:59.698082 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:59.697891    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2bbc545-9zqtg" event={"ID":"80094f69-bff5-4ccd-8849-4af37f390d86","Type":"ContainerDied","Data":"a996b6dfffdca38e9b4efcda39a85f52f332fe3c39b76fe4bfe42b416ffb5d1e"}
Oct 30 13:02:59.698082 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:59.697950    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a996b6dfffdca38e9b4efcda39a85f52f332fe3c39b76fe4bfe42b416ffb5d1e"
Oct 30 13:02:59.731245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:59.728388    4544 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 13:02:59.731245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:59.728444    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt" podUID="f8eb30f8-aa71-4121-9542-4f65aff15344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 13:02:59.901491 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:02:59.901453    4544 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80094f69_bff5_4ccd_8849_4af37f390d86.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 13:02:59.947205 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:59.947165    4544 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 13:02:59.947509 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:59.947479    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8" podUID="06d5354a-dc17-49c3-af9c-9b756ad02a00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 13:02:59.978686 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:59.978648    4544 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 13:02:59.978974 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:02:59.978946    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq" podUID="c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 13:03:00.422175 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:00.422134    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:00.727725 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:00.727630    4544 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 13:03:00.727982 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:00.727946    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt" podUID="f8eb30f8-aa71-4121-9542-4f65aff15344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 13:03:00.951251 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:00.946958    4544 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 13:03:00.951251 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:00.947036    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8" podUID="06d5354a-dc17-49c3-af9c-9b756ad02a00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 13:03:00.979796 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:00.979606    4544 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 13:03:00.979796 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:00.979669    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq" podUID="c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 13:03:01.729697 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:01.729649    4544 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 13:03:01.729979 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:01.729947    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt" podUID="f8eb30f8-aa71-4121-9542-4f65aff15344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 13:03:01.988322 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:01.987730    4544 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 13:03:01.988322 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:01.987791    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq" podUID="c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 13:03:01.988900 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:01.988445    4544 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 13:03:01.988900 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:01.988481    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8" podUID="06d5354a-dc17-49c3-af9c-9b756ad02a00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 13:03:02.024299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:02.024138    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:02.759246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:02.742443    4544 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 13:03:02.759246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:02.742528    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt" podUID="f8eb30f8-aa71-4121-9542-4f65aff15344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 13:03:02.959088 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:02.958631    4544 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 13:03:02.959088 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:02.958695    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8" podUID="06d5354a-dc17-49c3-af9c-9b756ad02a00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 13:03:02.977123 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:02.977083    4544 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 13:03:02.977308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:02.977159    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq" podUID="c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 13:03:03.728012 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:03.727969    4544 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 13:03:03.728509 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:03.728040    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt" podUID="f8eb30f8-aa71-4121-9542-4f65aff15344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 13:03:03.947184 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:03.947141    4544 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 13:03:03.947409 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:03.947215    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8" podUID="06d5354a-dc17-49c3-af9c-9b756ad02a00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 13:03:03.977298 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:03.977254    4544 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 13:03:03.977498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:03.977319    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq" podUID="c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 13:03:04.544326 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:04.543821    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:04.766542 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:04.766510    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:03:04.768749 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:04.768205    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:03:04.769817 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:04.769793    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:03:05.073606 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:05.073506    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:03:05.074539 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:05.074514    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:03:05.077184 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:05.077130    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:03:05.178545 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:05.178509    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:03:05.178859 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:05.178828    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:03:05.179958 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:05.179937    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:03:05.477034 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:05.476946    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23283e03-wgfxf"]
Oct 30 13:03:05.477721 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:05.477698    4544 cpu_manager.go:414] "containerMap: removing container" podUID="80094f69-bff5-4ccd-8849-4af37f390d86" containerName="revoke-keys"
Oct 30 13:03:05.477721 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:05.477721    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80094f69-bff5-4ccd-8849-4af37f390d86" containerName="revoke-keys"
Oct 30 13:03:05.477903 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:05.477888    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="80094f69-bff5-4ccd-8849-4af37f390d86" containerName="revoke-keys"
Oct 30 13:03:05.483080 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:05.483058    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23283e03-wgfxf"
Oct 30 13:03:05.523141 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:05.523099    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23283e03-wgfxf"]
Oct 30 13:03:05.527459 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:05.527425    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:03:05.529183 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:05.529155    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:03:05.559435 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:05.559391    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca24aa78-b130-48be-9da0-f37d84a0fd6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23283e03-wgfxf\" (UID: \"ca24aa78-b130-48be-9da0-f37d84a0fd6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23283e03-wgfxf"
Oct 30 13:03:05.559604 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:05.559568    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zh4z\" (UniqueName: \"kubernetes.io/projected/ca24aa78-b130-48be-9da0-f37d84a0fd6d-kube-api-access-2zh4z\") pod \"maas-api-revoke-sub-23283e03-wgfxf\" (UID: \"ca24aa78-b130-48be-9da0-f37d84a0fd6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23283e03-wgfxf"
Oct 30 13:03:05.559777 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:05.559752    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca24aa78-b130-48be-9da0-f37d84a0fd6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23283e03-wgfxf\" (UID: \"ca24aa78-b130-48be-9da0-f37d84a0fd6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23283e03-wgfxf"
Oct 30 13:03:05.660617 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:05.660582    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zh4z\" (UniqueName: \"kubernetes.io/projected/ca24aa78-b130-48be-9da0-f37d84a0fd6d-kube-api-access-2zh4z\") pod \"maas-api-revoke-sub-23283e03-wgfxf\" (UID: \"ca24aa78-b130-48be-9da0-f37d84a0fd6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23283e03-wgfxf"
Oct 30 13:03:05.660772 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:05.660720    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca24aa78-b130-48be-9da0-f37d84a0fd6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23283e03-wgfxf\" (UID: \"ca24aa78-b130-48be-9da0-f37d84a0fd6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23283e03-wgfxf"
Oct 30 13:03:05.660772 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:05.660754    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca24aa78-b130-48be-9da0-f37d84a0fd6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23283e03-wgfxf\" (UID: \"ca24aa78-b130-48be-9da0-f37d84a0fd6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23283e03-wgfxf"
Oct 30 13:03:05.661618 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:05.661536    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca24aa78-b130-48be-9da0-f37d84a0fd6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23283e03-wgfxf\" (UID: \"ca24aa78-b130-48be-9da0-f37d84a0fd6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23283e03-wgfxf"
Oct 30 13:03:05.663486 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:05.663458    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca24aa78-b130-48be-9da0-f37d84a0fd6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23283e03-wgfxf\" (UID: \"ca24aa78-b130-48be-9da0-f37d84a0fd6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23283e03-wgfxf"
Oct 30 13:03:05.685446 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:05.685419    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zh4z\" (UniqueName: \"kubernetes.io/projected/ca24aa78-b130-48be-9da0-f37d84a0fd6d-kube-api-access-2zh4z\") pod \"maas-api-revoke-sub-23283e03-wgfxf\" (UID: \"ca24aa78-b130-48be-9da0-f37d84a0fd6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23283e03-wgfxf"
Oct 30 13:03:05.796846 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:05.796764    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23283e03-wgfxf"
Oct 30 13:03:06.114747 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:06.114716    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23283e03-wgfxf"]
Oct 30 13:03:06.116562 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:03:06.116536    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca24aa78_b130_48be_9da0_f37d84a0fd6d.slice/crio-b1d087435f38be4a3df7c2a374221ae0900017a672374343a094ebac1bc9559b WatchSource:0}: Error finding container b1d087435f38be4a3df7c2a374221ae0900017a672374343a094ebac1bc9559b: Status 404 returned error can't find the container with id b1d087435f38be4a3df7c2a374221ae0900017a672374343a094ebac1bc9559b
Oct 30 13:03:06.555023 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:06.554978    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:06.830852 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:06.830814    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23283e03-wgfxf" event={"ID":"ca24aa78-b130-48be-9da0-f37d84a0fd6d","Type":"ContainerStarted","Data":"1c46156a40d4e705e107e50846912fd51df24f5db6d5d537683c14674ca086b5"}
Oct 30 13:03:06.831289 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:06.830911    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23283e03-wgfxf" event={"ID":"ca24aa78-b130-48be-9da0-f37d84a0fd6d","Type":"ContainerStarted","Data":"b1d087435f38be4a3df7c2a374221ae0900017a672374343a094ebac1bc9559b"}
Oct 30 13:03:06.864889 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:06.864855    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32ab2b8d-9pf49"]
Oct 30 13:03:06.869576 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:06.869550    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ab2b8d-9pf49"
Oct 30 13:03:06.878909 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:06.878856    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23283e03-wgfxf" podStartSLOduration=1.8788389749999999 podStartE2EDuration="1.878838975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:06.876319322 +0000 UTC m=+1081.923453277" watchObservedRunningTime="2026-10-30 13:03:06.878838975 +0000 UTC m=+1081.925972929"
Oct 30 13:03:06.884880 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:06.884856    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6034433c-144c-49b8-a3c8-38d2511e6c18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32ab2b8d-9pf49\" (UID: \"6034433c-144c-49b8-a3c8-38d2511e6c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ab2b8d-9pf49"
Oct 30 13:03:06.885206 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:06.885176    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6034433c-144c-49b8-a3c8-38d2511e6c18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32ab2b8d-9pf49\" (UID: \"6034433c-144c-49b8-a3c8-38d2511e6c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ab2b8d-9pf49"
Oct 30 13:03:06.885328 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:06.885302    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9h7x\" (UniqueName: \"kubernetes.io/projected/6034433c-144c-49b8-a3c8-38d2511e6c18-kube-api-access-d9h7x\") pod \"maas-api-revoke-sub-32ab2b8d-9pf49\" (UID: \"6034433c-144c-49b8-a3c8-38d2511e6c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ab2b8d-9pf49"
Oct 30 13:03:06.892600 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:06.892571    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32ab2b8d-9pf49"]
Oct 30 13:03:06.986650 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:06.986601    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6034433c-144c-49b8-a3c8-38d2511e6c18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32ab2b8d-9pf49\" (UID: \"6034433c-144c-49b8-a3c8-38d2511e6c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ab2b8d-9pf49"
Oct 30 13:03:06.986832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:06.986664    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6034433c-144c-49b8-a3c8-38d2511e6c18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32ab2b8d-9pf49\" (UID: \"6034433c-144c-49b8-a3c8-38d2511e6c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ab2b8d-9pf49"
Oct 30 13:03:06.986832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:06.986709    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9h7x\" (UniqueName: \"kubernetes.io/projected/6034433c-144c-49b8-a3c8-38d2511e6c18-kube-api-access-d9h7x\") pod \"maas-api-revoke-sub-32ab2b8d-9pf49\" (UID: \"6034433c-144c-49b8-a3c8-38d2511e6c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ab2b8d-9pf49"
Oct 30 13:03:06.987354 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:06.987329    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6034433c-144c-49b8-a3c8-38d2511e6c18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32ab2b8d-9pf49\" (UID: \"6034433c-144c-49b8-a3c8-38d2511e6c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ab2b8d-9pf49"
Oct 30 13:03:06.989143 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:06.989117    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6034433c-144c-49b8-a3c8-38d2511e6c18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32ab2b8d-9pf49\" (UID: \"6034433c-144c-49b8-a3c8-38d2511e6c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ab2b8d-9pf49"
Oct 30 13:03:07.010109 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:07.010079    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9h7x\" (UniqueName: \"kubernetes.io/projected/6034433c-144c-49b8-a3c8-38d2511e6c18-kube-api-access-d9h7x\") pod \"maas-api-revoke-sub-32ab2b8d-9pf49\" (UID: \"6034433c-144c-49b8-a3c8-38d2511e6c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ab2b8d-9pf49"
Oct 30 13:03:07.186017 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:07.185983    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ab2b8d-9pf49"
Oct 30 13:03:07.492519 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:07.492492    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32ab2b8d-9pf49"]
Oct 30 13:03:07.494344 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:03:07.494317    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6034433c_144c_49b8_a3c8_38d2511e6c18.slice/crio-bada2c1b3d4ae325c4475afee5ceba68ce8e9c403fbb100f293683cdd0db1d5e WatchSource:0}: Error finding container bada2c1b3d4ae325c4475afee5ceba68ce8e9c403fbb100f293683cdd0db1d5e: Status 404 returned error can't find the container with id bada2c1b3d4ae325c4475afee5ceba68ce8e9c403fbb100f293683cdd0db1d5e
Oct 30 13:03:07.839880 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:07.839837    4544 generic.go:356] "Generic (PLEG): container finished" podID="ca24aa78-b130-48be-9da0-f37d84a0fd6d" containerID="1c46156a40d4e705e107e50846912fd51df24f5db6d5d537683c14674ca086b5" exitCode=0
Oct 30 13:03:07.840301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:07.839920    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23283e03-wgfxf" event={"ID":"ca24aa78-b130-48be-9da0-f37d84a0fd6d","Type":"ContainerDied","Data":"1c46156a40d4e705e107e50846912fd51df24f5db6d5d537683c14674ca086b5"}
Oct 30 13:03:07.841492 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:07.841472    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ab2b8d-9pf49" event={"ID":"6034433c-144c-49b8-a3c8-38d2511e6c18","Type":"ContainerStarted","Data":"343aca8b21c018e28baaff4af6c65f56b341d45c54c481893bfe28187cc6330f"}
Oct 30 13:03:07.841578 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:07.841508    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ab2b8d-9pf49" event={"ID":"6034433c-144c-49b8-a3c8-38d2511e6c18","Type":"ContainerStarted","Data":"bada2c1b3d4ae325c4475afee5ceba68ce8e9c403fbb100f293683cdd0db1d5e"}
Oct 30 13:03:07.925336 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:07.925266    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ab2b8d-9pf49" podStartSLOduration=1.9252451910000001 podStartE2EDuration="1.925245191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:07.922317275 +0000 UTC m=+1082.969451228" watchObservedRunningTime="2026-10-30 13:03:07.925245191 +0000 UTC m=+1082.972379143"
Oct 30 13:03:08.408439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:08.408405    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:08.850553 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:08.850520    4544 generic.go:356] "Generic (PLEG): container finished" podID="6034433c-144c-49b8-a3c8-38d2511e6c18" containerID="343aca8b21c018e28baaff4af6c65f56b341d45c54c481893bfe28187cc6330f" exitCode=0
Oct 30 13:03:08.850970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:08.850610    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ab2b8d-9pf49" event={"ID":"6034433c-144c-49b8-a3c8-38d2511e6c18","Type":"ContainerDied","Data":"343aca8b21c018e28baaff4af6c65f56b341d45c54c481893bfe28187cc6330f"}
Oct 30 13:03:09.030991 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.030960    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23283e03-wgfxf"
Oct 30 13:03:09.123638 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.123538    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca24aa78-b130-48be-9da0-f37d84a0fd6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca24aa78-b130-48be-9da0-f37d84a0fd6d-maas-api-openshift-ca\") pod \"ca24aa78-b130-48be-9da0-f37d84a0fd6d\" (UID: \"ca24aa78-b130-48be-9da0-f37d84a0fd6d\") "
Oct 30 13:03:09.123791 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.123697    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca24aa78-b130-48be-9da0-f37d84a0fd6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca24aa78-b130-48be-9da0-f37d84a0fd6d-maas-api-xks-ca\") pod \"ca24aa78-b130-48be-9da0-f37d84a0fd6d\" (UID: \"ca24aa78-b130-48be-9da0-f37d84a0fd6d\") "
Oct 30 13:03:09.123867 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.123837    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca24aa78-b130-48be-9da0-f37d84a0fd6d-kube-api-access-2zh4z\" (UniqueName: \"kubernetes.io/projected/ca24aa78-b130-48be-9da0-f37d84a0fd6d-kube-api-access-2zh4z\") pod \"ca24aa78-b130-48be-9da0-f37d84a0fd6d\" (UID: \"ca24aa78-b130-48be-9da0-f37d84a0fd6d\") "
Oct 30 13:03:09.124056 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.124022    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca24aa78-b130-48be-9da0-f37d84a0fd6d-maas-api-openshift-ca" pod "ca24aa78-b130-48be-9da0-f37d84a0fd6d" (UID: "ca24aa78-b130-48be-9da0-f37d84a0fd6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:03:09.125166 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.124658    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca24aa78-b130-48be-9da0-f37d84a0fd6d-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:03:09.128739 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.128663    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca24aa78-b130-48be-9da0-f37d84a0fd6d-maas-api-xks-ca" pod "ca24aa78-b130-48be-9da0-f37d84a0fd6d" (UID: "ca24aa78-b130-48be-9da0-f37d84a0fd6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:03:09.128882 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.128781    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca24aa78-b130-48be-9da0-f37d84a0fd6d-kube-api-access-2zh4z" pod "ca24aa78-b130-48be-9da0-f37d84a0fd6d" (UID: "ca24aa78-b130-48be-9da0-f37d84a0fd6d"). InnerVolumeSpecName "kube-api-access-2zh4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:03:09.234243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.232394    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zh4z\" (UniqueName: \"kubernetes.io/projected/ca24aa78-b130-48be-9da0-f37d84a0fd6d-kube-api-access-2zh4z\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:03:09.234243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.232440    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca24aa78-b130-48be-9da0-f37d84a0fd6d-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:03:09.792289 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.790839    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"]
Oct 30 13:03:09.792289 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.791992    4544 cpu_manager.go:414] "containerMap: removing container" podUID="ca24aa78-b130-48be-9da0-f37d84a0fd6d" containerName="revoke-keys"
Oct 30 13:03:09.792289 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.792014    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca24aa78-b130-48be-9da0-f37d84a0fd6d" containerName="revoke-keys"
Oct 30 13:03:09.792289 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.792242    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca24aa78-b130-48be-9da0-f37d84a0fd6d" containerName="revoke-keys"
Oct 30 13:03:09.805065 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.804932    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:03:09.857271 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.848497    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"]
Oct 30 13:03:09.880316 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.879235    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/317201b4-13b3-4341-a5a2-e4be2531f5f3-istio-podinfo\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:03:09.880316 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.879322    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/317201b4-13b3-4341-a5a2-e4be2531f5f3-istiod-ca-cert\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:03:09.880316 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.879366    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-istio-data\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:03:09.880316 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.879481    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-workload-certs\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:03:09.880316 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.879510    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/317201b4-13b3-4341-a5a2-e4be2531f5f3-istio-token\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:03:09.880316 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.879630    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-credential-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:03:09.880316 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.879703    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-istio-envoy\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:03:09.880316 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.879749    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-workload-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:03:09.880316 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.879776    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkxk4\" (UniqueName: \"kubernetes.io/projected/317201b4-13b3-4341-a5a2-e4be2531f5f3-kube-api-access-tkxk4\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:03:09.906284 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.889907    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23283e03-wgfxf"
Oct 30 13:03:09.906284 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.898463    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23283e03-wgfxf" event={"ID":"ca24aa78-b130-48be-9da0-f37d84a0fd6d","Type":"ContainerDied","Data":"b1d087435f38be4a3df7c2a374221ae0900017a672374343a094ebac1bc9559b"}
Oct 30 13:03:09.906284 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.898536    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1d087435f38be4a3df7c2a374221ae0900017a672374343a094ebac1bc9559b"
Oct 30 13:03:09.986245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.981891    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/317201b4-13b3-4341-a5a2-e4be2531f5f3-istio-podinfo\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:03:09.986245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.982500    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/317201b4-13b3-4341-a5a2-e4be2531f5f3-istiod-ca-cert\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:03:09.986245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.983200    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/317201b4-13b3-4341-a5a2-e4be2531f5f3-istiod-ca-cert\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:03:09.986245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.983376    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-istio-data\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:03:09.986245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.983793    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-istio-data\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:03:09.986245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.984003    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-workload-certs\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:03:09.986245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.984346    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-workload-certs\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:03:09.986245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.984400    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/317201b4-13b3-4341-a5a2-e4be2531f5f3-istio-token\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:03:09.986245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.985189    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-credential-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:03:09.986245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.985623    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-credential-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:03:09.986245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.985789    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-istio-envoy\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:03:09.987037 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.986649    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-workload-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:03:09.987037 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.986281    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-workload-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:03:09.987037 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.986746    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkxk4\" (UniqueName: \"kubernetes.io/projected/317201b4-13b3-4341-a5a2-e4be2531f5f3-kube-api-access-tkxk4\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:03:10.015435 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.996931    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-istio-envoy\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:03:10.015435 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:09.998664    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/317201b4-13b3-4341-a5a2-e4be2531f5f3-istio-podinfo\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:03:10.015435 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:10.014126    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/317201b4-13b3-4341-a5a2-e4be2531f5f3-istio-token\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:03:10.015435 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:10.014128    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkxk4\" (UniqueName: \"kubernetes.io/projected/317201b4-13b3-4341-a5a2-e4be2531f5f3-kube-api-access-tkxk4\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:03:10.302606 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:10.283473    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:03:10.848182 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:10.828339    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ab2b8d-9pf49"
Oct 30 13:03:10.965292 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:10.954742    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6034433c-144c-49b8-a3c8-38d2511e6c18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6034433c-144c-49b8-a3c8-38d2511e6c18-maas-api-xks-ca\") pod \"6034433c-144c-49b8-a3c8-38d2511e6c18\" (UID: \"6034433c-144c-49b8-a3c8-38d2511e6c18\") "
Oct 30 13:03:10.965292 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:10.954831    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6034433c-144c-49b8-a3c8-38d2511e6c18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6034433c-144c-49b8-a3c8-38d2511e6c18-maas-api-openshift-ca\") pod \"6034433c-144c-49b8-a3c8-38d2511e6c18\" (UID: \"6034433c-144c-49b8-a3c8-38d2511e6c18\") "
Oct 30 13:03:10.965292 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:10.954901    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6034433c-144c-49b8-a3c8-38d2511e6c18-kube-api-access-d9h7x\" (UniqueName: \"kubernetes.io/projected/6034433c-144c-49b8-a3c8-38d2511e6c18-kube-api-access-d9h7x\") pod \"6034433c-144c-49b8-a3c8-38d2511e6c18\" (UID: \"6034433c-144c-49b8-a3c8-38d2511e6c18\") "
Oct 30 13:03:11.011587 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:11.011532    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6034433c-144c-49b8-a3c8-38d2511e6c18-maas-api-openshift-ca" pod "6034433c-144c-49b8-a3c8-38d2511e6c18" (UID: "6034433c-144c-49b8-a3c8-38d2511e6c18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:03:11.288360 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:11.285144    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6034433c-144c-49b8-a3c8-38d2511e6c18-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:03:11.390438 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:11.390368    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:11.618520 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:11.548906    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6034433c-144c-49b8-a3c8-38d2511e6c18-maas-api-xks-ca" pod "6034433c-144c-49b8-a3c8-38d2511e6c18" (UID: "6034433c-144c-49b8-a3c8-38d2511e6c18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:03:11.618520 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:11.604286    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:03:11.618520 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:11.605272    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6034433c-144c-49b8-a3c8-38d2511e6c18-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:03:11.636536 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:11.636486    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ab2b8d-9pf49" event={"ID":"6034433c-144c-49b8-a3c8-38d2511e6c18","Type":"ContainerDied","Data":"bada2c1b3d4ae325c4475afee5ceba68ce8e9c403fbb100f293683cdd0db1d5e"}
Oct 30 13:03:11.636947 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:11.636926    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bada2c1b3d4ae325c4475afee5ceba68ce8e9c403fbb100f293683cdd0db1d5e"
Oct 30 13:03:11.637079 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:11.636857    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ab2b8d-9pf49"
Oct 30 13:03:11.671673 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:11.670509    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6034433c-144c-49b8-a3c8-38d2511e6c18-kube-api-access-d9h7x" pod "6034433c-144c-49b8-a3c8-38d2511e6c18" (UID: "6034433c-144c-49b8-a3c8-38d2511e6c18"). InnerVolumeSpecName "kube-api-access-d9h7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:03:11.725253 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:11.716246    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9h7x\" (UniqueName: \"kubernetes.io/projected/6034433c-144c-49b8-a3c8-38d2511e6c18-kube-api-access-d9h7x\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:03:11.892312 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:11.890364    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:12.246306 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:12.245373    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:12.332744 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:12.330874    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:12.499314 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:12.491728    4544 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded"
Oct 30 13:03:12.525247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:12.501589    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:12.541245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:12.525911    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:03:12.729267 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:12.612928    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:12.892256 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:12.881849    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 13:03:12.892256 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:12.711339    4544 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:03:12.892256 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:12.889683    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v" podUID="1e40797f-db7b-4502-b06c-bb8b419e52b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:12.906248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:12.893337    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:12.906248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:12.896357    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:03:13.280295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:13.273014    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 13:03:13.550427 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:13.550318    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:13.682464 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:13.675023    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:03:13.682464 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:13.675130    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:03:13.682464 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:13.675179    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:03:13.696939 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:13.696902    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:03:13.723412 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:13.712642    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"]
Oct 30 13:03:13.746841 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:13.746805    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:03:13.890031 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:13.889983    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:03:13.924248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:13.919866    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:03:13.992839 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:13.992785    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:14.315264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:14.310348    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7" event={"ID":"317201b4-13b3-4341-a5a2-e4be2531f5f3","Type":"ContainerStarted","Data":"33ead284a0a4a18514f432c2ad2d06ffd11bcf7563a31598d4b988e0481ed783"}
Oct 30 13:03:14.367950 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:14.366261    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:03:14.433015 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:14.432981    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 13:03:15.201489 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:15.201453    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:15.336486 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:15.336448    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7" event={"ID":"317201b4-13b3-4341-a5a2-e4be2531f5f3","Type":"ContainerStarted","Data":"93bd1fd2fea5b74865f234e18c25a3615303d6b9db471c2efc2900b3436fe170"}
Oct 30 13:03:15.404683 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:15.404624    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7" podStartSLOduration=6.404608601 podStartE2EDuration="6.404608601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:15.392248659 +0000 UTC m=+1090.439382606" watchObservedRunningTime="2026-10-30 13:03:15.404608601 +0000 UTC m=+1090.451742553"
Oct 30 13:03:16.284627 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:16.284590    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:03:16.286339 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:16.286315    4544 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 13:03:16.286398 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:16.286370    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7" podUID="317201b4-13b3-4341-a5a2-e4be2531f5f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 13:03:17.192297 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:17.192255    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:17.287989 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:17.287952    4544 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 13:03:17.288645 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:17.288613    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7" podUID="317201b4-13b3-4341-a5a2-e4be2531f5f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 13:03:18.284512 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:18.284476    4544 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 13:03:18.284942 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:18.284533    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7" podUID="317201b4-13b3-4341-a5a2-e4be2531f5f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 13:03:19.095654 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:19.095613    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:19.285024 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:19.284985    4544 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 13:03:19.285457 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:19.285045    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7" podUID="317201b4-13b3-4341-a5a2-e4be2531f5f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 13:03:20.284653 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:20.284609    4544 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 13:03:20.284845 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:20.284669    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7" podUID="317201b4-13b3-4341-a5a2-e4be2531f5f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 13:03:20.878524 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:20.878492    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:21.284508 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:21.284410    4544 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 13:03:21.284508 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:21.284482    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7" podUID="317201b4-13b3-4341-a5a2-e4be2531f5f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 13:03:22.288888 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:22.288855    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:03:22.289369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:22.289347    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:03:22.290366 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:22.290249    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:03:22.860958 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:22.860918    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:24.672029 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:24.671987    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:24.818243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:24.818121    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"]
Oct 30 13:03:24.826058 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:24.819164    4544 cpu_manager.go:414] "containerMap: removing container" podUID="6034433c-144c-49b8-a3c8-38d2511e6c18" containerName="revoke-keys"
Oct 30 13:03:24.826058 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:24.819186    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6034433c-144c-49b8-a3c8-38d2511e6c18" containerName="revoke-keys"
Oct 30 13:03:24.826058 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:24.819367    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="6034433c-144c-49b8-a3c8-38d2511e6c18" containerName="revoke-keys"
Oct 30 13:03:24.843820 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:24.842440    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:03:24.850079 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:24.850043    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"]
Oct 30 13:03:24.935659 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:24.935553    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istio-envoy\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:03:24.935659 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:24.935628    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istio-data\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:03:24.935860 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:24.935685    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-workload-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:03:24.935860 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:24.935756    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istiod-ca-cert\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:03:24.935860 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:24.935820    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istio-podinfo\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:03:24.935967 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:24.935869    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-credential-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:03:24.936002 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:24.935974    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm87t\" (UniqueName: \"kubernetes.io/projected/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-kube-api-access-gm87t\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:03:24.936035 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:24.936028    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-workload-certs\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:03:24.936105 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:24.936091    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istio-token\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:03:25.037384 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.037339    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istio-podinfo\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:03:25.037590 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.037404    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-credential-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:03:25.037590 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.037493    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm87t\" (UniqueName: \"kubernetes.io/projected/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-kube-api-access-gm87t\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:03:25.037590 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.037561    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-workload-certs\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:03:25.037767 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.037614    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istio-token\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:03:25.037767 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.037699    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istio-envoy\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:03:25.037767 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.037744    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istio-data\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:03:25.037956 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.037801    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-workload-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:03:25.037956 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.037828    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istiod-ca-cert\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:03:25.037956 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.037914    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-credential-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:03:25.037956 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.037931    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-workload-certs\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:03:25.038178 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.038095    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istio-data\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:03:25.038440 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.038419    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-workload-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:03:25.038567 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.038548    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istiod-ca-cert\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:03:25.039966 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.039944    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istio-envoy\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:03:25.040028 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.039985    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istio-podinfo\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:03:25.044643 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.044613    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:25.060675 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.060643    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istio-token\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:03:25.060845 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.060736    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm87t\" (UniqueName: \"kubernetes.io/projected/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-kube-api-access-gm87t\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:03:25.169269 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.169209    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:03:25.183628 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.183600    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd085d5d-jpfpz"]
Oct 30 13:03:25.189354 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.189264    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd085d5d-jpfpz"
Oct 30 13:03:25.205809 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.205764    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:03:25.208948 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.208913    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd085d5d-jpfpz"]
Oct 30 13:03:25.210686 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.210618    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:03:25.240661 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.240631    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a254746-b5be-4d7c-b6ec-3c749effbebf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd085d5d-jpfpz\" (UID: \"3a254746-b5be-4d7c-b6ec-3c749effbebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd085d5d-jpfpz"
Oct 30 13:03:25.240830 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.240779    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4wqq\" (UniqueName: \"kubernetes.io/projected/3a254746-b5be-4d7c-b6ec-3c749effbebf-kube-api-access-j4wqq\") pod \"maas-api-revoke-sub-bd085d5d-jpfpz\" (UID: \"3a254746-b5be-4d7c-b6ec-3c749effbebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd085d5d-jpfpz"
Oct 30 13:03:25.240914 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.240889    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a254746-b5be-4d7c-b6ec-3c749effbebf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd085d5d-jpfpz\" (UID: \"3a254746-b5be-4d7c-b6ec-3c749effbebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd085d5d-jpfpz"
Oct 30 13:03:25.352244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.349245    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a254746-b5be-4d7c-b6ec-3c749effbebf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd085d5d-jpfpz\" (UID: \"3a254746-b5be-4d7c-b6ec-3c749effbebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd085d5d-jpfpz"
Oct 30 13:03:25.352244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.349396    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4wqq\" (UniqueName: \"kubernetes.io/projected/3a254746-b5be-4d7c-b6ec-3c749effbebf-kube-api-access-j4wqq\") pod \"maas-api-revoke-sub-bd085d5d-jpfpz\" (UID: \"3a254746-b5be-4d7c-b6ec-3c749effbebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd085d5d-jpfpz"
Oct 30 13:03:25.352244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.349490    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a254746-b5be-4d7c-b6ec-3c749effbebf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd085d5d-jpfpz\" (UID: \"3a254746-b5be-4d7c-b6ec-3c749effbebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd085d5d-jpfpz"
Oct 30 13:03:25.352244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.350766    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a254746-b5be-4d7c-b6ec-3c749effbebf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd085d5d-jpfpz\" (UID: \"3a254746-b5be-4d7c-b6ec-3c749effbebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd085d5d-jpfpz"
Oct 30 13:03:25.357245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.355175    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a254746-b5be-4d7c-b6ec-3c749effbebf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd085d5d-jpfpz\" (UID: \"3a254746-b5be-4d7c-b6ec-3c749effbebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd085d5d-jpfpz"
Oct 30 13:03:25.532313 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.521534    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4wqq\" (UniqueName: \"kubernetes.io/projected/3a254746-b5be-4d7c-b6ec-3c749effbebf-kube-api-access-j4wqq\") pod \"maas-api-revoke-sub-bd085d5d-jpfpz\" (UID: \"3a254746-b5be-4d7c-b6ec-3c749effbebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd085d5d-jpfpz"
Oct 30 13:03:25.553936 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.553886    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd085d5d-jpfpz"
Oct 30 13:03:25.579087 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.579050    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"]
Oct 30 13:03:25.587596 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.587568    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:03:25.610214 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.610187    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"]
Oct 30 13:03:25.632951 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:03:25.632898    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa2c83b0_34e5_4b64_ba1c_b38f86978e24.slice/crio-9daf87946886a2c1fd498ba76060a844e4186aa1efc42de27cf0a0aff74d2ce9 WatchSource:0}: Error finding container 9daf87946886a2c1fd498ba76060a844e4186aa1efc42de27cf0a0aff74d2ce9: Status 404 returned error can't find the container with id 9daf87946886a2c1fd498ba76060a844e4186aa1efc42de27cf0a0aff74d2ce9
Oct 30 13:03:25.637156 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.637067    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"]
Oct 30 13:03:25.639981 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.639453    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:03:25.639981 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.639542    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:03:25.639981 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.639586    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:03:25.756569 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.756518    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istio-envoy\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:03:25.756569 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.756572    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istio-podinfo\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:03:25.757128 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.756610    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmwng\" (UniqueName: \"kubernetes.io/projected/df98822e-ff7b-4c1d-ad2e-1e3b88523780-kube-api-access-wmwng\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:03:25.757128 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.756722    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istio-data\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:03:25.757128 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.756785    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-workload-certs\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:03:25.757128 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.756820    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-credential-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:03:25.757128 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.756855    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-workload-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:03:25.757128 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.756949    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istiod-ca-cert\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:03:25.757128 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.757064    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istio-token\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:03:25.860880 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.860832    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istio-envoy\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:03:25.861098 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.860914    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istio-podinfo\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:03:25.861098 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.860978    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmwng\" (UniqueName: \"kubernetes.io/projected/df98822e-ff7b-4c1d-ad2e-1e3b88523780-kube-api-access-wmwng\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:03:25.861266 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.861127    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istio-data\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:03:25.861266 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.861212    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-workload-certs\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:03:25.861379 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.861271    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-credential-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:03:25.861379 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.861325    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-workload-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:03:25.861488 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.861444    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istiod-ca-cert\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:03:25.861629 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.861597    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istio-token\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:03:25.866249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.865703    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istio-envoy\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:03:25.870250 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.869296    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istio-podinfo\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:03:25.870250 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.869675    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-workload-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:03:25.870250 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.869967    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istio-data\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:03:25.870250 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.870072    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-credential-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:03:25.878253 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.870635    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istiod-ca-cert\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:03:25.878253 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.872593    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-workload-certs\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:03:25.890248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.888512    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istio-token\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:03:25.896270 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.896197    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmwng\" (UniqueName: \"kubernetes.io/projected/df98822e-ff7b-4c1d-ad2e-1e3b88523780-kube-api-access-wmwng\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:03:25.925116 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:25.924831    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:03:26.030291 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:26.020412    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd085d5d-jpfpz"]
Oct 30 13:03:26.521931 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:26.521891    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg" event={"ID":"fa2c83b0-34e5-4b64-ba1c-b38f86978e24","Type":"ContainerStarted","Data":"5581df816922fa36c10d2cc3d7d0eb62a002e34ac63bf23f81e187283e043ced"}
Oct 30 13:03:26.522266 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:26.522215    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg" event={"ID":"fa2c83b0-34e5-4b64-ba1c-b38f86978e24","Type":"ContainerStarted","Data":"9daf87946886a2c1fd498ba76060a844e4186aa1efc42de27cf0a0aff74d2ce9"}
Oct 30 13:03:26.526396 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:26.526207    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd085d5d-jpfpz" event={"ID":"3a254746-b5be-4d7c-b6ec-3c749effbebf","Type":"ContainerStarted","Data":"8b8e0f0c0889ef9fad0292e0939c373a51917b7b3d8a16d975f36a8fc7892217"}
Oct 30 13:03:26.589142 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:26.589062    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg" podStartSLOduration=2.589037733 podStartE2EDuration="2.589037733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:26.568900633 +0000 UTC m=+1101.616034589" watchObservedRunningTime="2026-10-30 13:03:26.589037733 +0000 UTC m=+1101.636171690"
Oct 30 13:03:26.591469 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:26.591400    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:03:26.591652 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:26.591577    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:03:26.591727 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:26.591668    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:03:26.599497 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:26.599423    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"]
Oct 30 13:03:27.185246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:27.179321    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:03:27.185246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:27.179453    4544 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 13:03:27.185246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:27.179498    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg" podUID="fa2c83b0-34e5-4b64-ba1c-b38f86978e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 13:03:27.443245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:27.438788    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:27.742332 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:27.723807    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph" event={"ID":"df98822e-ff7b-4c1d-ad2e-1e3b88523780","Type":"ContainerStarted","Data":"b69d7d866f74dac1bace741db7894472244bbeecd92a111c60b6133cfcda4407"}
Oct 30 13:03:27.742332 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:27.723883    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph" event={"ID":"df98822e-ff7b-4c1d-ad2e-1e3b88523780","Type":"ContainerStarted","Data":"247b8d2da5d18724d2b531df344b95e8399c982754a7832183f8a8e45214d0be"}
Oct 30 13:03:27.763758 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:27.763658    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd085d5d-jpfpz" event={"ID":"3a254746-b5be-4d7c-b6ec-3c749effbebf","Type":"ContainerStarted","Data":"0c96b93ee14f69dbba7c4559fb3e944e0d12943c98d9a9a0b6c218dc5753dfc5"}
Oct 30 13:03:27.806575 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:27.806506    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph" podStartSLOduration=2.806481255 podStartE2EDuration="2.806481255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:27.781773314 +0000 UTC m=+1102.828907268" watchObservedRunningTime="2026-10-30 13:03:27.806481255 +0000 UTC m=+1102.853615210"
Oct 30 13:03:27.964414 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:27.964301    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:03:27.968731 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:27.968370    4544 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 13:03:27.968731 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:27.968440    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph" podUID="df98822e-ff7b-4c1d-ad2e-1e3b88523780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 13:03:28.179588 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:28.179540    4544 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 13:03:28.179798 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:28.179610    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg" podUID="fa2c83b0-34e5-4b64-ba1c-b38f86978e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 13:03:28.773946 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:28.773913    4544 generic.go:356] "Generic (PLEG): container finished" podID="3a254746-b5be-4d7c-b6ec-3c749effbebf" containerID="0c96b93ee14f69dbba7c4559fb3e944e0d12943c98d9a9a0b6c218dc5753dfc5" exitCode=0
Oct 30 13:03:28.774377 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:28.774335    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd085d5d-jpfpz" event={"ID":"3a254746-b5be-4d7c-b6ec-3c749effbebf","Type":"ContainerDied","Data":"0c96b93ee14f69dbba7c4559fb3e944e0d12943c98d9a9a0b6c218dc5753dfc5"}
Oct 30 13:03:28.794912 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:28.794876    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6e33f02-lm5p7"]
Oct 30 13:03:28.799499 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:28.799471    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6e33f02-lm5p7"
Oct 30 13:03:28.878506 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:28.878463    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6e33f02-lm5p7"]
Oct 30 13:03:28.971890 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:28.968759    4544 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 13:03:28.971890 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:28.968821    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph" podUID="df98822e-ff7b-4c1d-ad2e-1e3b88523780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 13:03:28.984761 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:28.981623    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56f7ecc5-d9a7-4f5c-a572-338b114f3e0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6e33f02-lm5p7\" (UID: \"56f7ecc5-d9a7-4f5c-a572-338b114f3e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6e33f02-lm5p7"
Oct 30 13:03:28.984761 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:28.981879    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56f7ecc5-d9a7-4f5c-a572-338b114f3e0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6e33f02-lm5p7\" (UID: \"56f7ecc5-d9a7-4f5c-a572-338b114f3e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6e33f02-lm5p7"
Oct 30 13:03:28.984761 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:28.982260    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p64df\" (UniqueName: \"kubernetes.io/projected/56f7ecc5-d9a7-4f5c-a572-338b114f3e0f-kube-api-access-p64df\") pod \"maas-api-revoke-sub-c6e33f02-lm5p7\" (UID: \"56f7ecc5-d9a7-4f5c-a572-338b114f3e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6e33f02-lm5p7"
Oct 30 13:03:29.078536 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:03:29.078498    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:03:29.084284 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:29.084248    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p64df\" (UniqueName: \"kubernetes.io/projected/56f7ecc5-d9a7-4f5c-a572-338b114f3e0f-kube-api-access-p64df\") pod \"maas-api-revoke-sub-c6e33f02-lm5p7\" (UID: \"56f7ecc5-d9a7-4f5c-a572-338b114f3e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6e33f02-lm5p7"
Oct 30 13:03:29.084517 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:29.084502    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56f7ecc5-d9a7-4f5c-a572-338b114f3e0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6e33f02-lm5p7\" (UID: \"56f7ecc5-d9a7-4f5c-a572-338b114f3e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6e33f02-lm5p7"
Oct 30 13:03:29.084717 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:29.084703    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56f7ecc5-d9a7-4f5c-a572-338b114f3e0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6e33f02-lm5p7\" (UID: \"56f7ecc5-d9a7-4f5c-a572-338b114f3e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6e33f02-lm5p7"
Oct 30 13:03:29.087269 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:29.085297    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56f7ecc5-d9a7-4f5c-a572-338b114f3e0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6e33f02-lm5p7\" (UID: \"56f7ecc5-d9a7-4f5c-a572-338b114f3e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6e33f02-lm5p7"
Oct 30 13:03:29.093359 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:29.093308    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56f7ecc5-d9a7-4f5c-a572-338b114f3e0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6e33f02-lm5p7\" (UID: \"56f7ecc5-d9a7-4f5c-a572-338b114f3e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6e33f02-lm5p7"
Oct 30 13:03:29.171258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:29.170719    4544 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 13:03:29.171258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:29.170776    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg" podUID="fa2c83b0-34e5-4b64-ba1c-b38f86978e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 13:03:29.171258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:29.171007    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p64df\" (UniqueName: \"kubernetes.io/projected/56f7ecc5-d9a7-4f5c-a572-338b114f3e0f-kube-api-access-p64df\") pod \"maas-api-revoke-sub-c6e33f02-lm5p7\" (UID: \"56f7ecc5-d9a7-4f5c-a572-338b114f3e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6e33f02-lm5p7"
Oct 30 13:03:29.178455 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:29.178427    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd085d5d-jpfpz"
Oct 30 13:03:29.178716 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:29.178705    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6e33f02-lm5p7"
Oct 30 13:03:29.185658 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:29.185624    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a254746-b5be-4d7c-b6ec-3c749effbebf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a254746-b5be-4d7c-b6ec-3c749effbebf-maas-api-openshift-ca\") pod \"3a254746-b5be-4d7c-b6ec-3c749effbebf\" (UID: \"3a254746-b5be-4d7c-b6ec-3c749effbebf\") "
Oct 30 13:03:29.185805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:29.185702    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a254746-b5be-4d7c-b6ec-3c749effbebf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a254746-b5be-4d7c-b6ec-3c749effbebf-maas-api-xks-ca\") pod \"3a254746-b5be-4d7c-b6ec-3c749effbebf\" (UID: \"3a254746-b5be-4d7c-b6ec-3c749effbebf\") "
Oct 30 13:03:29.185805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:29.185755    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a254746-b5be-4d7c-b6ec-3c749effbebf-kube-api-access-j4wqq\" (UniqueName: \"kubernetes.io/projected/3a254746-b5be-4d7c-b6ec-3c749effbebf-kube-api-access-j4wqq\") pod \"3a254746-b5be-4d7c-b6ec-3c749effbebf\" (UID: \"3a254746-b5be-4d7c-b6ec-3c749effbebf\") "
Oct 30 13:03:29.186632 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:29.186602    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a254746-b5be-4d7c-b6ec-3c749effbebf-maas-api-openshift-ca" pod "3a254746-b5be-4d7c-b6ec-3c749effbebf" (UID: "3a254746-b5be-4d7c-b6ec-3c749effbebf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:03:29.188530 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:29.188495    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a254746-b5be-4d7c-b6ec-3c749effbebf-kube-api-access-j4wqq" pod "3a254746-b5be-4d7c-b6ec-3c749effbebf" (UID: "3a254746-b5be-4d7c-b6ec-3c749effbebf"). InnerVolumeSpecName "kube-api-access-j4wqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:03:29.188648 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:29.188575    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a254746-b5be-4d7c-b6ec-3c749effbebf-maas-api-xks-ca" pod "3a254746-b5be-4d7c-b6ec-3c749effbebf" (UID: "3a254746-b5be-4d7c-b6ec-3c749effbebf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:03:29.291149 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:29.291103    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a254746-b5be-4d7c-b6ec-3c749effbebf-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:03:29.291149 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:29.291152    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4wqq\" (UniqueName: \"kubernetes.io/projected/3a254746-b5be-4d7c-b6ec-3c749effbebf-kube-api-access-j4wqq\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:03:29.291430 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:29.291178    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a254746-b5be-4d7c-b6ec-3c749effbebf-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:03:29.572661 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:29.572591    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6e33f02-lm5p7"]
Oct 30 13:03:29.649246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:29.648815    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:29.841256 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:29.840756    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6e33f02-lm5p7" event={"ID":"56f7ecc5-d9a7-4f5c-a572-338b114f3e0f","Type":"ContainerStarted","Data":"2ac3bd7386d034f6e456f630441f96885a913309ceac6c72ba4d7747a16c1b24"}
Oct 30 13:03:29.845276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:29.844367    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd085d5d-jpfpz"
Oct 30 13:03:29.873491 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:29.852195    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd085d5d-jpfpz" event={"ID":"3a254746-b5be-4d7c-b6ec-3c749effbebf","Type":"ContainerDied","Data":"8b8e0f0c0889ef9fad0292e0939c373a51917b7b3d8a16d975f36a8fc7892217"}
Oct 30 13:03:29.873491 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:29.852306    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b8e0f0c0889ef9fad0292e0939c373a51917b7b3d8a16d975f36a8fc7892217"
Oct 30 13:03:29.944601 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:29.944563    4544 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 13:03:29.944909 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:29.944881    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph" podUID="df98822e-ff7b-4c1d-ad2e-1e3b88523780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 13:03:30.170001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:30.169901    4544 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 13:03:30.170001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:30.169969    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg" podUID="fa2c83b0-34e5-4b64-ba1c-b38f86978e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 13:03:30.869762 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:30.869718    4544 generic.go:356] "Generic (PLEG): container finished" podID="56f7ecc5-d9a7-4f5c-a572-338b114f3e0f" containerID="b77d15e8a6c4f73db60579cc43f469153211bb44e822890e413285d47a5030da" exitCode=0
Oct 30 13:03:30.870201 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:30.869784    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6e33f02-lm5p7" event={"ID":"56f7ecc5-d9a7-4f5c-a572-338b114f3e0f","Type":"ContainerDied","Data":"b77d15e8a6c4f73db60579cc43f469153211bb44e822890e413285d47a5030da"}
Oct 30 13:03:30.953759 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:30.953656    4544 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 13:03:30.954020 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:30.953989    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph" podUID="df98822e-ff7b-4c1d-ad2e-1e3b88523780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 13:03:31.169638 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:31.169594    4544 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 13:03:31.169823 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:31.169653    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg" podUID="fa2c83b0-34e5-4b64-ba1c-b38f86978e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 13:03:31.735798 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:31.735759    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:31.947761 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:31.947718    4544 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 13:03:31.948414 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:31.948376    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph" podUID="df98822e-ff7b-4c1d-ad2e-1e3b88523780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 13:03:32.173487 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:32.173449    4544 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 13:03:32.174683 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:32.174638    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg" podUID="fa2c83b0-34e5-4b64-ba1c-b38f86978e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 13:03:32.270043 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:32.270009    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6e33f02-lm5p7"
Oct 30 13:03:32.335303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:32.329396    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56f7ecc5-d9a7-4f5c-a572-338b114f3e0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56f7ecc5-d9a7-4f5c-a572-338b114f3e0f-maas-api-openshift-ca\") pod \"56f7ecc5-d9a7-4f5c-a572-338b114f3e0f\" (UID: \"56f7ecc5-d9a7-4f5c-a572-338b114f3e0f\") "
Oct 30 13:03:32.335303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:32.329486    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56f7ecc5-d9a7-4f5c-a572-338b114f3e0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56f7ecc5-d9a7-4f5c-a572-338b114f3e0f-maas-api-xks-ca\") pod \"56f7ecc5-d9a7-4f5c-a572-338b114f3e0f\" (UID: \"56f7ecc5-d9a7-4f5c-a572-338b114f3e0f\") "
Oct 30 13:03:32.335303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:32.329606    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56f7ecc5-d9a7-4f5c-a572-338b114f3e0f-kube-api-access-p64df\" (UniqueName: \"kubernetes.io/projected/56f7ecc5-d9a7-4f5c-a572-338b114f3e0f-kube-api-access-p64df\") pod \"56f7ecc5-d9a7-4f5c-a572-338b114f3e0f\" (UID: \"56f7ecc5-d9a7-4f5c-a572-338b114f3e0f\") "
Oct 30 13:03:32.335303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:32.331030    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56f7ecc5-d9a7-4f5c-a572-338b114f3e0f-maas-api-openshift-ca" pod "56f7ecc5-d9a7-4f5c-a572-338b114f3e0f" (UID: "56f7ecc5-d9a7-4f5c-a572-338b114f3e0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:03:32.362305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:32.349469    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56f7ecc5-d9a7-4f5c-a572-338b114f3e0f-maas-api-xks-ca" pod "56f7ecc5-d9a7-4f5c-a572-338b114f3e0f" (UID: "56f7ecc5-d9a7-4f5c-a572-338b114f3e0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:03:32.375484 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:32.375449    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56f7ecc5-d9a7-4f5c-a572-338b114f3e0f-kube-api-access-p64df" pod "56f7ecc5-d9a7-4f5c-a572-338b114f3e0f" (UID: "56f7ecc5-d9a7-4f5c-a572-338b114f3e0f"). InnerVolumeSpecName "kube-api-access-p64df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:03:32.441248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:32.437500    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56f7ecc5-d9a7-4f5c-a572-338b114f3e0f-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:03:32.462491 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:32.462314    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56f7ecc5-d9a7-4f5c-a572-338b114f3e0f-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:03:32.462491 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:32.462357    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p64df\" (UniqueName: \"kubernetes.io/projected/56f7ecc5-d9a7-4f5c-a572-338b114f3e0f-kube-api-access-p64df\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:03:32.903927 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:32.903894    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6e33f02-lm5p7"
Oct 30 13:03:32.904097 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:32.903919    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6e33f02-lm5p7" event={"ID":"56f7ecc5-d9a7-4f5c-a572-338b114f3e0f","Type":"ContainerDied","Data":"2ac3bd7386d034f6e456f630441f96885a913309ceac6c72ba4d7747a16c1b24"}
Oct 30 13:03:32.904097 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:32.903980    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ac3bd7386d034f6e456f630441f96885a913309ceac6c72ba4d7747a16c1b24"
Oct 30 13:03:32.945738 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:32.945699    4544 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 13:03:32.945940 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:32.945774    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph" podUID="df98822e-ff7b-4c1d-ad2e-1e3b88523780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 13:03:33.170184 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:33.170082    4544 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 13:03:33.170184 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:33.170145    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg" podUID="fa2c83b0-34e5-4b64-ba1c-b38f86978e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 13:03:33.622950 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:33.622911    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:33.943322 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:33.943163    4544 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 13:03:33.943322 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:33.943246    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph" podUID="df98822e-ff7b-4c1d-ad2e-1e3b88523780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 13:03:34.413572 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:34.413537    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:03:34.415179 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:34.415154    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:03:34.416879 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:34.416849    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:03:35.419665 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:35.419630    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:03:35.421501 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:35.421472    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:03:35.422398 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:35.422370    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:03:35.624462 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:35.624421    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:36.342015 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:36.341974    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cdd98ad-nb2n7"]
Oct 30 13:03:36.342872 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:36.342848    4544 cpu_manager.go:414] "containerMap: removing container" podUID="3a254746-b5be-4d7c-b6ec-3c749effbebf" containerName="revoke-keys"
Oct 30 13:03:36.342872 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:36.342871    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a254746-b5be-4d7c-b6ec-3c749effbebf" containerName="revoke-keys"
Oct 30 13:03:36.343014 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:36.342907    4544 cpu_manager.go:414] "containerMap: removing container" podUID="56f7ecc5-d9a7-4f5c-a572-338b114f3e0f" containerName="revoke-keys"
Oct 30 13:03:36.343014 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:36.342912    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56f7ecc5-d9a7-4f5c-a572-338b114f3e0f" containerName="revoke-keys"
Oct 30 13:03:36.343014 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:36.343003    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="56f7ecc5-d9a7-4f5c-a572-338b114f3e0f" containerName="revoke-keys"
Oct 30 13:03:36.343014 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:36.343015    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a254746-b5be-4d7c-b6ec-3c749effbebf" containerName="revoke-keys"
Oct 30 13:03:36.347288 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:36.347263    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cdd98ad-nb2n7"
Oct 30 13:03:36.362272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:36.362245    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:03:36.373974 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:36.373945    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:03:36.381878 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:36.381830    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cdd98ad-nb2n7"]
Oct 30 13:03:36.444576 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:36.444539    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aa572f0-ed04-4786-8c41-3ddfa27d85b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cdd98ad-nb2n7\" (UID: \"5aa572f0-ed04-4786-8c41-3ddfa27d85b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cdd98ad-nb2n7"
Oct 30 13:03:36.445003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:36.444603    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aa572f0-ed04-4786-8c41-3ddfa27d85b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cdd98ad-nb2n7\" (UID: \"5aa572f0-ed04-4786-8c41-3ddfa27d85b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cdd98ad-nb2n7"
Oct 30 13:03:36.445003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:36.444782    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7smr\" (UniqueName: \"kubernetes.io/projected/5aa572f0-ed04-4786-8c41-3ddfa27d85b3-kube-api-access-f7smr\") pod \"maas-api-revoke-sub-9cdd98ad-nb2n7\" (UID: \"5aa572f0-ed04-4786-8c41-3ddfa27d85b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cdd98ad-nb2n7"
Oct 30 13:03:36.546107 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:36.546062    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aa572f0-ed04-4786-8c41-3ddfa27d85b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cdd98ad-nb2n7\" (UID: \"5aa572f0-ed04-4786-8c41-3ddfa27d85b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cdd98ad-nb2n7"
Oct 30 13:03:36.546323 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:36.546121    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aa572f0-ed04-4786-8c41-3ddfa27d85b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cdd98ad-nb2n7\" (UID: \"5aa572f0-ed04-4786-8c41-3ddfa27d85b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cdd98ad-nb2n7"
Oct 30 13:03:36.546323 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:36.546210    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7smr\" (UniqueName: \"kubernetes.io/projected/5aa572f0-ed04-4786-8c41-3ddfa27d85b3-kube-api-access-f7smr\") pod \"maas-api-revoke-sub-9cdd98ad-nb2n7\" (UID: \"5aa572f0-ed04-4786-8c41-3ddfa27d85b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cdd98ad-nb2n7"
Oct 30 13:03:36.546899 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:36.546870    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aa572f0-ed04-4786-8c41-3ddfa27d85b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cdd98ad-nb2n7\" (UID: \"5aa572f0-ed04-4786-8c41-3ddfa27d85b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cdd98ad-nb2n7"
Oct 30 13:03:36.548588 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:36.548565    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aa572f0-ed04-4786-8c41-3ddfa27d85b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cdd98ad-nb2n7\" (UID: \"5aa572f0-ed04-4786-8c41-3ddfa27d85b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cdd98ad-nb2n7"
Oct 30 13:03:36.569636 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:36.569597    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7smr\" (UniqueName: \"kubernetes.io/projected/5aa572f0-ed04-4786-8c41-3ddfa27d85b3-kube-api-access-f7smr\") pod \"maas-api-revoke-sub-9cdd98ad-nb2n7\" (UID: \"5aa572f0-ed04-4786-8c41-3ddfa27d85b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cdd98ad-nb2n7"
Oct 30 13:03:36.662327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:36.662200    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cdd98ad-nb2n7"
Oct 30 13:03:36.979757 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:36.979724    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cdd98ad-nb2n7"]
Oct 30 13:03:36.981660 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:03:36.981631    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aa572f0_ed04_4786_8c41_3ddfa27d85b3.slice/crio-734c491824e2412a0286ad99e9e80f33cc4da3d74320b009733bdf7e2ff7e845 WatchSource:0}: Error finding container 734c491824e2412a0286ad99e9e80f33cc4da3d74320b009733bdf7e2ff7e845: Status 404 returned error can't find the container with id 734c491824e2412a0286ad99e9e80f33cc4da3d74320b009733bdf7e2ff7e845
Oct 30 13:03:37.564504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:37.564423    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:37.948624 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:37.948586    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cdd98ad-nb2n7" event={"ID":"5aa572f0-ed04-4786-8c41-3ddfa27d85b3","Type":"ContainerStarted","Data":"a434da3d88b69d9bd9317048814abe0fc87ecfd54859de15eac194ec74eaab51"}
Oct 30 13:03:37.948773 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:37.948651    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cdd98ad-nb2n7" event={"ID":"5aa572f0-ed04-4786-8c41-3ddfa27d85b3","Type":"ContainerStarted","Data":"734c491824e2412a0286ad99e9e80f33cc4da3d74320b009733bdf7e2ff7e845"}
Oct 30 13:03:38.004407 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:38.004348    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cdd98ad-nb2n7" podStartSLOduration=2.004328035 podStartE2EDuration="2.004328035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:37.998421734 +0000 UTC m=+1113.045555688" watchObservedRunningTime="2026-10-30 13:03:38.004328035 +0000 UTC m=+1113.051461990"
Oct 30 13:03:38.400847 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:38.400806    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-269db3d3-bgv75"]
Oct 30 13:03:38.405426 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:38.405404    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-269db3d3-bgv75"
Oct 30 13:03:38.429878 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:38.429831    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-269db3d3-bgv75"]
Oct 30 13:03:38.489001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:38.488962    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01f0b3fb-8556-4fd5-a819-5b1f35a8c640-maas-api-xks-ca\") pod \"maas-api-revoke-sub-269db3d3-bgv75\" (UID: \"01f0b3fb-8556-4fd5-a819-5b1f35a8c640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-269db3d3-bgv75"
Oct 30 13:03:38.489149 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:38.489128    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtlt8\" (UniqueName: \"kubernetes.io/projected/01f0b3fb-8556-4fd5-a819-5b1f35a8c640-kube-api-access-xtlt8\") pod \"maas-api-revoke-sub-269db3d3-bgv75\" (UID: \"01f0b3fb-8556-4fd5-a819-5b1f35a8c640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-269db3d3-bgv75"
Oct 30 13:03:38.489325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:38.489309    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01f0b3fb-8556-4fd5-a819-5b1f35a8c640-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-269db3d3-bgv75\" (UID: \"01f0b3fb-8556-4fd5-a819-5b1f35a8c640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-269db3d3-bgv75"
Oct 30 13:03:38.590510 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:38.590471    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01f0b3fb-8556-4fd5-a819-5b1f35a8c640-maas-api-xks-ca\") pod \"maas-api-revoke-sub-269db3d3-bgv75\" (UID: \"01f0b3fb-8556-4fd5-a819-5b1f35a8c640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-269db3d3-bgv75"
Oct 30 13:03:38.590922 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:38.590600    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtlt8\" (UniqueName: \"kubernetes.io/projected/01f0b3fb-8556-4fd5-a819-5b1f35a8c640-kube-api-access-xtlt8\") pod \"maas-api-revoke-sub-269db3d3-bgv75\" (UID: \"01f0b3fb-8556-4fd5-a819-5b1f35a8c640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-269db3d3-bgv75"
Oct 30 13:03:38.590922 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:38.590719    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01f0b3fb-8556-4fd5-a819-5b1f35a8c640-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-269db3d3-bgv75\" (UID: \"01f0b3fb-8556-4fd5-a819-5b1f35a8c640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-269db3d3-bgv75"
Oct 30 13:03:38.591424 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:38.591401    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01f0b3fb-8556-4fd5-a819-5b1f35a8c640-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-269db3d3-bgv75\" (UID: \"01f0b3fb-8556-4fd5-a819-5b1f35a8c640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-269db3d3-bgv75"
Oct 30 13:03:38.593064 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:38.593039    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01f0b3fb-8556-4fd5-a819-5b1f35a8c640-maas-api-xks-ca\") pod \"maas-api-revoke-sub-269db3d3-bgv75\" (UID: \"01f0b3fb-8556-4fd5-a819-5b1f35a8c640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-269db3d3-bgv75"
Oct 30 13:03:38.633511 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:38.633480    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtlt8\" (UniqueName: \"kubernetes.io/projected/01f0b3fb-8556-4fd5-a819-5b1f35a8c640-kube-api-access-xtlt8\") pod \"maas-api-revoke-sub-269db3d3-bgv75\" (UID: \"01f0b3fb-8556-4fd5-a819-5b1f35a8c640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-269db3d3-bgv75"
Oct 30 13:03:38.719989 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:38.719895    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-269db3d3-bgv75"
Oct 30 13:03:38.958375 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:38.958336    4544 generic.go:356] "Generic (PLEG): container finished" podID="5aa572f0-ed04-4786-8c41-3ddfa27d85b3" containerID="a434da3d88b69d9bd9317048814abe0fc87ecfd54859de15eac194ec74eaab51" exitCode=0
Oct 30 13:03:38.958595 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:38.958460    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cdd98ad-nb2n7" event={"ID":"5aa572f0-ed04-4786-8c41-3ddfa27d85b3","Type":"ContainerDied","Data":"a434da3d88b69d9bd9317048814abe0fc87ecfd54859de15eac194ec74eaab51"}
Oct 30 13:03:39.062342 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:03:39.062309    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01f0b3fb_8556_4fd5_a819_5b1f35a8c640.slice/crio-5b25bf0b4ee8e7f97c277cc007dd66eccd945896af831ab311c7ef4bc897b781 WatchSource:0}: Error finding container 5b25bf0b4ee8e7f97c277cc007dd66eccd945896af831ab311c7ef4bc897b781: Status 404 returned error can't find the container with id 5b25bf0b4ee8e7f97c277cc007dd66eccd945896af831ab311c7ef4bc897b781
Oct 30 13:03:39.062651 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:39.062626    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-269db3d3-bgv75"]
Oct 30 13:03:39.792331 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:39.791982    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:39.967958 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:39.967918    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-269db3d3-bgv75" event={"ID":"01f0b3fb-8556-4fd5-a819-5b1f35a8c640","Type":"ContainerStarted","Data":"36486cf473b9ddf32ff2ff86ec58b5f02cae05c455be212317d8d21339f6e9a8"}
Oct 30 13:03:39.968200 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:39.967990    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-269db3d3-bgv75" event={"ID":"01f0b3fb-8556-4fd5-a819-5b1f35a8c640","Type":"ContainerStarted","Data":"5b25bf0b4ee8e7f97c277cc007dd66eccd945896af831ab311c7ef4bc897b781"}
Oct 30 13:03:40.011392 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:40.011339    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-269db3d3-bgv75" podStartSLOduration=2.011322823 podStartE2EDuration="2.011322823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:40.00742495 +0000 UTC m=+1115.054558905" watchObservedRunningTime="2026-10-30 13:03:40.011322823 +0000 UTC m=+1115.058456794"
Oct 30 13:03:40.140893 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:40.140861    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cdd98ad-nb2n7"
Oct 30 13:03:40.224048 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:40.224005    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5aa572f0-ed04-4786-8c41-3ddfa27d85b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aa572f0-ed04-4786-8c41-3ddfa27d85b3-maas-api-xks-ca\") pod \"5aa572f0-ed04-4786-8c41-3ddfa27d85b3\" (UID: \"5aa572f0-ed04-4786-8c41-3ddfa27d85b3\") "
Oct 30 13:03:40.224264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:40.224177    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aa572f0-ed04-4786-8c41-3ddfa27d85b3-kube-api-access-f7smr\" (UniqueName: \"kubernetes.io/projected/5aa572f0-ed04-4786-8c41-3ddfa27d85b3-kube-api-access-f7smr\") pod \"5aa572f0-ed04-4786-8c41-3ddfa27d85b3\" (UID: \"5aa572f0-ed04-4786-8c41-3ddfa27d85b3\") "
Oct 30 13:03:40.224336 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:40.224314    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5aa572f0-ed04-4786-8c41-3ddfa27d85b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aa572f0-ed04-4786-8c41-3ddfa27d85b3-maas-api-openshift-ca\") pod \"5aa572f0-ed04-4786-8c41-3ddfa27d85b3\" (UID: \"5aa572f0-ed04-4786-8c41-3ddfa27d85b3\") "
Oct 30 13:03:40.225238 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:40.224709    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aa572f0-ed04-4786-8c41-3ddfa27d85b3-maas-api-openshift-ca" pod "5aa572f0-ed04-4786-8c41-3ddfa27d85b3" (UID: "5aa572f0-ed04-4786-8c41-3ddfa27d85b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:03:40.225238 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:40.225121    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aa572f0-ed04-4786-8c41-3ddfa27d85b3-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:03:40.226563 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:40.226539    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5aa572f0-ed04-4786-8c41-3ddfa27d85b3-maas-api-xks-ca" pod "5aa572f0-ed04-4786-8c41-3ddfa27d85b3" (UID: "5aa572f0-ed04-4786-8c41-3ddfa27d85b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:03:40.226910 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:40.226893    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aa572f0-ed04-4786-8c41-3ddfa27d85b3-kube-api-access-f7smr" pod "5aa572f0-ed04-4786-8c41-3ddfa27d85b3" (UID: "5aa572f0-ed04-4786-8c41-3ddfa27d85b3"). InnerVolumeSpecName "kube-api-access-f7smr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:03:40.326968 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:40.326926    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aa572f0-ed04-4786-8c41-3ddfa27d85b3-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:03:40.326968 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:40.326963    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7smr\" (UniqueName: \"kubernetes.io/projected/5aa572f0-ed04-4786-8c41-3ddfa27d85b3-kube-api-access-f7smr\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:03:40.977106 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:40.977070    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cdd98ad-nb2n7" event={"ID":"5aa572f0-ed04-4786-8c41-3ddfa27d85b3","Type":"ContainerDied","Data":"734c491824e2412a0286ad99e9e80f33cc4da3d74320b009733bdf7e2ff7e845"}
Oct 30 13:03:40.977626 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:40.977084    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cdd98ad-nb2n7"
Oct 30 13:03:40.977626 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:40.977164    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="734c491824e2412a0286ad99e9e80f33cc4da3d74320b009733bdf7e2ff7e845"
Oct 30 13:03:40.979204 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:40.979173    4544 generic.go:356] "Generic (PLEG): container finished" podID="01f0b3fb-8556-4fd5-a819-5b1f35a8c640" containerID="36486cf473b9ddf32ff2ff86ec58b5f02cae05c455be212317d8d21339f6e9a8" exitCode=0
Oct 30 13:03:40.979349 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:40.979249    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-269db3d3-bgv75" event={"ID":"01f0b3fb-8556-4fd5-a819-5b1f35a8c640","Type":"ContainerDied","Data":"36486cf473b9ddf32ff2ff86ec58b5f02cae05c455be212317d8d21339f6e9a8"}
Oct 30 13:03:41.645396 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:41.645358    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:42.160502 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:42.160472    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-269db3d3-bgv75"
Oct 30 13:03:42.260819 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:42.260701    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01f0b3fb-8556-4fd5-a819-5b1f35a8c640-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01f0b3fb-8556-4fd5-a819-5b1f35a8c640-maas-api-openshift-ca\") pod \"01f0b3fb-8556-4fd5-a819-5b1f35a8c640\" (UID: \"01f0b3fb-8556-4fd5-a819-5b1f35a8c640\") "
Oct 30 13:03:42.260984 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:42.260834    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01f0b3fb-8556-4fd5-a819-5b1f35a8c640-kube-api-access-xtlt8\" (UniqueName: \"kubernetes.io/projected/01f0b3fb-8556-4fd5-a819-5b1f35a8c640-kube-api-access-xtlt8\") pod \"01f0b3fb-8556-4fd5-a819-5b1f35a8c640\" (UID: \"01f0b3fb-8556-4fd5-a819-5b1f35a8c640\") "
Oct 30 13:03:42.261069 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:42.261013    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01f0b3fb-8556-4fd5-a819-5b1f35a8c640-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01f0b3fb-8556-4fd5-a819-5b1f35a8c640-maas-api-xks-ca\") pod \"01f0b3fb-8556-4fd5-a819-5b1f35a8c640\" (UID: \"01f0b3fb-8556-4fd5-a819-5b1f35a8c640\") "
Oct 30 13:03:42.261187 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:42.261159    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01f0b3fb-8556-4fd5-a819-5b1f35a8c640-maas-api-openshift-ca" pod "01f0b3fb-8556-4fd5-a819-5b1f35a8c640" (UID: "01f0b3fb-8556-4fd5-a819-5b1f35a8c640"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:03:42.261727 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:42.261701    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01f0b3fb-8556-4fd5-a819-5b1f35a8c640-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:03:42.263135 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:42.263112    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01f0b3fb-8556-4fd5-a819-5b1f35a8c640-kube-api-access-xtlt8" pod "01f0b3fb-8556-4fd5-a819-5b1f35a8c640" (UID: "01f0b3fb-8556-4fd5-a819-5b1f35a8c640"). InnerVolumeSpecName "kube-api-access-xtlt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:03:42.263628 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:42.263607    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01f0b3fb-8556-4fd5-a819-5b1f35a8c640-maas-api-xks-ca" pod "01f0b3fb-8556-4fd5-a819-5b1f35a8c640" (UID: "01f0b3fb-8556-4fd5-a819-5b1f35a8c640"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:03:42.362871 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:42.362825    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtlt8\" (UniqueName: \"kubernetes.io/projected/01f0b3fb-8556-4fd5-a819-5b1f35a8c640-kube-api-access-xtlt8\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:03:42.362871 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:42.362863    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01f0b3fb-8556-4fd5-a819-5b1f35a8c640-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:03:42.996069 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:42.996041    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-269db3d3-bgv75"
Oct 30 13:03:42.996282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:42.996034    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-269db3d3-bgv75" event={"ID":"01f0b3fb-8556-4fd5-a819-5b1f35a8c640","Type":"ContainerDied","Data":"5b25bf0b4ee8e7f97c277cc007dd66eccd945896af831ab311c7ef4bc897b781"}
Oct 30 13:03:42.996282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:42.996175    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b25bf0b4ee8e7f97c277cc007dd66eccd945896af831ab311c7ef4bc897b781"
Oct 30 13:03:43.511805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:43.511752    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:43.825259 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:43.825212    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ab7218e-vfc2f"]
Oct 30 13:03:43.825936 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:43.825920    4544 cpu_manager.go:414] "containerMap: removing container" podUID="01f0b3fb-8556-4fd5-a819-5b1f35a8c640" containerName="revoke-keys"
Oct 30 13:03:43.825986 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:43.825938    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01f0b3fb-8556-4fd5-a819-5b1f35a8c640" containerName="revoke-keys"
Oct 30 13:03:43.825986 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:43.825972    4544 cpu_manager.go:414] "containerMap: removing container" podUID="5aa572f0-ed04-4786-8c41-3ddfa27d85b3" containerName="revoke-keys"
Oct 30 13:03:43.825986 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:43.825977    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aa572f0-ed04-4786-8c41-3ddfa27d85b3" containerName="revoke-keys"
Oct 30 13:03:43.826077 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:43.826067    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aa572f0-ed04-4786-8c41-3ddfa27d85b3" containerName="revoke-keys"
Oct 30 13:03:43.826118 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:43.826077    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="01f0b3fb-8556-4fd5-a819-5b1f35a8c640" containerName="revoke-keys"
Oct 30 13:03:43.830028 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:43.830010    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab7218e-vfc2f"
Oct 30 13:03:43.850783 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:43.850737    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:03:43.862119 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:43.862085    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:03:43.867098 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:43.867067    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ab7218e-vfc2f"]
Oct 30 13:03:43.993438 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:43.993392    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5zsm\" (UniqueName: \"kubernetes.io/projected/bee74c09-88b6-490d-8f7c-576adec205ba-kube-api-access-d5zsm\") pod \"maas-api-revoke-sub-8ab7218e-vfc2f\" (UID: \"bee74c09-88b6-490d-8f7c-576adec205ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab7218e-vfc2f"
Oct 30 13:03:43.993708 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:43.993683    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bee74c09-88b6-490d-8f7c-576adec205ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ab7218e-vfc2f\" (UID: \"bee74c09-88b6-490d-8f7c-576adec205ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab7218e-vfc2f"
Oct 30 13:03:43.993780 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:43.993736    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bee74c09-88b6-490d-8f7c-576adec205ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ab7218e-vfc2f\" (UID: \"bee74c09-88b6-490d-8f7c-576adec205ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab7218e-vfc2f"
Oct 30 13:03:44.095988 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:44.095898    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bee74c09-88b6-490d-8f7c-576adec205ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ab7218e-vfc2f\" (UID: \"bee74c09-88b6-490d-8f7c-576adec205ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab7218e-vfc2f"
Oct 30 13:03:44.095988 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:44.095948    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bee74c09-88b6-490d-8f7c-576adec205ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ab7218e-vfc2f\" (UID: \"bee74c09-88b6-490d-8f7c-576adec205ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab7218e-vfc2f"
Oct 30 13:03:44.096190 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:44.096059    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5zsm\" (UniqueName: \"kubernetes.io/projected/bee74c09-88b6-490d-8f7c-576adec205ba-kube-api-access-d5zsm\") pod \"maas-api-revoke-sub-8ab7218e-vfc2f\" (UID: \"bee74c09-88b6-490d-8f7c-576adec205ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab7218e-vfc2f"
Oct 30 13:03:44.096602 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:44.096571    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bee74c09-88b6-490d-8f7c-576adec205ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ab7218e-vfc2f\" (UID: \"bee74c09-88b6-490d-8f7c-576adec205ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab7218e-vfc2f"
Oct 30 13:03:44.098981 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:44.098950    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bee74c09-88b6-490d-8f7c-576adec205ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ab7218e-vfc2f\" (UID: \"bee74c09-88b6-490d-8f7c-576adec205ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab7218e-vfc2f"
Oct 30 13:03:44.124844 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:44.124807    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5zsm\" (UniqueName: \"kubernetes.io/projected/bee74c09-88b6-490d-8f7c-576adec205ba-kube-api-access-d5zsm\") pod \"maas-api-revoke-sub-8ab7218e-vfc2f\" (UID: \"bee74c09-88b6-490d-8f7c-576adec205ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab7218e-vfc2f"
Oct 30 13:03:44.144712 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:44.144674    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab7218e-vfc2f"
Oct 30 13:03:44.457923 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:44.457892    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ab7218e-vfc2f"]
Oct 30 13:03:44.460271 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:03:44.460209    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee74c09_88b6_490d_8f7c_576adec205ba.slice/crio-32b8d313f6a4e4f833e18eecb1ae4755202a1bc6d55fe0a48e3063d0983ae515 WatchSource:0}: Error finding container 32b8d313f6a4e4f833e18eecb1ae4755202a1bc6d55fe0a48e3063d0983ae515: Status 404 returned error can't find the container with id 32b8d313f6a4e4f833e18eecb1ae4755202a1bc6d55fe0a48e3063d0983ae515
Oct 30 13:03:45.025305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:45.025177    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab7218e-vfc2f" event={"ID":"bee74c09-88b6-490d-8f7c-576adec205ba","Type":"ContainerStarted","Data":"5fd2b3428780d878eae86ff221999d774b6b24fff9e96af0026a7ac20d9be269"}
Oct 30 13:03:45.025305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:45.025258    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab7218e-vfc2f" event={"ID":"bee74c09-88b6-490d-8f7c-576adec205ba","Type":"ContainerStarted","Data":"32b8d313f6a4e4f833e18eecb1ae4755202a1bc6d55fe0a48e3063d0983ae515"}
Oct 30 13:03:45.067534 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:45.067459    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab7218e-vfc2f" podStartSLOduration=2.067440178 podStartE2EDuration="2.067440178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:45.065199798 +0000 UTC m=+1120.112333751" watchObservedRunningTime="2026-10-30 13:03:45.067440178 +0000 UTC m=+1120.114574132"
Oct 30 13:03:45.309892 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:45.309759    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:46.034838 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:46.034805    4544 generic.go:356] "Generic (PLEG): container finished" podID="bee74c09-88b6-490d-8f7c-576adec205ba" containerID="5fd2b3428780d878eae86ff221999d774b6b24fff9e96af0026a7ac20d9be269" exitCode=0
Oct 30 13:03:46.035365 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:46.034891    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab7218e-vfc2f" event={"ID":"bee74c09-88b6-490d-8f7c-576adec205ba","Type":"ContainerDied","Data":"5fd2b3428780d878eae86ff221999d774b6b24fff9e96af0026a7ac20d9be269"}
Oct 30 13:03:47.263413 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:47.260489    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:47.288482 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:47.288444    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab7218e-vfc2f"
Oct 30 13:03:47.458682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:47.458642    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bee74c09-88b6-490d-8f7c-576adec205ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bee74c09-88b6-490d-8f7c-576adec205ba-maas-api-openshift-ca\") pod \"bee74c09-88b6-490d-8f7c-576adec205ba\" (UID: \"bee74c09-88b6-490d-8f7c-576adec205ba\") "
Oct 30 13:03:47.458869 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:47.458713    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bee74c09-88b6-490d-8f7c-576adec205ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bee74c09-88b6-490d-8f7c-576adec205ba-maas-api-xks-ca\") pod \"bee74c09-88b6-490d-8f7c-576adec205ba\" (UID: \"bee74c09-88b6-490d-8f7c-576adec205ba\") "
Oct 30 13:03:47.458869 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:47.458802    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bee74c09-88b6-490d-8f7c-576adec205ba-kube-api-access-d5zsm\" (UniqueName: \"kubernetes.io/projected/bee74c09-88b6-490d-8f7c-576adec205ba-kube-api-access-d5zsm\") pod \"bee74c09-88b6-490d-8f7c-576adec205ba\" (UID: \"bee74c09-88b6-490d-8f7c-576adec205ba\") "
Oct 30 13:03:47.459101 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:47.459070    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bee74c09-88b6-490d-8f7c-576adec205ba-maas-api-openshift-ca" pod "bee74c09-88b6-490d-8f7c-576adec205ba" (UID: "bee74c09-88b6-490d-8f7c-576adec205ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:03:47.459584 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:47.459561    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bee74c09-88b6-490d-8f7c-576adec205ba-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:03:47.460932 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:47.460901    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bee74c09-88b6-490d-8f7c-576adec205ba-maas-api-xks-ca" pod "bee74c09-88b6-490d-8f7c-576adec205ba" (UID: "bee74c09-88b6-490d-8f7c-576adec205ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:03:47.460932 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:47.460917    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee74c09-88b6-490d-8f7c-576adec205ba-kube-api-access-d5zsm" pod "bee74c09-88b6-490d-8f7c-576adec205ba" (UID: "bee74c09-88b6-490d-8f7c-576adec205ba"). InnerVolumeSpecName "kube-api-access-d5zsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:03:47.561175 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:47.561129    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bee74c09-88b6-490d-8f7c-576adec205ba-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:03:47.561175 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:47.561165    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5zsm\" (UniqueName: \"kubernetes.io/projected/bee74c09-88b6-490d-8f7c-576adec205ba-kube-api-access-d5zsm\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:03:48.052175 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:48.052138    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab7218e-vfc2f"
Oct 30 13:03:48.052175 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:48.052151    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ab7218e-vfc2f" event={"ID":"bee74c09-88b6-490d-8f7c-576adec205ba","Type":"ContainerDied","Data":"32b8d313f6a4e4f833e18eecb1ae4755202a1bc6d55fe0a48e3063d0983ae515"}
Oct 30 13:03:48.052536 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:48.052273    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32b8d313f6a4e4f833e18eecb1ae4755202a1bc6d55fe0a48e3063d0983ae515"
Oct 30 13:03:49.975150 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:49.974395    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:50.004506 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:50.004460    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 13:03:50.377246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:50.376366    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:50.751926 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:50.751766    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:50.893538 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:50.893491    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:50.945685 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:50.945630    4544 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:51.124054 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:51.123982    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 13:03:51.564072 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:51.188449    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 13:03:51.583182 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:51.388541    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:51.584127 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:51.547022    4544 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:03:51.584403 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:51.584353    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph" podUID="df98822e-ff7b-4c1d-ad2e-1e3b88523780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:51.857258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:51.857080    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:03:51.857258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:51.857182    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:03:51.857258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:51.857261    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:03:52.095252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:52.027365    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:03:52.161551 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:52.090741    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:52.161551 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:52.143012    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 13:03:52.161551 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:52.143108    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:03:52.467311 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:52.451182    4544 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:52.629594 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:52.629535    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:52.635794 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:52.635751    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 13:03:52.815389 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:52.815272    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 13:03:53.148249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:53.147591    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 13:03:53.178247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:53.163600    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:03:53.401483 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:53.401380    4544 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:03:53.409377 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:53.409320    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7" podUID="317201b4-13b3-4341-a5a2-e4be2531f5f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:53.545258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:53.537870    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:53.637365 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:53.637315    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:53.662483 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:53.644390    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 13:03:53.792847 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:53.764881    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:53.884452 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:53.884395    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:53.958543 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:53.958382    4544 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded"
Oct 30 13:03:54.226308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:54.214325    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:54.311155 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:54.311110    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:03:54.645635 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:54.645584    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:54.648159 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:54.648122    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:54.902301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:54.900082    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:03:55.214726 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:55.214559    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:55.313999 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:55.313949    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:55.667269 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:55.654551    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:55.814213 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:55.814170    4544 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:55.917674 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:55.916329    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 13:03:55.918361 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:55.918311    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 13:03:56.257160 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:56.257040    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d1c7a0c-slws2"]
Oct 30 13:03:56.258839 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:56.258815    4544 cpu_manager.go:414] "containerMap: removing container" podUID="bee74c09-88b6-490d-8f7c-576adec205ba" containerName="revoke-keys"
Oct 30 13:03:56.259048 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:56.259032    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bee74c09-88b6-490d-8f7c-576adec205ba" containerName="revoke-keys"
Oct 30 13:03:56.259495 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:56.259479    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="bee74c09-88b6-490d-8f7c-576adec205ba" containerName="revoke-keys"
Oct 30 13:03:56.388277 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:56.387391    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:56.406784 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:56.388956    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:56.464279 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:56.453395    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:03:56.464279 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:56.453818    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1c7a0c-slws2"
Oct 30 13:03:56.464279 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:56.458145    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:03:56.558999 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:56.558884    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 13:03:56.560388 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:56.560170    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:03:56.608404 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:56.596523    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:03:56.661678 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:56.656926    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30a39639-10fa-4c33-a923-f3ddfaf3d9aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d1c7a0c-slws2\" (UID: \"30a39639-10fa-4c33-a923-f3ddfaf3d9aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1c7a0c-slws2"
Oct 30 13:03:56.661678 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:56.657264    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdgjl\" (UniqueName: \"kubernetes.io/projected/30a39639-10fa-4c33-a923-f3ddfaf3d9aa-kube-api-access-zdgjl\") pod \"maas-api-revoke-sub-9d1c7a0c-slws2\" (UID: \"30a39639-10fa-4c33-a923-f3ddfaf3d9aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1c7a0c-slws2"
Oct 30 13:03:56.661678 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:56.657473    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30a39639-10fa-4c33-a923-f3ddfaf3d9aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d1c7a0c-slws2\" (UID: \"30a39639-10fa-4c33-a923-f3ddfaf3d9aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1c7a0c-slws2"
Oct 30 13:03:56.662348 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:56.662208    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d1c7a0c-slws2"]
Oct 30 13:03:56.759771 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:56.759620    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30a39639-10fa-4c33-a923-f3ddfaf3d9aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d1c7a0c-slws2\" (UID: \"30a39639-10fa-4c33-a923-f3ddfaf3d9aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1c7a0c-slws2"
Oct 30 13:03:56.760091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:56.760067    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30a39639-10fa-4c33-a923-f3ddfaf3d9aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d1c7a0c-slws2\" (UID: \"30a39639-10fa-4c33-a923-f3ddfaf3d9aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1c7a0c-slws2"
Oct 30 13:03:56.760878 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:56.760851    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30a39639-10fa-4c33-a923-f3ddfaf3d9aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d1c7a0c-slws2\" (UID: \"30a39639-10fa-4c33-a923-f3ddfaf3d9aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1c7a0c-slws2"
Oct 30 13:03:56.762201 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:56.761812    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:03:56.782070 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:56.781192    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdgjl\" (UniqueName: \"kubernetes.io/projected/30a39639-10fa-4c33-a923-f3ddfaf3d9aa-kube-api-access-zdgjl\") pod \"maas-api-revoke-sub-9d1c7a0c-slws2\" (UID: \"30a39639-10fa-4c33-a923-f3ddfaf3d9aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1c7a0c-slws2"
Oct 30 13:03:56.802747 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:56.789949    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30a39639-10fa-4c33-a923-f3ddfaf3d9aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d1c7a0c-slws2\" (UID: \"30a39639-10fa-4c33-a923-f3ddfaf3d9aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1c7a0c-slws2"
Oct 30 13:03:56.809608 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:56.809186    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdgjl\" (UniqueName: \"kubernetes.io/projected/30a39639-10fa-4c33-a923-f3ddfaf3d9aa-kube-api-access-zdgjl\") pod \"maas-api-revoke-sub-9d1c7a0c-slws2\" (UID: \"30a39639-10fa-4c33-a923-f3ddfaf3d9aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1c7a0c-slws2"
Oct 30 13:03:57.076652 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:57.076621    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1c7a0c-slws2"
Oct 30 13:03:57.370093 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:57.369999    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:03:57.393586 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:03:57.393551    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a39639_10fa_4c33_a923_f3ddfaf3d9aa.slice/crio-943fc8a9a898f93185a413f4239ce6d0f20d22b3fb45016d2616beda4de56fbc WatchSource:0}: Error finding container 943fc8a9a898f93185a413f4239ce6d0f20d22b3fb45016d2616beda4de56fbc: Status 404 returned error can't find the container with id 943fc8a9a898f93185a413f4239ce6d0f20d22b3fb45016d2616beda4de56fbc
Oct 30 13:03:57.394073 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:57.393948    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d1c7a0c-slws2"]
Oct 30 13:03:58.020574 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:58.020478    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1c7a0c-slws2" event={"ID":"30a39639-10fa-4c33-a923-f3ddfaf3d9aa","Type":"ContainerStarted","Data":"2c0acec0b4ac372ae25a356166fdfb5343fb080113f128d2d604cc8eb8dd96d7"}
Oct 30 13:03:58.020574 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:58.020546    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1c7a0c-slws2" event={"ID":"30a39639-10fa-4c33-a923-f3ddfaf3d9aa","Type":"ContainerStarted","Data":"943fc8a9a898f93185a413f4239ce6d0f20d22b3fb45016d2616beda4de56fbc"}
Oct 30 13:03:58.072336 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:58.072277    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1c7a0c-slws2" podStartSLOduration=3.072260881 podStartE2EDuration="3.072260881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:58.069399175 +0000 UTC m=+1133.116533128" watchObservedRunningTime="2026-10-30 13:03:58.072260881 +0000 UTC m=+1133.119394870"
Oct 30 13:03:59.008664 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:03:59.008633    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:03:59.034065 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:59.034031    4544 generic.go:356] "Generic (PLEG): container finished" podID="30a39639-10fa-4c33-a923-f3ddfaf3d9aa" containerID="2c0acec0b4ac372ae25a356166fdfb5343fb080113f128d2d604cc8eb8dd96d7" exitCode=0
Oct 30 13:03:59.034297 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:59.034125    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1c7a0c-slws2" event={"ID":"30a39639-10fa-4c33-a923-f3ddfaf3d9aa","Type":"ContainerDied","Data":"2c0acec0b4ac372ae25a356166fdfb5343fb080113f128d2d604cc8eb8dd96d7"}
Oct 30 13:03:59.154479 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:03:59.154439    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:00.203316 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:00.203283    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1c7a0c-slws2"
Oct 30 13:04:00.282357 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:00.282309    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30a39639-10fa-4c33-a923-f3ddfaf3d9aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30a39639-10fa-4c33-a923-f3ddfaf3d9aa-maas-api-xks-ca\") pod \"30a39639-10fa-4c33-a923-f3ddfaf3d9aa\" (UID: \"30a39639-10fa-4c33-a923-f3ddfaf3d9aa\") "
Oct 30 13:04:00.282566 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:00.282512    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30a39639-10fa-4c33-a923-f3ddfaf3d9aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30a39639-10fa-4c33-a923-f3ddfaf3d9aa-maas-api-openshift-ca\") pod \"30a39639-10fa-4c33-a923-f3ddfaf3d9aa\" (UID: \"30a39639-10fa-4c33-a923-f3ddfaf3d9aa\") "
Oct 30 13:04:00.282639 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:00.282567    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30a39639-10fa-4c33-a923-f3ddfaf3d9aa-kube-api-access-zdgjl\" (UniqueName: \"kubernetes.io/projected/30a39639-10fa-4c33-a923-f3ddfaf3d9aa-kube-api-access-zdgjl\") pod \"30a39639-10fa-4c33-a923-f3ddfaf3d9aa\" (UID: \"30a39639-10fa-4c33-a923-f3ddfaf3d9aa\") "
Oct 30 13:04:00.283295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:00.282922    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30a39639-10fa-4c33-a923-f3ddfaf3d9aa-maas-api-openshift-ca" pod "30a39639-10fa-4c33-a923-f3ddfaf3d9aa" (UID: "30a39639-10fa-4c33-a923-f3ddfaf3d9aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:00.283295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:00.283261    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30a39639-10fa-4c33-a923-f3ddfaf3d9aa-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:04:00.285167 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:00.285135    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30a39639-10fa-4c33-a923-f3ddfaf3d9aa-maas-api-xks-ca" pod "30a39639-10fa-4c33-a923-f3ddfaf3d9aa" (UID: "30a39639-10fa-4c33-a923-f3ddfaf3d9aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:04:00.285167 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:00.285161    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30a39639-10fa-4c33-a923-f3ddfaf3d9aa-kube-api-access-zdgjl" pod "30a39639-10fa-4c33-a923-f3ddfaf3d9aa" (UID: "30a39639-10fa-4c33-a923-f3ddfaf3d9aa"). InnerVolumeSpecName "kube-api-access-zdgjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:00.384538 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:00.384499    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdgjl\" (UniqueName: \"kubernetes.io/projected/30a39639-10fa-4c33-a923-f3ddfaf3d9aa-kube-api-access-zdgjl\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:04:00.384538 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:00.384536    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30a39639-10fa-4c33-a923-f3ddfaf3d9aa-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:04:01.078921 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:01.076513    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:01.118743 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:01.118713    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1c7a0c-slws2"
Oct 30 13:04:01.125067 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:01.125029    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1c7a0c-slws2" event={"ID":"30a39639-10fa-4c33-a923-f3ddfaf3d9aa","Type":"ContainerDied","Data":"943fc8a9a898f93185a413f4239ce6d0f20d22b3fb45016d2616beda4de56fbc"}
Oct 30 13:04:01.125367 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:01.125345    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="943fc8a9a898f93185a413f4239ce6d0f20d22b3fb45016d2616beda4de56fbc"
Oct 30 13:04:02.737772 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:02.737735    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:04.478893 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:04.478856    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:06.551325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:06.547365    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c974f1-22fj9"]
Oct 30 13:04:06.551325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:06.548486    4544 cpu_manager.go:414] "containerMap: removing container" podUID="30a39639-10fa-4c33-a923-f3ddfaf3d9aa" containerName="revoke-keys"
Oct 30 13:04:06.551325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:06.548510    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30a39639-10fa-4c33-a923-f3ddfaf3d9aa" containerName="revoke-keys"
Oct 30 13:04:06.551325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:06.548687    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="30a39639-10fa-4c33-a923-f3ddfaf3d9aa" containerName="revoke-keys"
Oct 30 13:04:06.580778 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:06.580740    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:06.654194 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:06.586736    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c974f1-22fj9"
Oct 30 13:04:06.654194 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:06.602568    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c974f1-22fj9"]
Oct 30 13:04:06.669048 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:06.665859    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:04:06.669048 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:06.666660    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:04:06.767915 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:06.767878    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d04db7d3-e497-4c07-97b4-3d2732f4f1cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30c974f1-22fj9\" (UID: \"d04db7d3-e497-4c07-97b4-3d2732f4f1cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c974f1-22fj9"
Oct 30 13:04:06.768151 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:06.768089    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srrh8\" (UniqueName: \"kubernetes.io/projected/d04db7d3-e497-4c07-97b4-3d2732f4f1cb-kube-api-access-srrh8\") pod \"maas-api-revoke-sub-30c974f1-22fj9\" (UID: \"d04db7d3-e497-4c07-97b4-3d2732f4f1cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c974f1-22fj9"
Oct 30 13:04:06.768275 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:06.768251    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d04db7d3-e497-4c07-97b4-3d2732f4f1cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30c974f1-22fj9\" (UID: \"d04db7d3-e497-4c07-97b4-3d2732f4f1cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c974f1-22fj9"
Oct 30 13:04:06.869992 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:06.869957    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d04db7d3-e497-4c07-97b4-3d2732f4f1cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30c974f1-22fj9\" (UID: \"d04db7d3-e497-4c07-97b4-3d2732f4f1cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c974f1-22fj9"
Oct 30 13:04:06.870189 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:06.870106    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d04db7d3-e497-4c07-97b4-3d2732f4f1cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30c974f1-22fj9\" (UID: \"d04db7d3-e497-4c07-97b4-3d2732f4f1cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c974f1-22fj9"
Oct 30 13:04:06.870189 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:06.870181    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srrh8\" (UniqueName: \"kubernetes.io/projected/d04db7d3-e497-4c07-97b4-3d2732f4f1cb-kube-api-access-srrh8\") pod \"maas-api-revoke-sub-30c974f1-22fj9\" (UID: \"d04db7d3-e497-4c07-97b4-3d2732f4f1cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c974f1-22fj9"
Oct 30 13:04:06.870784 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:06.870755    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d04db7d3-e497-4c07-97b4-3d2732f4f1cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30c974f1-22fj9\" (UID: \"d04db7d3-e497-4c07-97b4-3d2732f4f1cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c974f1-22fj9"
Oct 30 13:04:06.872601 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:06.872576    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d04db7d3-e497-4c07-97b4-3d2732f4f1cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30c974f1-22fj9\" (UID: \"d04db7d3-e497-4c07-97b4-3d2732f4f1cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c974f1-22fj9"
Oct 30 13:04:06.893341 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:06.893312    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srrh8\" (UniqueName: \"kubernetes.io/projected/d04db7d3-e497-4c07-97b4-3d2732f4f1cb-kube-api-access-srrh8\") pod \"maas-api-revoke-sub-30c974f1-22fj9\" (UID: \"d04db7d3-e497-4c07-97b4-3d2732f4f1cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c974f1-22fj9"
Oct 30 13:04:06.990753 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:06.990714    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c974f1-22fj9"
Oct 30 13:04:07.300265 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:07.300202    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c974f1-22fj9"]
Oct 30 13:04:07.301625 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:04:07.301599    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd04db7d3_e497_4c07_97b4_3d2732f4f1cb.slice/crio-e58310e282e3a3da5fb64b863499e39cf9fa7c4c8379d63cb28482a54d3b9da3 WatchSource:0}: Error finding container e58310e282e3a3da5fb64b863499e39cf9fa7c4c8379d63cb28482a54d3b9da3: Status 404 returned error can't find the container with id e58310e282e3a3da5fb64b863499e39cf9fa7c4c8379d63cb28482a54d3b9da3
Oct 30 13:04:08.194969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:08.194926    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c974f1-22fj9" event={"ID":"d04db7d3-e497-4c07-97b4-3d2732f4f1cb","Type":"ContainerStarted","Data":"8ce24da7c0298914401e0c5b39b2851eb7035207d9a98e44858b8521d35b481a"}
Oct 30 13:04:08.195488 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:08.194994    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c974f1-22fj9" event={"ID":"d04db7d3-e497-4c07-97b4-3d2732f4f1cb","Type":"ContainerStarted","Data":"e58310e282e3a3da5fb64b863499e39cf9fa7c4c8379d63cb28482a54d3b9da3"}
Oct 30 13:04:08.237899 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:08.237784    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c974f1-22fj9" podStartSLOduration=2.237770001 podStartE2EDuration="2.237770001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:08.236574913 +0000 UTC m=+1143.283708865" watchObservedRunningTime="2026-10-30 13:04:08.237770001 +0000 UTC m=+1143.284903953"
Oct 30 13:04:08.502085 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:08.501963    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:09.204743 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:09.204710    4544 generic.go:356] "Generic (PLEG): container finished" podID="d04db7d3-e497-4c07-97b4-3d2732f4f1cb" containerID="8ce24da7c0298914401e0c5b39b2851eb7035207d9a98e44858b8521d35b481a" exitCode=0
Oct 30 13:04:09.205161 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:09.204790    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c974f1-22fj9" event={"ID":"d04db7d3-e497-4c07-97b4-3d2732f4f1cb","Type":"ContainerDied","Data":"8ce24da7c0298914401e0c5b39b2851eb7035207d9a98e44858b8521d35b481a"}
Oct 30 13:04:11.202246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:11.181699    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:11.645477 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:11.643727    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:11.674213 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:11.672853    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:11.816524 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:11.816471    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c974f1-22fj9"
Oct 30 13:04:11.928593 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:11.928486    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d04db7d3-e497-4c07-97b4-3d2732f4f1cb-kube-api-access-srrh8\" (UniqueName: \"kubernetes.io/projected/d04db7d3-e497-4c07-97b4-3d2732f4f1cb-kube-api-access-srrh8\") pod \"d04db7d3-e497-4c07-97b4-3d2732f4f1cb\" (UID: \"d04db7d3-e497-4c07-97b4-3d2732f4f1cb\") "
Oct 30 13:04:11.929457 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:11.929406    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d04db7d3-e497-4c07-97b4-3d2732f4f1cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d04db7d3-e497-4c07-97b4-3d2732f4f1cb-maas-api-xks-ca\") pod \"d04db7d3-e497-4c07-97b4-3d2732f4f1cb\" (UID: \"d04db7d3-e497-4c07-97b4-3d2732f4f1cb\") "
Oct 30 13:04:11.930058 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:11.930035    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d04db7d3-e497-4c07-97b4-3d2732f4f1cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d04db7d3-e497-4c07-97b4-3d2732f4f1cb-maas-api-openshift-ca\") pod \"d04db7d3-e497-4c07-97b4-3d2732f4f1cb\" (UID: \"d04db7d3-e497-4c07-97b4-3d2732f4f1cb\") "
Oct 30 13:04:11.932418 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:11.932370    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d04db7d3-e497-4c07-97b4-3d2732f4f1cb-kube-api-access-srrh8" pod "d04db7d3-e497-4c07-97b4-3d2732f4f1cb" (UID: "d04db7d3-e497-4c07-97b4-3d2732f4f1cb"). InnerVolumeSpecName "kube-api-access-srrh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:11.946246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:11.942490    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d04db7d3-e497-4c07-97b4-3d2732f4f1cb-maas-api-openshift-ca" pod "d04db7d3-e497-4c07-97b4-3d2732f4f1cb" (UID: "d04db7d3-e497-4c07-97b4-3d2732f4f1cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:11.951816 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:11.950316    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 13:04:11.964252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:11.963472    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d04db7d3-e497-4c07-97b4-3d2732f4f1cb-maas-api-xks-ca" pod "d04db7d3-e497-4c07-97b4-3d2732f4f1cb" (UID: "d04db7d3-e497-4c07-97b4-3d2732f4f1cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:04:12.068496 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:12.068452    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srrh8\" (UniqueName: \"kubernetes.io/projected/d04db7d3-e497-4c07-97b4-3d2732f4f1cb-kube-api-access-srrh8\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:04:12.068752 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:12.068733    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d04db7d3-e497-4c07-97b4-3d2732f4f1cb-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:04:12.068852 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:12.068839    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d04db7d3-e497-4c07-97b4-3d2732f4f1cb-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:04:12.186980 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:12.186870    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:12.202317 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:12.202264    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 13:04:12.329306 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:12.203799    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:04:12.376123 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:12.376075    4544 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:12.682984 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:12.652187    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:12.682984 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:12.654945    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:04:12.721575 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:12.684868    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 13:04:12.721575 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:12.685635    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:04:12.970364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:12.953065    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 13:04:12.995438 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:12.995402    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:04:13.248753 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:13.248640    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:04:13.401249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:13.377433    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:13.500318 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:13.384007    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c974f1-22fj9"
Oct 30 13:04:13.670592 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:13.670511    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6nbzr" podUID="8c41830a-e18f-4b80-a625-d013bc6f6f5a" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.21:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:13.673589 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:13.672556    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:13.686896 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:13.686850    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 13:04:13.942772 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:13.942253    4544 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded"
Oct 30 13:04:13.965704 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:13.963614    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:14.076836 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:14.026211    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c974f1-22fj9" event={"ID":"d04db7d3-e497-4c07-97b4-3d2732f4f1cb","Type":"ContainerDied","Data":"e58310e282e3a3da5fb64b863499e39cf9fa7c4c8379d63cb28482a54d3b9da3"}
Oct 30 13:04:14.077080 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:14.077060    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e58310e282e3a3da5fb64b863499e39cf9fa7c4c8379d63cb28482a54d3b9da3"
Oct 30 13:04:14.077258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:14.077244    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 13:04:14.291306 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:14.290585    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:04:14.378358 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:14.353604    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:14.546308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:14.463434    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 13:04:14.714247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:14.693571    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:14.856993 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:14.856309    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 13:04:14.966380 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:14.966323    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 13:04:15.321305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:15.320354    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:04:15.477670 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:15.476451    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:15.735181 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:15.721648    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:15.799276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:15.794640    4544 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": context deadline exceeded"
Oct 30 13:04:15.858268 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:15.856650    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:16.076046 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:16.074813    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:16.333736 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:16.333638    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:04:16.490246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:16.486946    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:16.708311 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:16.688738    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:04:16.753246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:16.741394    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:16.779502 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:16.777959    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:04:17.073294 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:17.073252    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 13:04:17.339317 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:17.338453    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:04:17.537079 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:17.537047    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 13:04:17.537284 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:17.537097    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:04:17.537284 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:17.537129    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:04:17.796905 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:17.796808    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:17.899315 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:17.899273    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9185e827-6j8kb"]
Oct 30 13:04:17.900660 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:17.900627    4544 cpu_manager.go:414] "containerMap: removing container" podUID="d04db7d3-e497-4c07-97b4-3d2732f4f1cb" containerName="revoke-keys"
Oct 30 13:04:17.900870 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:17.900855    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d04db7d3-e497-4c07-97b4-3d2732f4f1cb" containerName="revoke-keys"
Oct 30 13:04:17.901191 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:17.901176    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="d04db7d3-e497-4c07-97b4-3d2732f4f1cb" containerName="revoke-keys"
Oct 30 13:04:17.931694 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:17.931657    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9185e827-6j8kb"
Oct 30 13:04:17.941751 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:17.941720    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9185e827-6j8kb"]
Oct 30 13:04:17.951237 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:17.950513    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:04:17.951652 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:17.951629    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:04:18.040363 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:18.040312    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88815187-6e50-4b15-984a-ac4efe339e0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9185e827-6j8kb\" (UID: \"88815187-6e50-4b15-984a-ac4efe339e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9185e827-6j8kb"
Oct 30 13:04:18.040549 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:18.040371    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flbsg\" (UniqueName: \"kubernetes.io/projected/88815187-6e50-4b15-984a-ac4efe339e0e-kube-api-access-flbsg\") pod \"maas-api-revoke-sub-9185e827-6j8kb\" (UID: \"88815187-6e50-4b15-984a-ac4efe339e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9185e827-6j8kb"
Oct 30 13:04:18.040598 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:18.040540    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88815187-6e50-4b15-984a-ac4efe339e0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9185e827-6j8kb\" (UID: \"88815187-6e50-4b15-984a-ac4efe339e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9185e827-6j8kb"
Oct 30 13:04:18.141715 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:18.141670    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88815187-6e50-4b15-984a-ac4efe339e0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9185e827-6j8kb\" (UID: \"88815187-6e50-4b15-984a-ac4efe339e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9185e827-6j8kb"
Oct 30 13:04:18.141923 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:18.141851    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88815187-6e50-4b15-984a-ac4efe339e0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9185e827-6j8kb\" (UID: \"88815187-6e50-4b15-984a-ac4efe339e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9185e827-6j8kb"
Oct 30 13:04:18.141923 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:18.141872    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flbsg\" (UniqueName: \"kubernetes.io/projected/88815187-6e50-4b15-984a-ac4efe339e0e-kube-api-access-flbsg\") pod \"maas-api-revoke-sub-9185e827-6j8kb\" (UID: \"88815187-6e50-4b15-984a-ac4efe339e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9185e827-6j8kb"
Oct 30 13:04:18.142607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:18.142578    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88815187-6e50-4b15-984a-ac4efe339e0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9185e827-6j8kb\" (UID: \"88815187-6e50-4b15-984a-ac4efe339e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9185e827-6j8kb"
Oct 30 13:04:18.144371 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:18.144347    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88815187-6e50-4b15-984a-ac4efe339e0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9185e827-6j8kb\" (UID: \"88815187-6e50-4b15-984a-ac4efe339e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9185e827-6j8kb"
Oct 30 13:04:18.166481 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:18.166449    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flbsg\" (UniqueName: \"kubernetes.io/projected/88815187-6e50-4b15-984a-ac4efe339e0e-kube-api-access-flbsg\") pod \"maas-api-revoke-sub-9185e827-6j8kb\" (UID: \"88815187-6e50-4b15-984a-ac4efe339e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9185e827-6j8kb"
Oct 30 13:04:18.262305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:18.262267    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9185e827-6j8kb"
Oct 30 13:04:18.579351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:18.579320    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9185e827-6j8kb"]
Oct 30 13:04:18.582263 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:04:18.582231    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88815187_6e50_4b15_984a_ac4efe339e0e.slice/crio-759d9d9ceb3cbd0af8cd8e75d67cb237b5c5f6ab348c81578554bb2205ca7b8b WatchSource:0}: Error finding container 759d9d9ceb3cbd0af8cd8e75d67cb237b5c5f6ab348c81578554bb2205ca7b8b: Status 404 returned error can't find the container with id 759d9d9ceb3cbd0af8cd8e75d67cb237b5c5f6ab348c81578554bb2205ca7b8b
Oct 30 13:04:19.558316 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:19.558282    4544 generic.go:356] "Generic (PLEG): container finished" podID="88815187-6e50-4b15-984a-ac4efe339e0e" containerID="deb1ac6eb807798d16153741b602c257267cec2dba65e7412b62a97f85e8efa6" exitCode=0
Oct 30 13:04:19.558541 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:19.558368    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9185e827-6j8kb" event={"ID":"88815187-6e50-4b15-984a-ac4efe339e0e","Type":"ContainerDied","Data":"deb1ac6eb807798d16153741b602c257267cec2dba65e7412b62a97f85e8efa6"}
Oct 30 13:04:19.558541 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:19.558427    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9185e827-6j8kb" event={"ID":"88815187-6e50-4b15-984a-ac4efe339e0e","Type":"ContainerStarted","Data":"759d9d9ceb3cbd0af8cd8e75d67cb237b5c5f6ab348c81578554bb2205ca7b8b"}
Oct 30 13:04:19.887598 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:19.887532    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:20.722815 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:20.722784    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9185e827-6j8kb"
Oct 30 13:04:20.891736 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:20.891692    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88815187-6e50-4b15-984a-ac4efe339e0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88815187-6e50-4b15-984a-ac4efe339e0e-maas-api-openshift-ca\") pod \"88815187-6e50-4b15-984a-ac4efe339e0e\" (UID: \"88815187-6e50-4b15-984a-ac4efe339e0e\") "
Oct 30 13:04:20.892240 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:20.891749    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88815187-6e50-4b15-984a-ac4efe339e0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88815187-6e50-4b15-984a-ac4efe339e0e-maas-api-xks-ca\") pod \"88815187-6e50-4b15-984a-ac4efe339e0e\" (UID: \"88815187-6e50-4b15-984a-ac4efe339e0e\") "
Oct 30 13:04:20.892240 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:20.891800    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88815187-6e50-4b15-984a-ac4efe339e0e-kube-api-access-flbsg\" (UniqueName: \"kubernetes.io/projected/88815187-6e50-4b15-984a-ac4efe339e0e-kube-api-access-flbsg\") pod \"88815187-6e50-4b15-984a-ac4efe339e0e\" (UID: \"88815187-6e50-4b15-984a-ac4efe339e0e\") "
Oct 30 13:04:20.892240 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:20.892133    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88815187-6e50-4b15-984a-ac4efe339e0e-maas-api-openshift-ca" pod "88815187-6e50-4b15-984a-ac4efe339e0e" (UID: "88815187-6e50-4b15-984a-ac4efe339e0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:20.892524 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:20.892502    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88815187-6e50-4b15-984a-ac4efe339e0e-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:04:20.894157 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:20.894130    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88815187-6e50-4b15-984a-ac4efe339e0e-kube-api-access-flbsg" pod "88815187-6e50-4b15-984a-ac4efe339e0e" (UID: "88815187-6e50-4b15-984a-ac4efe339e0e"). InnerVolumeSpecName "kube-api-access-flbsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:20.894324 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:20.894298    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88815187-6e50-4b15-984a-ac4efe339e0e-maas-api-xks-ca" pod "88815187-6e50-4b15-984a-ac4efe339e0e" (UID: "88815187-6e50-4b15-984a-ac4efe339e0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:04:20.994443 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:20.994403    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88815187-6e50-4b15-984a-ac4efe339e0e-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:04:20.994443 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:20.994436    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flbsg\" (UniqueName: \"kubernetes.io/projected/88815187-6e50-4b15-984a-ac4efe339e0e-kube-api-access-flbsg\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:04:21.148790 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:21.148692    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c7d0d09-5wsrb"]
Oct 30 13:04:21.149503 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:21.149481    4544 cpu_manager.go:414] "containerMap: removing container" podUID="88815187-6e50-4b15-984a-ac4efe339e0e" containerName="revoke-keys"
Oct 30 13:04:21.149503 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:21.149503    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88815187-6e50-4b15-984a-ac4efe339e0e" containerName="revoke-keys"
Oct 30 13:04:21.149684 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:21.149670    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="88815187-6e50-4b15-984a-ac4efe339e0e" containerName="revoke-keys"
Oct 30 13:04:21.153672 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:21.153645    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7d0d09-5wsrb"
Oct 30 13:04:21.175754 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:21.175718    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c7d0d09-5wsrb"]
Oct 30 13:04:21.298974 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:21.298934    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b74c7aa0-c2d0-4da4-a64f-259f43155a74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c7d0d09-5wsrb\" (UID: \"b74c7aa0-c2d0-4da4-a64f-259f43155a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7d0d09-5wsrb"
Oct 30 13:04:21.299158 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:21.298996    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b74c7aa0-c2d0-4da4-a64f-259f43155a74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c7d0d09-5wsrb\" (UID: \"b74c7aa0-c2d0-4da4-a64f-259f43155a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7d0d09-5wsrb"
Oct 30 13:04:21.299158 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:21.299102    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d686q\" (UniqueName: \"kubernetes.io/projected/b74c7aa0-c2d0-4da4-a64f-259f43155a74-kube-api-access-d686q\") pod \"maas-api-revoke-sub-6c7d0d09-5wsrb\" (UID: \"b74c7aa0-c2d0-4da4-a64f-259f43155a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7d0d09-5wsrb"
Oct 30 13:04:21.400956 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:21.400851    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b74c7aa0-c2d0-4da4-a64f-259f43155a74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c7d0d09-5wsrb\" (UID: \"b74c7aa0-c2d0-4da4-a64f-259f43155a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7d0d09-5wsrb"
Oct 30 13:04:21.401138 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:21.400953    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b74c7aa0-c2d0-4da4-a64f-259f43155a74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c7d0d09-5wsrb\" (UID: \"b74c7aa0-c2d0-4da4-a64f-259f43155a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7d0d09-5wsrb"
Oct 30 13:04:21.401138 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:21.401018    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d686q\" (UniqueName: \"kubernetes.io/projected/b74c7aa0-c2d0-4da4-a64f-259f43155a74-kube-api-access-d686q\") pod \"maas-api-revoke-sub-6c7d0d09-5wsrb\" (UID: \"b74c7aa0-c2d0-4da4-a64f-259f43155a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7d0d09-5wsrb"
Oct 30 13:04:21.401645 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:21.401622    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b74c7aa0-c2d0-4da4-a64f-259f43155a74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c7d0d09-5wsrb\" (UID: \"b74c7aa0-c2d0-4da4-a64f-259f43155a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7d0d09-5wsrb"
Oct 30 13:04:21.403498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:21.403480    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b74c7aa0-c2d0-4da4-a64f-259f43155a74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c7d0d09-5wsrb\" (UID: \"b74c7aa0-c2d0-4da4-a64f-259f43155a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7d0d09-5wsrb"
Oct 30 13:04:21.425297 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:21.425265    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d686q\" (UniqueName: \"kubernetes.io/projected/b74c7aa0-c2d0-4da4-a64f-259f43155a74-kube-api-access-d686q\") pod \"maas-api-revoke-sub-6c7d0d09-5wsrb\" (UID: \"b74c7aa0-c2d0-4da4-a64f-259f43155a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7d0d09-5wsrb"
Oct 30 13:04:21.468330 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:21.468284    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7d0d09-5wsrb"
Oct 30 13:04:21.587280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:21.580512    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9185e827-6j8kb" event={"ID":"88815187-6e50-4b15-984a-ac4efe339e0e","Type":"ContainerDied","Data":"759d9d9ceb3cbd0af8cd8e75d67cb237b5c5f6ab348c81578554bb2205ca7b8b"}
Oct 30 13:04:21.587280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:21.580590    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="759d9d9ceb3cbd0af8cd8e75d67cb237b5c5f6ab348c81578554bb2205ca7b8b"
Oct 30 13:04:21.587280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:21.580669    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9185e827-6j8kb"
Oct 30 13:04:21.738456 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:21.738363    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:21.818905 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:04:21.818839    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb74c7aa0_c2d0_4da4_a64f_259f43155a74.slice/crio-d034b440e3e00690a4cd39274ff9f8acec1446899b73cab6d2c9350c7906c6a0 WatchSource:0}: Error finding container d034b440e3e00690a4cd39274ff9f8acec1446899b73cab6d2c9350c7906c6a0: Status 404 returned error can't find the container with id d034b440e3e00690a4cd39274ff9f8acec1446899b73cab6d2c9350c7906c6a0
Oct 30 13:04:21.879927 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:21.879887    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c7d0d09-5wsrb"]
Oct 30 13:04:22.590213 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:22.590176    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7d0d09-5wsrb" event={"ID":"b74c7aa0-c2d0-4da4-a64f-259f43155a74","Type":"ContainerStarted","Data":"f3b302a80dbbe2d99a9a505b3bf1b06f64e89ce0c895242ce5a2c307e248a79f"}
Oct 30 13:04:22.590740 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:22.590251    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7d0d09-5wsrb" event={"ID":"b74c7aa0-c2d0-4da4-a64f-259f43155a74","Type":"ContainerStarted","Data":"d034b440e3e00690a4cd39274ff9f8acec1446899b73cab6d2c9350c7906c6a0"}
Oct 30 13:04:22.631038 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:22.630983    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7d0d09-5wsrb" podStartSLOduration=1.630967099 podStartE2EDuration="1.630967099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:22.630441936 +0000 UTC m=+1157.677575890" watchObservedRunningTime="2026-10-30 13:04:22.630967099 +0000 UTC m=+1157.678101099"
Oct 30 13:04:22.677881 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:22.677838    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:22.829029 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:22.829001    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a3c14f0-tt7vj"]
Oct 30 13:04:22.833608 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:22.833586    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3c14f0-tt7vj"
Oct 30 13:04:22.838070 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:22.838043    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:22.858256 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:22.858147    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a3c14f0-tt7vj"]
Oct 30 13:04:22.927893 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:22.927854    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6143936-5028-4fab-bbf5-f61a445ed99b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a3c14f0-tt7vj\" (UID: \"a6143936-5028-4fab-bbf5-f61a445ed99b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3c14f0-tt7vj"
Oct 30 13:04:22.928118 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:22.928093    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kh26\" (UniqueName: \"kubernetes.io/projected/a6143936-5028-4fab-bbf5-f61a445ed99b-kube-api-access-6kh26\") pod \"maas-api-revoke-sub-8a3c14f0-tt7vj\" (UID: \"a6143936-5028-4fab-bbf5-f61a445ed99b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3c14f0-tt7vj"
Oct 30 13:04:22.928173 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:22.928157    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6143936-5028-4fab-bbf5-f61a445ed99b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a3c14f0-tt7vj\" (UID: \"a6143936-5028-4fab-bbf5-f61a445ed99b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3c14f0-tt7vj"
Oct 30 13:04:23.029771 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:23.029726    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kh26\" (UniqueName: \"kubernetes.io/projected/a6143936-5028-4fab-bbf5-f61a445ed99b-kube-api-access-6kh26\") pod \"maas-api-revoke-sub-8a3c14f0-tt7vj\" (UID: \"a6143936-5028-4fab-bbf5-f61a445ed99b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3c14f0-tt7vj"
Oct 30 13:04:23.029998 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:23.029810    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6143936-5028-4fab-bbf5-f61a445ed99b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a3c14f0-tt7vj\" (UID: \"a6143936-5028-4fab-bbf5-f61a445ed99b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3c14f0-tt7vj"
Oct 30 13:04:23.029998 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:23.029939    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6143936-5028-4fab-bbf5-f61a445ed99b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a3c14f0-tt7vj\" (UID: \"a6143936-5028-4fab-bbf5-f61a445ed99b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3c14f0-tt7vj"
Oct 30 13:04:23.030723 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:23.030698    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6143936-5028-4fab-bbf5-f61a445ed99b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a3c14f0-tt7vj\" (UID: \"a6143936-5028-4fab-bbf5-f61a445ed99b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3c14f0-tt7vj"
Oct 30 13:04:23.032581 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:23.032531    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6143936-5028-4fab-bbf5-f61a445ed99b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a3c14f0-tt7vj\" (UID: \"a6143936-5028-4fab-bbf5-f61a445ed99b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3c14f0-tt7vj"
Oct 30 13:04:23.062695 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:23.062654    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kh26\" (UniqueName: \"kubernetes.io/projected/a6143936-5028-4fab-bbf5-f61a445ed99b-kube-api-access-6kh26\") pod \"maas-api-revoke-sub-8a3c14f0-tt7vj\" (UID: \"a6143936-5028-4fab-bbf5-f61a445ed99b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3c14f0-tt7vj"
Oct 30 13:04:23.205922 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:23.205806    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3c14f0-tt7vj"
Oct 30 13:04:23.519046 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:23.519011    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a3c14f0-tt7vj"]
Oct 30 13:04:23.520309 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:04:23.520278    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6143936_5028_4fab_bbf5_f61a445ed99b.slice/crio-17c1bf01226e17b5c5c3868db67a2c1a84cd1e5177c5206f1bab0b3a2fd11542 WatchSource:0}: Error finding container 17c1bf01226e17b5c5c3868db67a2c1a84cd1e5177c5206f1bab0b3a2fd11542: Status 404 returned error can't find the container with id 17c1bf01226e17b5c5c3868db67a2c1a84cd1e5177c5206f1bab0b3a2fd11542
Oct 30 13:04:23.600676 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:23.600640    4544 generic.go:356] "Generic (PLEG): container finished" podID="b74c7aa0-c2d0-4da4-a64f-259f43155a74" containerID="f3b302a80dbbe2d99a9a505b3bf1b06f64e89ce0c895242ce5a2c307e248a79f" exitCode=0
Oct 30 13:04:23.601081 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:23.600720    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7d0d09-5wsrb" event={"ID":"b74c7aa0-c2d0-4da4-a64f-259f43155a74","Type":"ContainerDied","Data":"f3b302a80dbbe2d99a9a505b3bf1b06f64e89ce0c895242ce5a2c307e248a79f"}
Oct 30 13:04:23.602137 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:23.602111    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3c14f0-tt7vj" event={"ID":"a6143936-5028-4fab-bbf5-f61a445ed99b","Type":"ContainerStarted","Data":"17c1bf01226e17b5c5c3868db67a2c1a84cd1e5177c5206f1bab0b3a2fd11542"}
Oct 30 13:04:23.703233 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:23.703186    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:24.611421 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:24.611382    4544 generic.go:356] "Generic (PLEG): container finished" podID="a6143936-5028-4fab-bbf5-f61a445ed99b" containerID="a048be130c1f683a1ecf6ca91dbc1ba01d265d2d1b92216ac720836acbd99d6d" exitCode=0
Oct 30 13:04:24.611874 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:24.611463    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3c14f0-tt7vj" event={"ID":"a6143936-5028-4fab-bbf5-f61a445ed99b","Type":"ContainerDied","Data":"a048be130c1f683a1ecf6ca91dbc1ba01d265d2d1b92216ac720836acbd99d6d"}
Oct 30 13:04:24.789656 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:24.789587    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7d0d09-5wsrb"
Oct 30 13:04:24.874059 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:24.874016    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b74c7aa0-c2d0-4da4-a64f-259f43155a74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b74c7aa0-c2d0-4da4-a64f-259f43155a74-maas-api-xks-ca\") pod \"b74c7aa0-c2d0-4da4-a64f-259f43155a74\" (UID: \"b74c7aa0-c2d0-4da4-a64f-259f43155a74\") "
Oct 30 13:04:24.874278 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:24.874095    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b74c7aa0-c2d0-4da4-a64f-259f43155a74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b74c7aa0-c2d0-4da4-a64f-259f43155a74-maas-api-openshift-ca\") pod \"b74c7aa0-c2d0-4da4-a64f-259f43155a74\" (UID: \"b74c7aa0-c2d0-4da4-a64f-259f43155a74\") "
Oct 30 13:04:24.874278 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:24.874122    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b74c7aa0-c2d0-4da4-a64f-259f43155a74-kube-api-access-d686q\" (UniqueName: \"kubernetes.io/projected/b74c7aa0-c2d0-4da4-a64f-259f43155a74-kube-api-access-d686q\") pod \"b74c7aa0-c2d0-4da4-a64f-259f43155a74\" (UID: \"b74c7aa0-c2d0-4da4-a64f-259f43155a74\") "
Oct 30 13:04:24.874492 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:24.874466    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b74c7aa0-c2d0-4da4-a64f-259f43155a74-maas-api-openshift-ca" pod "b74c7aa0-c2d0-4da4-a64f-259f43155a74" (UID: "b74c7aa0-c2d0-4da4-a64f-259f43155a74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:24.874791 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:24.874769    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b74c7aa0-c2d0-4da4-a64f-259f43155a74-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:04:24.876563 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:24.876537    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b74c7aa0-c2d0-4da4-a64f-259f43155a74-maas-api-xks-ca" pod "b74c7aa0-c2d0-4da4-a64f-259f43155a74" (UID: "b74c7aa0-c2d0-4da4-a64f-259f43155a74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:04:24.876703 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:24.876685    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b74c7aa0-c2d0-4da4-a64f-259f43155a74-kube-api-access-d686q" pod "b74c7aa0-c2d0-4da4-a64f-259f43155a74" (UID: "b74c7aa0-c2d0-4da4-a64f-259f43155a74"). InnerVolumeSpecName "kube-api-access-d686q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:24.976236 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:24.976194    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b74c7aa0-c2d0-4da4-a64f-259f43155a74-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:04:24.976414 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:24.976272    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d686q\" (UniqueName: \"kubernetes.io/projected/b74c7aa0-c2d0-4da4-a64f-259f43155a74-kube-api-access-d686q\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:04:25.599161 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:25.599124    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:25.621105 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:25.621074    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7d0d09-5wsrb"
Oct 30 13:04:25.621539 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:25.621111    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7d0d09-5wsrb" event={"ID":"b74c7aa0-c2d0-4da4-a64f-259f43155a74","Type":"ContainerDied","Data":"d034b440e3e00690a4cd39274ff9f8acec1446899b73cab6d2c9350c7906c6a0"}
Oct 30 13:04:25.621539 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:25.621161    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d034b440e3e00690a4cd39274ff9f8acec1446899b73cab6d2c9350c7906c6a0"
Oct 30 13:04:26.008258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:26.008098    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3c14f0-tt7vj"
Oct 30 13:04:26.245334 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:26.245202    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6143936-5028-4fab-bbf5-f61a445ed99b-kube-api-access-6kh26\" (UniqueName: \"kubernetes.io/projected/a6143936-5028-4fab-bbf5-f61a445ed99b-kube-api-access-6kh26\") pod \"a6143936-5028-4fab-bbf5-f61a445ed99b\" (UID: \"a6143936-5028-4fab-bbf5-f61a445ed99b\") "
Oct 30 13:04:26.245613 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:26.245586    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6143936-5028-4fab-bbf5-f61a445ed99b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6143936-5028-4fab-bbf5-f61a445ed99b-maas-api-xks-ca\") pod \"a6143936-5028-4fab-bbf5-f61a445ed99b\" (UID: \"a6143936-5028-4fab-bbf5-f61a445ed99b\") "
Oct 30 13:04:26.245936 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:26.245909    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6143936-5028-4fab-bbf5-f61a445ed99b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6143936-5028-4fab-bbf5-f61a445ed99b-maas-api-openshift-ca\") pod \"a6143936-5028-4fab-bbf5-f61a445ed99b\" (UID: \"a6143936-5028-4fab-bbf5-f61a445ed99b\") "
Oct 30 13:04:26.248799 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:26.248764    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6143936-5028-4fab-bbf5-f61a445ed99b-maas-api-openshift-ca" pod "a6143936-5028-4fab-bbf5-f61a445ed99b" (UID: "a6143936-5028-4fab-bbf5-f61a445ed99b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:26.285659 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:26.285605    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6143936-5028-4fab-bbf5-f61a445ed99b-kube-api-access-6kh26" pod "a6143936-5028-4fab-bbf5-f61a445ed99b" (UID: "a6143936-5028-4fab-bbf5-f61a445ed99b"). InnerVolumeSpecName "kube-api-access-6kh26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:26.307242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:26.301599    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6143936-5028-4fab-bbf5-f61a445ed99b-maas-api-xks-ca" pod "a6143936-5028-4fab-bbf5-f61a445ed99b" (UID: "a6143936-5028-4fab-bbf5-f61a445ed99b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:04:26.348821 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:26.348767    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6143936-5028-4fab-bbf5-f61a445ed99b-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:04:26.348821 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:26.348814    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kh26\" (UniqueName: \"kubernetes.io/projected/a6143936-5028-4fab-bbf5-f61a445ed99b-kube-api-access-6kh26\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:04:26.348821 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:26.348831    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6143936-5028-4fab-bbf5-f61a445ed99b-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:04:26.696246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:26.691448    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3c14f0-tt7vj" event={"ID":"a6143936-5028-4fab-bbf5-f61a445ed99b","Type":"ContainerDied","Data":"17c1bf01226e17b5c5c3868db67a2c1a84cd1e5177c5206f1bab0b3a2fd11542"}
Oct 30 13:04:26.696246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:26.691525    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17c1bf01226e17b5c5c3868db67a2c1a84cd1e5177c5206f1bab0b3a2fd11542"
Oct 30 13:04:26.696246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:26.691605    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3c14f0-tt7vj"
Oct 30 13:04:27.700248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:27.699322    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:27.732508 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:27.732374    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 13:04:27.937381 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:27.937325    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:28.364370 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:28.364311    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:28.468557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:28.467346    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:28.712322 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:28.710488    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:28.712322 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:28.710906    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:04:28.735536 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:28.735494    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:28.740285 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:28.736494    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:04:28.939861 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:28.939799    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:28.953404 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:28.941015    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:04:29.368177 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:29.368118    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:29.370282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:29.370254    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 13:04:29.400999 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:29.400967    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 13:04:29.494910 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:29.494853    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:29.495695 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:29.495664    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:04:29.718525 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:29.718416    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:29.744452 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:04:29.723902    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:04:29.782244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:29.750702    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 13:04:29.782244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:29.771144    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dad92b6-bpxw8"]
Oct 30 13:04:29.782244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:29.772376    4544 cpu_manager.go:414] "containerMap: removing container" podUID="a6143936-5028-4fab-bbf5-f61a445ed99b" containerName="revoke-keys"
Oct 30 13:04:29.782244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:29.772400    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6143936-5028-4fab-bbf5-f61a445ed99b" containerName="revoke-keys"
Oct 30 13:04:29.782244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:29.772441    4544 cpu_manager.go:414] "containerMap: removing container" podUID="b74c7aa0-c2d0-4da4-a64f-259f43155a74" containerName="revoke-keys"
Oct 30 13:04:29.782244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:29.772450    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b74c7aa0-c2d0-4da4-a64f-259f43155a74" containerName="revoke-keys"
Oct 30 13:04:29.782244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:29.772626    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="b74c7aa0-c2d0-4da4-a64f-259f43155a74" containerName="revoke-keys"
Oct 30 13:04:29.782244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:29.772649    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6143936-5028-4fab-bbf5-f61a445ed99b" containerName="revoke-keys"
Oct 30 13:04:29.784991 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:29.784956    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dad92b6-bpxw8"]
Oct 30 13:04:29.785301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:29.785282    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad92b6-bpxw8"
Oct 30 13:04:29.796637 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:29.796606    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:04:29.799021 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:29.798927    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:04:29.802996 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:29.802973    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:04:29.805019 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:29.804999    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:04:29.805331 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:29.805312    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:04:29.817330 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:29.817292    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:04:29.904075 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:29.904038    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hdr2\" (UniqueName: \"kubernetes.io/projected/2928d5f2-e80e-4c43-9f56-6ff5d1b86759-kube-api-access-2hdr2\") pod \"maas-api-revoke-sub-0dad92b6-bpxw8\" (UID: \"2928d5f2-e80e-4c43-9f56-6ff5d1b86759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad92b6-bpxw8"
Oct 30 13:04:29.904356 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:29.904332    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2928d5f2-e80e-4c43-9f56-6ff5d1b86759-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dad92b6-bpxw8\" (UID: \"2928d5f2-e80e-4c43-9f56-6ff5d1b86759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad92b6-bpxw8"
Oct 30 13:04:29.904435 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:29.904376    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2928d5f2-e80e-4c43-9f56-6ff5d1b86759-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dad92b6-bpxw8\" (UID: \"2928d5f2-e80e-4c43-9f56-6ff5d1b86759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad92b6-bpxw8"
Oct 30 13:04:29.997856 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:29.997763    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:30.005616 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:30.005589    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hdr2\" (UniqueName: \"kubernetes.io/projected/2928d5f2-e80e-4c43-9f56-6ff5d1b86759-kube-api-access-2hdr2\") pod \"maas-api-revoke-sub-0dad92b6-bpxw8\" (UID: \"2928d5f2-e80e-4c43-9f56-6ff5d1b86759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad92b6-bpxw8"
Oct 30 13:04:30.006127 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:30.006093    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2928d5f2-e80e-4c43-9f56-6ff5d1b86759-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dad92b6-bpxw8\" (UID: \"2928d5f2-e80e-4c43-9f56-6ff5d1b86759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad92b6-bpxw8"
Oct 30 13:04:30.006127 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:30.006126    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2928d5f2-e80e-4c43-9f56-6ff5d1b86759-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dad92b6-bpxw8\" (UID: \"2928d5f2-e80e-4c43-9f56-6ff5d1b86759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad92b6-bpxw8"
Oct 30 13:04:30.006865 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:30.006837    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2928d5f2-e80e-4c43-9f56-6ff5d1b86759-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dad92b6-bpxw8\" (UID: \"2928d5f2-e80e-4c43-9f56-6ff5d1b86759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad92b6-bpxw8"
Oct 30 13:04:30.008840 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:30.008813    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2928d5f2-e80e-4c43-9f56-6ff5d1b86759-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dad92b6-bpxw8\" (UID: \"2928d5f2-e80e-4c43-9f56-6ff5d1b86759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad92b6-bpxw8"
Oct 30 13:04:30.039684 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:30.039645    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hdr2\" (UniqueName: \"kubernetes.io/projected/2928d5f2-e80e-4c43-9f56-6ff5d1b86759-kube-api-access-2hdr2\") pod \"maas-api-revoke-sub-0dad92b6-bpxw8\" (UID: \"2928d5f2-e80e-4c43-9f56-6ff5d1b86759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad92b6-bpxw8"
Oct 30 13:04:30.125722 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:30.125684    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad92b6-bpxw8"
Oct 30 13:04:30.544918 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:30.544891    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dad92b6-bpxw8"]
Oct 30 13:04:30.547419 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:04:30.547368    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2928d5f2_e80e_4c43_9f56_6ff5d1b86759.slice/crio-c3de067aa8c9d79b69e8768a7fb5539fae52b8010b119220b236222650723eb5 WatchSource:0}: Error finding container c3de067aa8c9d79b69e8768a7fb5539fae52b8010b119220b236222650723eb5: Status 404 returned error can't find the container with id c3de067aa8c9d79b69e8768a7fb5539fae52b8010b119220b236222650723eb5
Oct 30 13:04:30.958998 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:30.958955    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad92b6-bpxw8" event={"ID":"2928d5f2-e80e-4c43-9f56-6ff5d1b86759","Type":"ContainerStarted","Data":"8c252ad0a4cfef24a293d860b2ed60a068d2a1be7500423af51e94243bb68c97"}
Oct 30 13:04:30.959467 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:30.959019    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad92b6-bpxw8" event={"ID":"2928d5f2-e80e-4c43-9f56-6ff5d1b86759","Type":"ContainerStarted","Data":"c3de067aa8c9d79b69e8768a7fb5539fae52b8010b119220b236222650723eb5"}
Oct 30 13:04:31.005836 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:31.005784    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad92b6-bpxw8" podStartSLOduration=2.005769798 podStartE2EDuration="2.005769798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:31.001907248 +0000 UTC m=+1166.049041202" watchObservedRunningTime="2026-10-30 13:04:31.005769798 +0000 UTC m=+1166.052903752"
Oct 30 13:04:31.547835 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:31.547783    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3f4ca9d-48vtn"]
Oct 30 13:04:31.552541 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:31.552508    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f4ca9d-48vtn"
Oct 30 13:04:31.583332 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:31.583296    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3f4ca9d-48vtn"]
Oct 30 13:04:31.631249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:31.631135    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b82845b6-026b-4509-ac36-de4bee8e710b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3f4ca9d-48vtn\" (UID: \"b82845b6-026b-4509-ac36-de4bee8e710b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f4ca9d-48vtn"
Oct 30 13:04:31.631421 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:31.631334    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97q49\" (UniqueName: \"kubernetes.io/projected/b82845b6-026b-4509-ac36-de4bee8e710b-kube-api-access-97q49\") pod \"maas-api-revoke-sub-e3f4ca9d-48vtn\" (UID: \"b82845b6-026b-4509-ac36-de4bee8e710b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f4ca9d-48vtn"
Oct 30 13:04:31.631421 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:31.631412    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b82845b6-026b-4509-ac36-de4bee8e710b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3f4ca9d-48vtn\" (UID: \"b82845b6-026b-4509-ac36-de4bee8e710b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f4ca9d-48vtn"
Oct 30 13:04:31.733019 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:31.732960    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b82845b6-026b-4509-ac36-de4bee8e710b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3f4ca9d-48vtn\" (UID: \"b82845b6-026b-4509-ac36-de4bee8e710b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f4ca9d-48vtn"
Oct 30 13:04:31.733269 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:31.733082    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97q49\" (UniqueName: \"kubernetes.io/projected/b82845b6-026b-4509-ac36-de4bee8e710b-kube-api-access-97q49\") pod \"maas-api-revoke-sub-e3f4ca9d-48vtn\" (UID: \"b82845b6-026b-4509-ac36-de4bee8e710b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f4ca9d-48vtn"
Oct 30 13:04:31.733269 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:31.733139    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b82845b6-026b-4509-ac36-de4bee8e710b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3f4ca9d-48vtn\" (UID: \"b82845b6-026b-4509-ac36-de4bee8e710b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f4ca9d-48vtn"
Oct 30 13:04:31.733808 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:31.733782    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b82845b6-026b-4509-ac36-de4bee8e710b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3f4ca9d-48vtn\" (UID: \"b82845b6-026b-4509-ac36-de4bee8e710b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f4ca9d-48vtn"
Oct 30 13:04:31.735631 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:31.735607    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b82845b6-026b-4509-ac36-de4bee8e710b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3f4ca9d-48vtn\" (UID: \"b82845b6-026b-4509-ac36-de4bee8e710b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f4ca9d-48vtn"
Oct 30 13:04:31.758445 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:31.758413    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97q49\" (UniqueName: \"kubernetes.io/projected/b82845b6-026b-4509-ac36-de4bee8e710b-kube-api-access-97q49\") pod \"maas-api-revoke-sub-e3f4ca9d-48vtn\" (UID: \"b82845b6-026b-4509-ac36-de4bee8e710b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f4ca9d-48vtn"
Oct 30 13:04:31.867543 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:31.867508    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f4ca9d-48vtn"
Oct 30 13:04:31.913546 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:31.913505    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:32.068538 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:32.068504    4544 generic.go:356] "Generic (PLEG): container finished" podID="2928d5f2-e80e-4c43-9f56-6ff5d1b86759" containerID="8c252ad0a4cfef24a293d860b2ed60a068d2a1be7500423af51e94243bb68c97" exitCode=0
Oct 30 13:04:32.069018 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:32.068667    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad92b6-bpxw8" event={"ID":"2928d5f2-e80e-4c43-9f56-6ff5d1b86759","Type":"ContainerDied","Data":"8c252ad0a4cfef24a293d860b2ed60a068d2a1be7500423af51e94243bb68c97"}
Oct 30 13:04:32.219514 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:32.219448    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3f4ca9d-48vtn"]
Oct 30 13:04:32.220445 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:04:32.220398    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb82845b6_026b_4509_ac36_de4bee8e710b.slice/crio-7e4572432aefe662747fa20d8c3ba7120b7147a23aa0e39f2178fdf5aa6146e8 WatchSource:0}: Error finding container 7e4572432aefe662747fa20d8c3ba7120b7147a23aa0e39f2178fdf5aa6146e8: Status 404 returned error can't find the container with id 7e4572432aefe662747fa20d8c3ba7120b7147a23aa0e39f2178fdf5aa6146e8
Oct 30 13:04:33.090806 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:33.090762    4544 generic.go:356] "Generic (PLEG): container finished" podID="b82845b6-026b-4509-ac36-de4bee8e710b" containerID="4540eee2a30c7ea7d097659dbc328fb822ed67eaba79a0677b2a6158598e0b1f" exitCode=0
Oct 30 13:04:33.097504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:33.097465    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f4ca9d-48vtn" event={"ID":"b82845b6-026b-4509-ac36-de4bee8e710b","Type":"ContainerDied","Data":"4540eee2a30c7ea7d097659dbc328fb822ed67eaba79a0677b2a6158598e0b1f"}
Oct 30 13:04:33.097655 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:33.097525    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f4ca9d-48vtn" event={"ID":"b82845b6-026b-4509-ac36-de4bee8e710b","Type":"ContainerStarted","Data":"7e4572432aefe662747fa20d8c3ba7120b7147a23aa0e39f2178fdf5aa6146e8"}
Oct 30 13:04:33.277724 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:33.277688    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad92b6-bpxw8"
Oct 30 13:04:33.365951 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:33.365912    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2928d5f2-e80e-4c43-9f56-6ff5d1b86759-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2928d5f2-e80e-4c43-9f56-6ff5d1b86759-maas-api-openshift-ca\") pod \"2928d5f2-e80e-4c43-9f56-6ff5d1b86759\" (UID: \"2928d5f2-e80e-4c43-9f56-6ff5d1b86759\") "
Oct 30 13:04:33.366205 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:33.366179    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2928d5f2-e80e-4c43-9f56-6ff5d1b86759-kube-api-access-2hdr2\" (UniqueName: \"kubernetes.io/projected/2928d5f2-e80e-4c43-9f56-6ff5d1b86759-kube-api-access-2hdr2\") pod \"2928d5f2-e80e-4c43-9f56-6ff5d1b86759\" (UID: \"2928d5f2-e80e-4c43-9f56-6ff5d1b86759\") "
Oct 30 13:04:33.366342 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:33.366247    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2928d5f2-e80e-4c43-9f56-6ff5d1b86759-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2928d5f2-e80e-4c43-9f56-6ff5d1b86759-maas-api-xks-ca\") pod \"2928d5f2-e80e-4c43-9f56-6ff5d1b86759\" (UID: \"2928d5f2-e80e-4c43-9f56-6ff5d1b86759\") "
Oct 30 13:04:33.366403 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:33.366363    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2928d5f2-e80e-4c43-9f56-6ff5d1b86759-maas-api-openshift-ca" pod "2928d5f2-e80e-4c43-9f56-6ff5d1b86759" (UID: "2928d5f2-e80e-4c43-9f56-6ff5d1b86759"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:33.366959 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:33.366933    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2928d5f2-e80e-4c43-9f56-6ff5d1b86759-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:04:33.368564 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:33.368539    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2928d5f2-e80e-4c43-9f56-6ff5d1b86759-maas-api-xks-ca" pod "2928d5f2-e80e-4c43-9f56-6ff5d1b86759" (UID: "2928d5f2-e80e-4c43-9f56-6ff5d1b86759"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:04:33.368655 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:33.368593    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2928d5f2-e80e-4c43-9f56-6ff5d1b86759-kube-api-access-2hdr2" pod "2928d5f2-e80e-4c43-9f56-6ff5d1b86759" (UID: "2928d5f2-e80e-4c43-9f56-6ff5d1b86759"). InnerVolumeSpecName "kube-api-access-2hdr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:33.468367 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:33.468329    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hdr2\" (UniqueName: \"kubernetes.io/projected/2928d5f2-e80e-4c43-9f56-6ff5d1b86759-kube-api-access-2hdr2\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:04:33.468367 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:33.468363    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2928d5f2-e80e-4c43-9f56-6ff5d1b86759-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:04:33.904391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:33.904357    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:34.101432 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:34.101399    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad92b6-bpxw8"
Oct 30 13:04:34.101854 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:34.101431    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad92b6-bpxw8" event={"ID":"2928d5f2-e80e-4c43-9f56-6ff5d1b86759","Type":"ContainerDied","Data":"c3de067aa8c9d79b69e8768a7fb5539fae52b8010b119220b236222650723eb5"}
Oct 30 13:04:34.101854 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:34.101488    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3de067aa8c9d79b69e8768a7fb5539fae52b8010b119220b236222650723eb5"
Oct 30 13:04:34.268446 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:34.268418    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f4ca9d-48vtn"
Oct 30 13:04:34.391098 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:34.391065    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b82845b6-026b-4509-ac36-de4bee8e710b-kube-api-access-97q49\" (UniqueName: \"kubernetes.io/projected/b82845b6-026b-4509-ac36-de4bee8e710b-kube-api-access-97q49\") pod \"b82845b6-026b-4509-ac36-de4bee8e710b\" (UID: \"b82845b6-026b-4509-ac36-de4bee8e710b\") "
Oct 30 13:04:34.391296 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:34.391275    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b82845b6-026b-4509-ac36-de4bee8e710b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b82845b6-026b-4509-ac36-de4bee8e710b-maas-api-xks-ca\") pod \"b82845b6-026b-4509-ac36-de4bee8e710b\" (UID: \"b82845b6-026b-4509-ac36-de4bee8e710b\") "
Oct 30 13:04:34.391383 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:34.391316    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b82845b6-026b-4509-ac36-de4bee8e710b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b82845b6-026b-4509-ac36-de4bee8e710b-maas-api-openshift-ca\") pod \"b82845b6-026b-4509-ac36-de4bee8e710b\" (UID: \"b82845b6-026b-4509-ac36-de4bee8e710b\") "
Oct 30 13:04:34.391704 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:34.391668    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b82845b6-026b-4509-ac36-de4bee8e710b-maas-api-openshift-ca" pod "b82845b6-026b-4509-ac36-de4bee8e710b" (UID: "b82845b6-026b-4509-ac36-de4bee8e710b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:34.392035 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:34.392015    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b82845b6-026b-4509-ac36-de4bee8e710b-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:04:34.393431 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:34.393402    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b82845b6-026b-4509-ac36-de4bee8e710b-maas-api-xks-ca" pod "b82845b6-026b-4509-ac36-de4bee8e710b" (UID: "b82845b6-026b-4509-ac36-de4bee8e710b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:04:34.393431 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:34.393411    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b82845b6-026b-4509-ac36-de4bee8e710b-kube-api-access-97q49" pod "b82845b6-026b-4509-ac36-de4bee8e710b" (UID: "b82845b6-026b-4509-ac36-de4bee8e710b"). InnerVolumeSpecName "kube-api-access-97q49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:34.493159 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:34.493075    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b82845b6-026b-4509-ac36-de4bee8e710b-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:04:34.493159 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:34.493104    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97q49\" (UniqueName: \"kubernetes.io/projected/b82845b6-026b-4509-ac36-de4bee8e710b-kube-api-access-97q49\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:04:35.111021 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:35.110991    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f4ca9d-48vtn"
Oct 30 13:04:35.111470 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:35.111025    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f4ca9d-48vtn" event={"ID":"b82845b6-026b-4509-ac36-de4bee8e710b","Type":"ContainerDied","Data":"7e4572432aefe662747fa20d8c3ba7120b7147a23aa0e39f2178fdf5aa6146e8"}
Oct 30 13:04:35.111470 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:35.111082    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e4572432aefe662747fa20d8c3ba7120b7147a23aa0e39f2178fdf5aa6146e8"
Oct 30 13:04:35.989974 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:35.989936    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:37.851498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:37.851464    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:38.644065 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:38.644029    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9080-0372a6"]
Oct 30 13:04:38.644707 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:38.644693    4544 cpu_manager.go:414] "containerMap: removing container" podUID="2928d5f2-e80e-4c43-9f56-6ff5d1b86759" containerName="revoke-keys"
Oct 30 13:04:38.644765 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:38.644708    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2928d5f2-e80e-4c43-9f56-6ff5d1b86759" containerName="revoke-keys"
Oct 30 13:04:38.644765 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:38.644733    4544 cpu_manager.go:414] "containerMap: removing container" podUID="b82845b6-026b-4509-ac36-de4bee8e710b" containerName="revoke-keys"
Oct 30 13:04:38.644765 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:38.644738    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b82845b6-026b-4509-ac36-de4bee8e710b" containerName="revoke-keys"
Oct 30 13:04:38.644858 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:38.644840    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="2928d5f2-e80e-4c43-9f56-6ff5d1b86759" containerName="revoke-keys"
Oct 30 13:04:38.644858 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:38.644852    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="b82845b6-026b-4509-ac36-de4bee8e710b" containerName="revoke-keys"
Oct 30 13:04:38.649532 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:38.649510    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9080-0372a6"
Oct 30 13:04:38.671295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:38.671265    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9080-0372a6"]
Oct 30 13:04:38.676338 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:38.676305    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8xhdg\""
Oct 30 13:04:38.771401 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:38.771368    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv4vs\" (UniqueName: \"kubernetes.io/projected/a10ae01b-d031-41bb-b2c6-cb03ed054119-kube-api-access-lv4vs\") pod \"test-curl-9080-0372a6\" (UID: \"a10ae01b-d031-41bb-b2c6-cb03ed054119\") " pod="opendatahub/test-curl-9080-0372a6"
Oct 30 13:04:38.821588 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:38.821555    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d33cb1-9c6zf"]
Oct 30 13:04:38.826248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:38.826212    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d33cb1-9c6zf"
Oct 30 13:04:38.843097 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:38.843069    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:04:38.843270 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:38.843253    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:04:38.849476 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:38.849429    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d33cb1-9c6zf"]
Oct 30 13:04:38.876514 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:38.876476    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv4vs\" (UniqueName: \"kubernetes.io/projected/a10ae01b-d031-41bb-b2c6-cb03ed054119-kube-api-access-lv4vs\") pod \"test-curl-9080-0372a6\" (UID: \"a10ae01b-d031-41bb-b2c6-cb03ed054119\") " pod="opendatahub/test-curl-9080-0372a6"
Oct 30 13:04:38.900884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:38.900808    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv4vs\" (UniqueName: \"kubernetes.io/projected/a10ae01b-d031-41bb-b2c6-cb03ed054119-kube-api-access-lv4vs\") pod \"test-curl-9080-0372a6\" (UID: \"a10ae01b-d031-41bb-b2c6-cb03ed054119\") " pod="opendatahub/test-curl-9080-0372a6"
Oct 30 13:04:38.962816 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:38.962782    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9080-0372a6"
Oct 30 13:04:38.978052 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:38.978021    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1bb3236-c672-4abd-85f4-56f8ae6f6c65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59d33cb1-9c6zf\" (UID: \"e1bb3236-c672-4abd-85f4-56f8ae6f6c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d33cb1-9c6zf"
Oct 30 13:04:38.978155 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:38.978063    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dshtk\" (UniqueName: \"kubernetes.io/projected/e1bb3236-c672-4abd-85f4-56f8ae6f6c65-kube-api-access-dshtk\") pod \"maas-api-revoke-sub-59d33cb1-9c6zf\" (UID: \"e1bb3236-c672-4abd-85f4-56f8ae6f6c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d33cb1-9c6zf"
Oct 30 13:04:38.978264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:38.978235    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1bb3236-c672-4abd-85f4-56f8ae6f6c65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59d33cb1-9c6zf\" (UID: \"e1bb3236-c672-4abd-85f4-56f8ae6f6c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d33cb1-9c6zf"
Oct 30 13:04:39.080119 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:39.080080    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1bb3236-c672-4abd-85f4-56f8ae6f6c65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59d33cb1-9c6zf\" (UID: \"e1bb3236-c672-4abd-85f4-56f8ae6f6c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d33cb1-9c6zf"
Oct 30 13:04:39.080332 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:39.080133    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dshtk\" (UniqueName: \"kubernetes.io/projected/e1bb3236-c672-4abd-85f4-56f8ae6f6c65-kube-api-access-dshtk\") pod \"maas-api-revoke-sub-59d33cb1-9c6zf\" (UID: \"e1bb3236-c672-4abd-85f4-56f8ae6f6c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d33cb1-9c6zf"
Oct 30 13:04:39.080332 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:39.080183    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1bb3236-c672-4abd-85f4-56f8ae6f6c65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59d33cb1-9c6zf\" (UID: \"e1bb3236-c672-4abd-85f4-56f8ae6f6c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d33cb1-9c6zf"
Oct 30 13:04:39.081092 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:39.081068    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1bb3236-c672-4abd-85f4-56f8ae6f6c65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59d33cb1-9c6zf\" (UID: \"e1bb3236-c672-4abd-85f4-56f8ae6f6c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d33cb1-9c6zf"
Oct 30 13:04:39.084036 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:39.084005    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1bb3236-c672-4abd-85f4-56f8ae6f6c65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59d33cb1-9c6zf\" (UID: \"e1bb3236-c672-4abd-85f4-56f8ae6f6c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d33cb1-9c6zf"
Oct 30 13:04:39.103876 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:39.103849    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dshtk\" (UniqueName: \"kubernetes.io/projected/e1bb3236-c672-4abd-85f4-56f8ae6f6c65-kube-api-access-dshtk\") pod \"maas-api-revoke-sub-59d33cb1-9c6zf\" (UID: \"e1bb3236-c672-4abd-85f4-56f8ae6f6c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d33cb1-9c6zf"
Oct 30 13:04:39.140622 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:39.140594    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d33cb1-9c6zf"
Oct 30 13:04:39.279982 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:04:39.279950    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda10ae01b_d031_41bb_b2c6_cb03ed054119.slice/crio-7deeaaa59ee22c4e6d9d7d8876a0fe20db387d8f085d3ed4b1e08b8436a4db35 WatchSource:0}: Error finding container 7deeaaa59ee22c4e6d9d7d8876a0fe20db387d8f085d3ed4b1e08b8436a4db35: Status 404 returned error can't find the container with id 7deeaaa59ee22c4e6d9d7d8876a0fe20db387d8f085d3ed4b1e08b8436a4db35
Oct 30 13:04:39.280861 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:39.280836    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9080-0372a6"]
Oct 30 13:04:39.282517 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:39.282496    4544 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:04:39.453406 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:39.453375    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d33cb1-9c6zf"]
Oct 30 13:04:39.453595 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:04:39.453576    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1bb3236_c672_4abd_85f4_56f8ae6f6c65.slice/crio-c6a0d1c041af25e4a86d5cd5e29a398855d323dc99e3fa86a63135fcf5cc9c1b WatchSource:0}: Error finding container c6a0d1c041af25e4a86d5cd5e29a398855d323dc99e3fa86a63135fcf5cc9c1b: Status 404 returned error can't find the container with id c6a0d1c041af25e4a86d5cd5e29a398855d323dc99e3fa86a63135fcf5cc9c1b
Oct 30 13:04:39.777676 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:39.777591    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:40.194190 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:40.194151    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9080-0372a6" event={"ID":"a10ae01b-d031-41bb-b2c6-cb03ed054119","Type":"ContainerStarted","Data":"7deeaaa59ee22c4e6d9d7d8876a0fe20db387d8f085d3ed4b1e08b8436a4db35"}
Oct 30 13:04:40.197664 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:40.197626    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d33cb1-9c6zf" event={"ID":"e1bb3236-c672-4abd-85f4-56f8ae6f6c65","Type":"ContainerStarted","Data":"8a3c2b920964a86803c87c0ad8794f3389e0a9fc2e1cd77342b0ffa4965572c3"}
Oct 30 13:04:40.197820 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:40.197689    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d33cb1-9c6zf" event={"ID":"e1bb3236-c672-4abd-85f4-56f8ae6f6c65","Type":"ContainerStarted","Data":"c6a0d1c041af25e4a86d5cd5e29a398855d323dc99e3fa86a63135fcf5cc9c1b"}
Oct 30 13:04:40.239854 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:40.239798    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d33cb1-9c6zf" podStartSLOduration=2.239781774 podStartE2EDuration="2.239781774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:40.238186814 +0000 UTC m=+1175.285320769" watchObservedRunningTime="2026-10-30 13:04:40.239781774 +0000 UTC m=+1175.286915730"
Oct 30 13:04:41.214859 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:41.214826    4544 generic.go:356] "Generic (PLEG): container finished" podID="e1bb3236-c672-4abd-85f4-56f8ae6f6c65" containerID="8a3c2b920964a86803c87c0ad8794f3389e0a9fc2e1cd77342b0ffa4965572c3" exitCode=0
Oct 30 13:04:41.215255 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:41.214907    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d33cb1-9c6zf" event={"ID":"e1bb3236-c672-4abd-85f4-56f8ae6f6c65","Type":"ContainerDied","Data":"8a3c2b920964a86803c87c0ad8794f3389e0a9fc2e1cd77342b0ffa4965572c3"}
Oct 30 13:04:41.404124 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:41.404088    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:41.617560 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:41.617524    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:41.799336 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:41.799300    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:42.154618 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:42.154584    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:42.224999 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:42.224961    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9080-0372a6" event={"ID":"a10ae01b-d031-41bb-b2c6-cb03ed054119","Type":"ContainerStarted","Data":"9a86eadfa5add7418686f807c18f6e78fc46a319cbd036ef7a1fdf38c6a31c0c"}
Oct 30 13:04:42.286061 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:42.285945    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9080-0372a6" podStartSLOduration=2.290981404 podStartE2EDuration="4.285925442s" totalImagesPullingTime="1.994944038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:42.282548249 +0000 UTC m=+1177.329682201" watchObservedRunningTime="2026-10-30 13:04:42.285925442 +0000 UTC m=+1177.333059394"
Oct 30 13:04:42.387911 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:42.387884    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d33cb1-9c6zf"
Oct 30 13:04:42.433764 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:42.433696    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1bb3236-c672-4abd-85f4-56f8ae6f6c65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1bb3236-c672-4abd-85f4-56f8ae6f6c65-maas-api-openshift-ca\") pod \"e1bb3236-c672-4abd-85f4-56f8ae6f6c65\" (UID: \"e1bb3236-c672-4abd-85f4-56f8ae6f6c65\") "
Oct 30 13:04:42.433764 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:42.433747    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1bb3236-c672-4abd-85f4-56f8ae6f6c65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1bb3236-c672-4abd-85f4-56f8ae6f6c65-maas-api-xks-ca\") pod \"e1bb3236-c672-4abd-85f4-56f8ae6f6c65\" (UID: \"e1bb3236-c672-4abd-85f4-56f8ae6f6c65\") "
Oct 30 13:04:42.433957 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:42.433821    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1bb3236-c672-4abd-85f4-56f8ae6f6c65-kube-api-access-dshtk\" (UniqueName: \"kubernetes.io/projected/e1bb3236-c672-4abd-85f4-56f8ae6f6c65-kube-api-access-dshtk\") pod \"e1bb3236-c672-4abd-85f4-56f8ae6f6c65\" (UID: \"e1bb3236-c672-4abd-85f4-56f8ae6f6c65\") "
Oct 30 13:04:42.434045 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:42.434021    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1bb3236-c672-4abd-85f4-56f8ae6f6c65-maas-api-openshift-ca" pod "e1bb3236-c672-4abd-85f4-56f8ae6f6c65" (UID: "e1bb3236-c672-4abd-85f4-56f8ae6f6c65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:42.434334 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:42.434317    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1bb3236-c672-4abd-85f4-56f8ae6f6c65-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:04:42.435877 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:42.435849    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1bb3236-c672-4abd-85f4-56f8ae6f6c65-maas-api-xks-ca" pod "e1bb3236-c672-4abd-85f4-56f8ae6f6c65" (UID: "e1bb3236-c672-4abd-85f4-56f8ae6f6c65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:04:42.436096 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:42.436080    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1bb3236-c672-4abd-85f4-56f8ae6f6c65-kube-api-access-dshtk" pod "e1bb3236-c672-4abd-85f4-56f8ae6f6c65" (UID: "e1bb3236-c672-4abd-85f4-56f8ae6f6c65"). InnerVolumeSpecName "kube-api-access-dshtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:42.535491 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:42.535456    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1bb3236-c672-4abd-85f4-56f8ae6f6c65-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:04:42.535491 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:42.535492    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dshtk\" (UniqueName: \"kubernetes.io/projected/e1bb3236-c672-4abd-85f4-56f8ae6f6c65-kube-api-access-dshtk\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:04:43.235627 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:43.235602    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d33cb1-9c6zf"
Oct 30 13:04:43.235627 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:43.235609    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d33cb1-9c6zf" event={"ID":"e1bb3236-c672-4abd-85f4-56f8ae6f6c65","Type":"ContainerDied","Data":"c6a0d1c041af25e4a86d5cd5e29a398855d323dc99e3fa86a63135fcf5cc9c1b"}
Oct 30 13:04:43.236062 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:43.235666    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6a0d1c041af25e4a86d5cd5e29a398855d323dc99e3fa86a63135fcf5cc9c1b"
Oct 30 13:04:43.438474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:43.438382    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9080-0372a6"]
Oct 30 13:04:43.449159 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:43.449119    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9080-0372a6"]
Oct 30 13:04:43.527004 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:43.526970    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:43.701321 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:43.696585    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9080-0e5863"]
Oct 30 13:04:43.701321 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:43.697958    4544 cpu_manager.go:414] "containerMap: removing container" podUID="a10ae01b-d031-41bb-b2c6-cb03ed054119" containerName="test-curl-9080-0372a6"
Oct 30 13:04:43.701321 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:43.697981    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a10ae01b-d031-41bb-b2c6-cb03ed054119" containerName="test-curl-9080-0372a6"
Oct 30 13:04:43.701321 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:43.697996    4544 cpu_manager.go:414] "containerMap: removing container" podUID="e1bb3236-c672-4abd-85f4-56f8ae6f6c65" containerName="revoke-keys"
Oct 30 13:04:43.701321 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:43.698003    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1bb3236-c672-4abd-85f4-56f8ae6f6c65" containerName="revoke-keys"
Oct 30 13:04:43.701321 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:43.698191    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="a10ae01b-d031-41bb-b2c6-cb03ed054119" containerName="test-curl-9080-0372a6"
Oct 30 13:04:43.701321 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:43.698204    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1bb3236-c672-4abd-85f4-56f8ae6f6c65" containerName="revoke-keys"
Oct 30 13:04:43.730571 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:43.730543    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9080-0e5863"
Oct 30 13:04:43.796258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:43.780952    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8jwn\" (UniqueName: \"kubernetes.io/projected/f5148465-0d6d-4e8c-9b87-e7362225a6a5-kube-api-access-v8jwn\") pod \"test-curl-9080-0e5863\" (UID: \"f5148465-0d6d-4e8c-9b87-e7362225a6a5\") " pod="opendatahub/test-curl-9080-0e5863"
Oct 30 13:04:43.835940 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:43.835908    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9080-0e5863"]
Oct 30 13:04:43.882203 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:43.882172    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8jwn\" (UniqueName: \"kubernetes.io/projected/f5148465-0d6d-4e8c-9b87-e7362225a6a5-kube-api-access-v8jwn\") pod \"test-curl-9080-0e5863\" (UID: \"f5148465-0d6d-4e8c-9b87-e7362225a6a5\") " pod="opendatahub/test-curl-9080-0e5863"
Oct 30 13:04:43.906446 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:43.906417    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8jwn\" (UniqueName: \"kubernetes.io/projected/f5148465-0d6d-4e8c-9b87-e7362225a6a5-kube-api-access-v8jwn\") pod \"test-curl-9080-0e5863\" (UID: \"f5148465-0d6d-4e8c-9b87-e7362225a6a5\") " pod="opendatahub/test-curl-9080-0e5863"
Oct 30 13:04:44.125682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.125647    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9080-0e5863"
Oct 30 13:04:44.247329 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.247286    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9080-0372a6" podUID="a10ae01b-d031-41bb-b2c6-cb03ed054119" containerName="test-curl-9080-0372a6" containerID="cri-o://9a86eadfa5add7418686f807c18f6e78fc46a319cbd036ef7a1fdf38c6a31c0c" gracePeriod=2
Oct 30 13:04:44.256068 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.256044    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:44.261071 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:04:44.261043    4544 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9080-0372a6\" is forbidden: User \"system:node:ip-10-0-143-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-38.ec2.internal' and this object" podUID="a10ae01b-d031-41bb-b2c6-cb03ed054119" pod="opendatahub/test-curl-9080-0372a6"
Oct 30 13:04:44.442731 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.442638    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:44.462932 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.462907    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9080-0e5863"]
Oct 30 13:04:44.463785 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:04:44.463757    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5148465_0d6d_4e8c_9b87_e7362225a6a5.slice/crio-ccc84544ea0ec68bab1b58d78516d1a3d61dc2c792519e5a1bfcb31e4a8124dc WatchSource:0}: Error finding container ccc84544ea0ec68bab1b58d78516d1a3d61dc2c792519e5a1bfcb31e4a8124dc: Status 404 returned error can't find the container with id ccc84544ea0ec68bab1b58d78516d1a3d61dc2c792519e5a1bfcb31e4a8124dc
Oct 30 13:04:44.740335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.740255    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"]
Oct 30 13:04:44.745177 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.745152    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:04:44.765264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.765197    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-68573cef-openshift-default-dockercfg-7wzhj\""
Oct 30 13:04:44.782955 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.782927    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"]
Oct 30 13:04:44.808991 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.808959    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-credential-socket\") pod \"e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") " pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:04:44.809130 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.809072    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-istio-envoy\") pod \"e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") " pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:04:44.809200 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.809156    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgkf8\" (UniqueName: \"kubernetes.io/projected/78230db8-1d2b-4d35-b025-c54111d5bfd2-kube-api-access-cgkf8\") pod \"e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") " pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:04:44.809200 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.809191    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78230db8-1d2b-4d35-b025-c54111d5bfd2-istiod-ca-cert\") pod \"e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") " pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:04:44.809321 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.809243    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78230db8-1d2b-4d35-b025-c54111d5bfd2-istio-podinfo\") pod \"e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") " pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:04:44.809375 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.809324    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-workload-certs\") pod \"e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") " pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:04:44.809425 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.809380    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78230db8-1d2b-4d35-b025-c54111d5bfd2-istio-token\") pod \"e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") " pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:04:44.809474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.809449    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-workload-socket\") pod \"e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") " pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:04:44.809560 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.809543    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-istio-data\") pod \"e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") " pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:04:44.812709 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:04:44.812675    4544 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9080-0372a6\" is forbidden: User \"system:node:ip-10-0-143-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-38.ec2.internal' and this object" podUID="a10ae01b-d031-41bb-b2c6-cb03ed054119" pod="opendatahub/test-curl-9080-0372a6"
Oct 30 13:04:44.823269 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.823199    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:44.921232 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.914437    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-istio-envoy\") pod \"e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") " pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:04:44.921232 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.914528    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgkf8\" (UniqueName: \"kubernetes.io/projected/78230db8-1d2b-4d35-b025-c54111d5bfd2-kube-api-access-cgkf8\") pod \"e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") " pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:04:44.921232 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.914564    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78230db8-1d2b-4d35-b025-c54111d5bfd2-istiod-ca-cert\") pod \"e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") " pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:04:44.921232 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.914634    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78230db8-1d2b-4d35-b025-c54111d5bfd2-istio-podinfo\") pod \"e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") " pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:04:44.921232 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.914668    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-workload-certs\") pod \"e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") " pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:04:44.921232 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.914713    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78230db8-1d2b-4d35-b025-c54111d5bfd2-istio-token\") pod \"e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") " pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:04:44.921232 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.914825    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-workload-socket\") pod \"e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") " pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:04:44.921232 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.914936    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-istio-data\") pod \"e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") " pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:04:44.921232 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.914996    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-credential-socket\") pod \"e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") " pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:04:44.921232 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.915573    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-credential-socket\") pod \"e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") " pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:04:44.933651 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.932383    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-workload-certs\") pod \"e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") " pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:04:44.933651 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.933211    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78230db8-1d2b-4d35-b025-c54111d5bfd2-istiod-ca-cert\") pod \"e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") " pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:04:44.934681 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.934651    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-istio-data\") pod \"e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") " pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:04:44.934981 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.934962    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-workload-socket\") pod \"e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") " pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:04:44.961650 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.953983    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78230db8-1d2b-4d35-b025-c54111d5bfd2-istio-podinfo\") pod \"e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") " pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:04:44.984288 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.964477    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-istio-envoy\") pod \"e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") " pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:04:44.984288 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.978552    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78230db8-1d2b-4d35-b025-c54111d5bfd2-istio-token\") pod \"e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") " pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:04:45.001297 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:44.991868    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgkf8\" (UniqueName: \"kubernetes.io/projected/78230db8-1d2b-4d35-b025-c54111d5bfd2-kube-api-access-cgkf8\") pod \"e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") " pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:04:45.036258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:45.035585    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:45.070245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:45.069505    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:04:45.184247 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:04:45.181454    4544 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9080-0372a6\" is forbidden: User \"system:node:ip-10-0-143-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-38.ec2.internal' and this object" podUID="a10ae01b-d031-41bb-b2c6-cb03ed054119" pod="opendatahub/test-curl-9080-0372a6"
Oct 30 13:04:45.290306 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:45.287672    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9080-0e5863" event={"ID":"f5148465-0d6d-4e8c-9b87-e7362225a6a5","Type":"ContainerStarted","Data":"fd4a00ab1ca690cf2821cc0ae13c057ffc8f59dc642c902567531d952c617a1b"}
Oct 30 13:04:45.297253 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:45.292045    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9080-0e5863" event={"ID":"f5148465-0d6d-4e8c-9b87-e7362225a6a5","Type":"ContainerStarted","Data":"ccc84544ea0ec68bab1b58d78516d1a3d61dc2c792519e5a1bfcb31e4a8124dc"}
Oct 30 13:04:45.370239 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:45.369163    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9080-0e5863" podStartSLOduration=2.369143987 podStartE2EDuration="2.369143987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:45.3686623 +0000 UTC m=+1180.415796251" watchObservedRunningTime="2026-10-30 13:04:45.369143987 +0000 UTC m=+1180.416277941"
Oct 30 13:04:45.676944 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:45.671371    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"]
Oct 30 13:04:45.676944 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:45.671525    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:04:45.676944 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:45.671606    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:04:45.676944 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:45.671653    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:04:46.360185 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:46.360147    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s" event={"ID":"78230db8-1d2b-4d35-b025-c54111d5bfd2","Type":"ContainerStarted","Data":"7114a1bbff5bec10c67ee27609e2050975014ad8c423d168677467f74811f98f"}
Oct 30 13:04:46.391157 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:46.391115    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s" event={"ID":"78230db8-1d2b-4d35-b025-c54111d5bfd2","Type":"ContainerStarted","Data":"b9b2f2b78571d21414f7dcfbb77bcf5768d3662d25843da90f35d1d038cd4ea0"}
Oct 30 13:04:46.541244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:46.518493    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s" podStartSLOduration=2.518469998 podStartE2EDuration="2.518469998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:46.449497286 +0000 UTC m=+1181.496631240" watchObservedRunningTime="2026-10-30 13:04:46.518469998 +0000 UTC m=+1181.565603952"
Oct 30 13:04:47.071242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:47.070886    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:04:47.234248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:47.221187    4544 patch_prober.go:29] interesting pod/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 13:04:47.234248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:47.221284    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s" podUID="78230db8-1d2b-4d35-b025-c54111d5bfd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 13:04:47.404622 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:47.404572    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:47.408477 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:47.408446    4544 generic.go:356] "Generic (PLEG): container finished" podID="a10ae01b-d031-41bb-b2c6-cb03ed054119" containerID="9a86eadfa5add7418686f807c18f6e78fc46a319cbd036ef7a1fdf38c6a31c0c" exitCode=137
Oct 30 13:04:47.780803 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:47.768320    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:47.881252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:47.834344    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:47.888986 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:47.854354    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:47.888986 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:47.856080    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 13:04:48.080324 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:48.080290    4544 patch_prober.go:29] interesting pod/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 13:04:48.100695 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:48.100652    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s" podUID="78230db8-1d2b-4d35-b025-c54111d5bfd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 13:04:48.123248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:48.110627    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9080-0372a6"
Oct 30 13:04:48.240006 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:48.239959    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a10ae01b-d031-41bb-b2c6-cb03ed054119-kube-api-access-lv4vs\" (UniqueName: \"kubernetes.io/projected/a10ae01b-d031-41bb-b2c6-cb03ed054119-kube-api-access-lv4vs\") pod \"a10ae01b-d031-41bb-b2c6-cb03ed054119\" (UID: \"a10ae01b-d031-41bb-b2c6-cb03ed054119\") "
Oct 30 13:04:48.270380 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:48.246096    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10ae01b-d031-41bb-b2c6-cb03ed054119-kube-api-access-lv4vs" pod "a10ae01b-d031-41bb-b2c6-cb03ed054119" (UID: "a10ae01b-d031-41bb-b2c6-cb03ed054119"). InnerVolumeSpecName "kube-api-access-lv4vs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:48.271650 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:48.271611    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv4vs\" (UniqueName: \"kubernetes.io/projected/a10ae01b-d031-41bb-b2c6-cb03ed054119-kube-api-access-lv4vs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:04:48.580695 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:48.580650    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:48.674337 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:48.669160    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 13:04:48.901304 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:48.888419    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:48.901304 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:48.890979    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:04:48.901304 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:48.892798    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:48.901304 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:48.892887    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:48.901304 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:48.892928    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 13:04:49.078245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:49.071584    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:04:49.078245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:49.071687    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:04:49.078245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:49.071738    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:04:49.485922 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:49.485862    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:49.640636 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:49.640603    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9080-0372a6"
Oct 30 13:04:49.672616 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:49.672573    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:49.901265 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:49.894200    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 13:04:49.952282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:49.941687    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a10ae01b-d031-41bb-b2c6-cb03ed054119" path="/var/lib/kubelet/pods/a10ae01b-d031-41bb-b2c6-cb03ed054119/volumes"
Oct 30 13:04:49.972394 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:49.952815    4544 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:04:49.972753 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:49.972718    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g" podUID="75a99162-5892-4899-8487-071967579551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:50.102013 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:50.020822    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:04:50.102608 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:50.102584    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:04:50.133676 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:50.133641    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:04:50.381310 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:50.218447    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:50.425419 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:50.422789    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 13:04:50.444798 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:50.443759    4544 scope.go:122] "RemoveContainer" containerID="9a86eadfa5add7418686f807c18f6e78fc46a319cbd036ef7a1fdf38c6a31c0c"
Oct 30 13:04:50.471596 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:50.471521    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:50.592254 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:50.590344    4544 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:04:50.592254 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:50.590430    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz" podUID="1bf8120f-fb85-451c-922f-7e1cafe165df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:50.648315 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:50.627171    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:50.725564 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:50.686110    4544 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:04:50.725564 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:50.686167    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn" podUID="7f93aa79-27fa-4f24-9b20-b6da0dee9ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:50.725564 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:50.694770    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:04:50.958301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:50.925191    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:50.958301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:50.952965    4544 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:04:50.958301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:50.953031    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt" podUID="f8eb30f8-aa71-4121-9542-4f65aff15344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:50.977528 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:50.976087    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:51.251912 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:51.233138    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:51.476260 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:51.471366    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:04:51.480252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:51.476579    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:04:51.511235 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:51.501060    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:51.616868 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:51.615778    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:51.638314 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:51.634954    4544 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:04:51.638314 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:51.635021    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph" podUID="df98822e-ff7b-4c1d-ad2e-1e3b88523780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:51.704548 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:51.704508    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:51.726327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:51.725037    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:04:51.820673 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:51.820581    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:04:51.927206 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:51.927164    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 13:04:51.992246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:51.991969    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:52.154167 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.154127    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9080-0e5863"]
Oct 30 13:04:52.154374 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.154198    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9080-0e5863"]
Oct 30 13:04:52.154471 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.154387    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9080-0e5863" podUID="f5148465-0d6d-4e8c-9b87-e7362225a6a5" containerName="test-curl-9080-0e5863" containerID="cri-o://fd4a00ab1ca690cf2821cc0ae13c057ffc8f59dc642c902567531d952c617a1b" gracePeriod=2
Oct 30 13:04:52.198571 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:04:52.198504    4544 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9080-0e5863\" is forbidden: User \"system:node:ip-10-0-143-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-38.ec2.internal' and this object" podUID="f5148465-0d6d-4e8c-9b87-e7362225a6a5" pod="opendatahub/test-curl-9080-0e5863"
Oct 30 13:04:52.346107 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.346049    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"]
Oct 30 13:04:52.347467 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.347438    4544 cpu_manager.go:414] "containerMap: removing container" podUID="f5148465-0d6d-4e8c-9b87-e7362225a6a5" containerName="test-curl-9080-0e5863"
Oct 30 13:04:52.347615 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.347602    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5148465-0d6d-4e8c-9b87-e7362225a6a5" containerName="test-curl-9080-0e5863"
Oct 30 13:04:52.347892 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.347879    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5148465-0d6d-4e8c-9b87-e7362225a6a5" containerName="test-curl-9080-0e5863"
Oct 30 13:04:52.355554 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.354963    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:04:52.373310 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:04:52.371665    4544 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9080-0e5863\" is forbidden: User \"system:node:ip-10-0-143-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-38.ec2.internal' and this object" podUID="f5148465-0d6d-4e8c-9b87-e7362225a6a5" pod="opendatahub/test-curl-9080-0e5863"
Oct 30 13:04:52.375826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.375787    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-752d3a58-openshift-default-dockercfg-x2zkx\""
Oct 30 13:04:52.422341 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.422242    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-istio-envoy\") pod \"e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") " pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:04:52.422670 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.422640    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldqzn\" (UniqueName: \"kubernetes.io/projected/5d0c65ab-624c-40ab-9529-327c79d6567a-kube-api-access-ldqzn\") pod \"e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") " pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:04:52.422889 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.422868    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d0c65ab-624c-40ab-9529-327c79d6567a-istiod-ca-cert\") pod \"e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") " pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:04:52.423070 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.423052    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d0c65ab-624c-40ab-9529-327c79d6567a-istio-podinfo\") pod \"e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") " pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:04:52.423272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.423251    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d0c65ab-624c-40ab-9529-327c79d6567a-istio-token\") pod \"e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") " pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:04:52.423416 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.423400    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-workload-socket\") pod \"e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") " pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:04:52.423575 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.423556    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-credential-socket\") pod \"e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") " pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:04:52.423717 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.423702    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-istio-data\") pod \"e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") " pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:04:52.423840 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.423813    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-workload-certs\") pod \"e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") " pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:04:52.600259 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.567162    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-istio-data\") pod \"e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") " pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:04:52.600259 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.567295    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-workload-certs\") pod \"e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") " pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:04:52.600259 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.567441    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-istio-envoy\") pod \"e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") " pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:04:52.600259 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.567832    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldqzn\" (UniqueName: \"kubernetes.io/projected/5d0c65ab-624c-40ab-9529-327c79d6567a-kube-api-access-ldqzn\") pod \"e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") " pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:04:52.600259 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.567945    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d0c65ab-624c-40ab-9529-327c79d6567a-istiod-ca-cert\") pod \"e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") " pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:04:52.600259 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.568051    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d0c65ab-624c-40ab-9529-327c79d6567a-istio-podinfo\") pod \"e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") " pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:04:52.600259 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.568164    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d0c65ab-624c-40ab-9529-327c79d6567a-istio-token\") pod \"e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") " pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:04:52.600259 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.568203    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-workload-socket\") pod \"e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") " pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:04:52.600259 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.568282    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-credential-socket\") pod \"e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") " pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:04:52.600259 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.568977    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-credential-socket\") pod \"e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") " pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:04:52.650430 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.626877    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-istio-data\") pod \"e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") " pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:04:52.650430 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.627129    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-workload-certs\") pod \"e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") " pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:04:52.775192 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.773865    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-workload-socket\") pod \"e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") " pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:04:52.775192 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.775024    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:52.777950 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.777918    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d0c65ab-624c-40ab-9529-327c79d6567a-istiod-ca-cert\") pod \"e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") " pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:04:52.829680 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.829641    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"]
Oct 30 13:04:52.891287 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.889899    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e65256d4-89rzm"]
Oct 30 13:04:52.897283 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.894054    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d0c65ab-624c-40ab-9529-327c79d6567a-istio-token\") pod \"e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") " pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:04:52.897283 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.896626    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-istio-envoy\") pod \"e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") " pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:04:52.935637 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.904404    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65256d4-89rzm"
Oct 30 13:04:53.007248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:52.957444    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldqzn\" (UniqueName: \"kubernetes.io/projected/5d0c65ab-624c-40ab-9529-327c79d6567a-kube-api-access-ldqzn\") pod \"e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") " pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:04:53.166765 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.008253    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d0c65ab-624c-40ab-9529-327c79d6567a-istio-podinfo\") pod \"e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") " pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:04:53.167023 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.011900    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1334f7-9c63-429d-b0e0-b16dc4e8f690-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e65256d4-89rzm\" (UID: \"9f1334f7-9c63-429d-b0e0-b16dc4e8f690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65256d4-89rzm"
Oct 30 13:04:53.167460 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.167428    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65gb9\" (UniqueName: \"kubernetes.io/projected/9f1334f7-9c63-429d-b0e0-b16dc4e8f690-kube-api-access-65gb9\") pod \"maas-api-revoke-sub-e65256d4-89rzm\" (UID: \"9f1334f7-9c63-429d-b0e0-b16dc4e8f690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65256d4-89rzm"
Oct 30 13:04:53.167755 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.167733    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1334f7-9c63-429d-b0e0-b16dc4e8f690-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e65256d4-89rzm\" (UID: \"9f1334f7-9c63-429d-b0e0-b16dc4e8f690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65256d4-89rzm"
Oct 30 13:04:53.168065 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.012348    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:04:53.168187 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:04:53.070118    4544 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9080-0e5863\" is forbidden: User \"system:node:ip-10-0-143-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-38.ec2.internal' and this object" podUID="f5148465-0d6d-4e8c-9b87-e7362225a6a5" pod="opendatahub/test-curl-9080-0e5863"
Oct 30 13:04:53.168313 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.013181    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:04:53.187897 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.047952    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e65256d4-89rzm"]
Oct 30 13:04:53.188528 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.188503    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn"]
Oct 30 13:04:53.206982 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.205675    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:53.227637 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.227583    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn"
Oct 30 13:04:53.279189 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.279137    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65gb9\" (UniqueName: \"kubernetes.io/projected/9f1334f7-9c63-429d-b0e0-b16dc4e8f690-kube-api-access-65gb9\") pod \"maas-api-revoke-sub-e65256d4-89rzm\" (UID: \"9f1334f7-9c63-429d-b0e0-b16dc4e8f690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65256d4-89rzm"
Oct 30 13:04:53.279592 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.279567    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1334f7-9c63-429d-b0e0-b16dc4e8f690-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e65256d4-89rzm\" (UID: \"9f1334f7-9c63-429d-b0e0-b16dc4e8f690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65256d4-89rzm"
Oct 30 13:04:53.291272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.285649    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1334f7-9c63-429d-b0e0-b16dc4e8f690-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e65256d4-89rzm\" (UID: \"9f1334f7-9c63-429d-b0e0-b16dc4e8f690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65256d4-89rzm"
Oct 30 13:04:53.302243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.291694    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1334f7-9c63-429d-b0e0-b16dc4e8f690-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e65256d4-89rzm\" (UID: \"9f1334f7-9c63-429d-b0e0-b16dc4e8f690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65256d4-89rzm"
Oct 30 13:04:53.302627 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.302581    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48gmr\" (UniqueName: \"kubernetes.io/projected/b46598ca-1b3b-45b6-9f6d-3ba5edcc539d-kube-api-access-48gmr\") pod \"payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn\" (UID: \"b46598ca-1b3b-45b6-9f6d-3ba5edcc539d\") " pod="openshift-ingress/payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn"
Oct 30 13:04:53.302876 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.302855    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b46598ca-1b3b-45b6-9f6d-3ba5edcc539d-plugins-config-volume\") pod \"payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn\" (UID: \"b46598ca-1b3b-45b6-9f6d-3ba5edcc539d\") " pod="openshift-ingress/payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn"
Oct 30 13:04:53.315542 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.315503    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:04:53.328483 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:04:53.327840    4544 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9080-0e5863\" is forbidden: User \"system:node:ip-10-0-143-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-38.ec2.internal' and this object" podUID="f5148465-0d6d-4e8c-9b87-e7362225a6a5" pod="opendatahub/test-curl-9080-0e5863"
Oct 30 13:04:53.404575 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.328065    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-68573cef-dockercfg-xfvf4\""
Oct 30 13:04:53.404575 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.328148    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-68573cef\""
Oct 30 13:04:53.412796 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.412766    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1334f7-9c63-429d-b0e0-b16dc4e8f690-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e65256d4-89rzm\" (UID: \"9f1334f7-9c63-429d-b0e0-b16dc4e8f690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65256d4-89rzm"
Oct 30 13:04:53.421639 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.419357    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48gmr\" (UniqueName: \"kubernetes.io/projected/b46598ca-1b3b-45b6-9f6d-3ba5edcc539d-kube-api-access-48gmr\") pod \"payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn\" (UID: \"b46598ca-1b3b-45b6-9f6d-3ba5edcc539d\") " pod="openshift-ingress/payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn"
Oct 30 13:04:53.421639 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.419433    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b46598ca-1b3b-45b6-9f6d-3ba5edcc539d-plugins-config-volume\") pod \"payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn\" (UID: \"b46598ca-1b3b-45b6-9f6d-3ba5edcc539d\") " pod="openshift-ingress/payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn"
Oct 30 13:04:53.421639 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.420628    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b46598ca-1b3b-45b6-9f6d-3ba5edcc539d-plugins-config-volume\") pod \"payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn\" (UID: \"b46598ca-1b3b-45b6-9f6d-3ba5edcc539d\") " pod="openshift-ingress/payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn"
Oct 30 13:04:53.440173 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.438633    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb"]
Oct 30 13:04:53.457667 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.453695    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn"]
Oct 30 13:04:53.457667 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.453751    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb"]
Oct 30 13:04:53.457667 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.453863    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb"
Oct 30 13:04:53.457667 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.454713    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48gmr\" (UniqueName: \"kubernetes.io/projected/b46598ca-1b3b-45b6-9f6d-3ba5edcc539d-kube-api-access-48gmr\") pod \"payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn\" (UID: \"b46598ca-1b3b-45b6-9f6d-3ba5edcc539d\") " pod="openshift-ingress/payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn"
Oct 30 13:04:53.498602 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.461203    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65gb9\" (UniqueName: \"kubernetes.io/projected/9f1334f7-9c63-429d-b0e0-b16dc4e8f690-kube-api-access-65gb9\") pod \"maas-api-revoke-sub-e65256d4-89rzm\" (UID: \"9f1334f7-9c63-429d-b0e0-b16dc4e8f690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65256d4-89rzm"
Oct 30 13:04:53.499514 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.499249    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn"
Oct 30 13:04:53.499888 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.499857    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65256d4-89rzm"
Oct 30 13:04:53.576361 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.576312    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:53.624389 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.624347    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbf8b\" (UniqueName: \"kubernetes.io/projected/489f7620-9d5f-40e9-9225-1aee1181fb5c-kube-api-access-wbf8b\") pod \"payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb\" (UID: \"489f7620-9d5f-40e9-9225-1aee1181fb5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb"
Oct 30 13:04:53.625564 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.625533    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489f7620-9d5f-40e9-9225-1aee1181fb5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb\" (UID: \"489f7620-9d5f-40e9-9225-1aee1181fb5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb"
Oct 30 13:04:53.727907 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.727044    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489f7620-9d5f-40e9-9225-1aee1181fb5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb\" (UID: \"489f7620-9d5f-40e9-9225-1aee1181fb5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb"
Oct 30 13:04:53.727907 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.727142    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbf8b\" (UniqueName: \"kubernetes.io/projected/489f7620-9d5f-40e9-9225-1aee1181fb5c-kube-api-access-wbf8b\") pod \"payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb\" (UID: \"489f7620-9d5f-40e9-9225-1aee1181fb5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb"
Oct 30 13:04:53.727907 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.727856    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489f7620-9d5f-40e9-9225-1aee1181fb5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb\" (UID: \"489f7620-9d5f-40e9-9225-1aee1181fb5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb"
Oct 30 13:04:53.762980 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.762942    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbf8b\" (UniqueName: \"kubernetes.io/projected/489f7620-9d5f-40e9-9225-1aee1181fb5c-kube-api-access-wbf8b\") pod \"payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb\" (UID: \"489f7620-9d5f-40e9-9225-1aee1181fb5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb"
Oct 30 13:04:53.820742 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.820695    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb"
Oct 30 13:04:53.927296 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.927260    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"]
Oct 30 13:04:53.946058 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:04:53.946029    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d0c65ab_624c_40ab_9529_327c79d6567a.slice/crio-c588b67a810e4152c54e735bf4816f5f5b6227682d077584cc631cc23e2cdd42 WatchSource:0}: Error finding container c588b67a810e4152c54e735bf4816f5f5b6227682d077584cc631cc23e2cdd42: Status 404 returned error can't find the container with id c588b67a810e4152c54e735bf4816f5f5b6227682d077584cc631cc23e2cdd42
Oct 30 13:04:53.949451 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.949405    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:04:53.949537 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.949509    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:04:53.949585 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:53.949558    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:04:54.002643 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:54.002611    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e65256d4-89rzm"]
Oct 30 13:04:54.008840 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:04:54.008800    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f1334f7_9c63_429d_b0e0_b16dc4e8f690.slice/crio-eeeee0e53e8a855132ad027f9b04fd43c439859aaec680587fc44947fd087c90 WatchSource:0}: Error finding container eeeee0e53e8a855132ad027f9b04fd43c439859aaec680587fc44947fd087c90: Status 404 returned error can't find the container with id eeeee0e53e8a855132ad027f9b04fd43c439859aaec680587fc44947fd087c90
Oct 30 13:04:54.020155 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:54.020128    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn"]
Oct 30 13:04:54.022820 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:04:54.022790    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb46598ca_1b3b_45b6_9f6d_3ba5edcc539d.slice/crio-2928d550205a8b1acd15ccf111e834e7480937d831535380d0a3dbbc29e970c1 WatchSource:0}: Error finding container 2928d550205a8b1acd15ccf111e834e7480937d831535380d0a3dbbc29e970c1: Status 404 returned error can't find the container with id 2928d550205a8b1acd15ccf111e834e7480937d831535380d0a3dbbc29e970c1
Oct 30 13:04:54.136355 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:54.136252    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn" event={"ID":"b46598ca-1b3b-45b6-9f6d-3ba5edcc539d","Type":"ContainerStarted","Data":"1732ef6b1eee696f7205ddf69b7ce54454b9289f64fc376bd7be427c087da148"}
Oct 30 13:04:54.136355 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:54.136320    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn" event={"ID":"b46598ca-1b3b-45b6-9f6d-3ba5edcc539d","Type":"ContainerStarted","Data":"2928d550205a8b1acd15ccf111e834e7480937d831535380d0a3dbbc29e970c1"}
Oct 30 13:04:54.136612 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:54.136589    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn"
Oct 30 13:04:54.138273 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:54.138248    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz" event={"ID":"5d0c65ab-624c-40ab-9529-327c79d6567a","Type":"ContainerStarted","Data":"a099f064cb205f4a7ec204b4cda78f73f91353fdcf3caaea5bb9e26fd02db3ab"}
Oct 30 13:04:54.138413 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:54.138294    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz" event={"ID":"5d0c65ab-624c-40ab-9529-327c79d6567a","Type":"ContainerStarted","Data":"c588b67a810e4152c54e735bf4816f5f5b6227682d077584cc631cc23e2cdd42"}
Oct 30 13:04:54.139969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:54.139943    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65256d4-89rzm" event={"ID":"9f1334f7-9c63-429d-b0e0-b16dc4e8f690","Type":"ContainerStarted","Data":"eeeee0e53e8a855132ad027f9b04fd43c439859aaec680587fc44947fd087c90"}
Oct 30 13:04:54.272385 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:54.272331    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn" podStartSLOduration=2.27231627 podStartE2EDuration="2.27231627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:54.267420391 +0000 UTC m=+1189.314554340" watchObservedRunningTime="2026-10-30 13:04:54.27231627 +0000 UTC m=+1189.319450223"
Oct 30 13:04:54.279878 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:54.279848    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb"]
Oct 30 13:04:54.282055 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:04:54.282028    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod489f7620_9d5f_40e9_9225_1aee1181fb5c.slice/crio-61431b1246c05f8add3aeae0c9bbad0a4591c3d3ae48a230595d9104e85f8c4d WatchSource:0}: Error finding container 61431b1246c05f8add3aeae0c9bbad0a4591c3d3ae48a230595d9104e85f8c4d: Status 404 returned error can't find the container with id 61431b1246c05f8add3aeae0c9bbad0a4591c3d3ae48a230595d9104e85f8c4d
Oct 30 13:04:54.318749 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:54.318699    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:54.340024 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:54.336637    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:04:54.405489 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:54.405440    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz" podStartSLOduration=2.405423321 podStartE2EDuration="2.405423321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:54.403165831 +0000 UTC m=+1189.450299809" watchObservedRunningTime="2026-10-30 13:04:54.405423321 +0000 UTC m=+1189.452557274"
Oct 30 13:04:54.421946 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:54.421702    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9080-0e5863"
Oct 30 13:04:54.544890 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:54.544853    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5148465-0d6d-4e8c-9b87-e7362225a6a5-kube-api-access-v8jwn\" (UniqueName: \"kubernetes.io/projected/f5148465-0d6d-4e8c-9b87-e7362225a6a5-kube-api-access-v8jwn\") pod \"f5148465-0d6d-4e8c-9b87-e7362225a6a5\" (UID: \"f5148465-0d6d-4e8c-9b87-e7362225a6a5\") "
Oct 30 13:04:54.548154 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:54.548118    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5148465-0d6d-4e8c-9b87-e7362225a6a5-kube-api-access-v8jwn" pod "f5148465-0d6d-4e8c-9b87-e7362225a6a5" (UID: "f5148465-0d6d-4e8c-9b87-e7362225a6a5"). InnerVolumeSpecName "kube-api-access-v8jwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:54.654242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:54.653683    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8jwn\" (UniqueName: \"kubernetes.io/projected/f5148465-0d6d-4e8c-9b87-e7362225a6a5-kube-api-access-v8jwn\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:04:55.084191 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:55.084132    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:55.088319 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:55.088285    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5148465-0d6d-4e8c-9b87-e7362225a6a5" path="/var/lib/kubelet/pods/f5148465-0d6d-4e8c-9b87-e7362225a6a5/volumes"
Oct 30 13:04:55.150509 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:55.150457    4544 generic.go:356] "Generic (PLEG): container finished" podID="f5148465-0d6d-4e8c-9b87-e7362225a6a5" containerID="fd4a00ab1ca690cf2821cc0ae13c057ffc8f59dc642c902567531d952c617a1b" exitCode=137
Oct 30 13:04:55.150702 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:55.150519    4544 scope.go:122] "RemoveContainer" containerID="fd4a00ab1ca690cf2821cc0ae13c057ffc8f59dc642c902567531d952c617a1b"
Oct 30 13:04:55.150702 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:55.150531    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9080-0e5863"
Oct 30 13:04:55.154013 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:55.153981    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb" event={"ID":"489f7620-9d5f-40e9-9225-1aee1181fb5c","Type":"ContainerStarted","Data":"4eba388de30e57ec5b13b7d61caf3ebaf5b7547aa275ba2302d58a70d5dd1bce"}
Oct 30 13:04:55.154278 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:55.154044    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb" event={"ID":"489f7620-9d5f-40e9-9225-1aee1181fb5c","Type":"ContainerStarted","Data":"61431b1246c05f8add3aeae0c9bbad0a4591c3d3ae48a230595d9104e85f8c4d"}
Oct 30 13:04:55.154867 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:55.154365    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb"
Oct 30 13:04:55.162597 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:55.162560    4544 generic.go:356] "Generic (PLEG): container finished" podID="9f1334f7-9c63-429d-b0e0-b16dc4e8f690" containerID="3718035008b80be809c9330b72c1f334c74ea3d4db68be5e51605d5e149a855f" exitCode=0
Oct 30 13:04:55.162839 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:55.162645    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65256d4-89rzm" event={"ID":"9f1334f7-9c63-429d-b0e0-b16dc4e8f690","Type":"ContainerDied","Data":"3718035008b80be809c9330b72c1f334c74ea3d4db68be5e51605d5e149a855f"}
Oct 30 13:04:55.165558 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:55.165535    4544 scope.go:122] "RemoveContainer" containerID="fd4a00ab1ca690cf2821cc0ae13c057ffc8f59dc642c902567531d952c617a1b"
Oct 30 13:04:55.165957 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:04:55.165929    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd4a00ab1ca690cf2821cc0ae13c057ffc8f59dc642c902567531d952c617a1b\": container with ID starting with fd4a00ab1ca690cf2821cc0ae13c057ffc8f59dc642c902567531d952c617a1b not found: ID does not exist" containerID="fd4a00ab1ca690cf2821cc0ae13c057ffc8f59dc642c902567531d952c617a1b"
Oct 30 13:04:55.166048 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:55.165968    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd4a00ab1ca690cf2821cc0ae13c057ffc8f59dc642c902567531d952c617a1b"} err="failed to get container status \"fd4a00ab1ca690cf2821cc0ae13c057ffc8f59dc642c902567531d952c617a1b\": rpc error: code = NotFound desc = could not find container \"fd4a00ab1ca690cf2821cc0ae13c057ffc8f59dc642c902567531d952c617a1b\": container with ID starting with fd4a00ab1ca690cf2821cc0ae13c057ffc8f59dc642c902567531d952c617a1b not found: ID does not exist"
Oct 30 13:04:55.258359 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:55.258302    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb" podStartSLOduration=3.258285593 podStartE2EDuration="3.258285593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:55.255113625 +0000 UTC m=+1190.302247580" watchObservedRunningTime="2026-10-30 13:04:55.258285593 +0000 UTC m=+1190.305419546"
Oct 30 13:04:55.342657 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:55.342625    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:04:55.813175 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:55.813119    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:56.173321 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:56.172913    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:04:56.174032 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:56.173996    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:04:56.290101 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:56.290056    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:56.368895 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:56.368868    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65256d4-89rzm"
Oct 30 13:04:56.395673 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:56.395637    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f1334f7-9c63-429d-b0e0-b16dc4e8f690-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1334f7-9c63-429d-b0e0-b16dc4e8f690-maas-api-openshift-ca\") pod \"9f1334f7-9c63-429d-b0e0-b16dc4e8f690\" (UID: \"9f1334f7-9c63-429d-b0e0-b16dc4e8f690\") "
Oct 30 13:04:56.395854 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:56.395723    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f1334f7-9c63-429d-b0e0-b16dc4e8f690-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1334f7-9c63-429d-b0e0-b16dc4e8f690-maas-api-xks-ca\") pod \"9f1334f7-9c63-429d-b0e0-b16dc4e8f690\" (UID: \"9f1334f7-9c63-429d-b0e0-b16dc4e8f690\") "
Oct 30 13:04:56.395854 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:56.395759    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f1334f7-9c63-429d-b0e0-b16dc4e8f690-kube-api-access-65gb9\" (UniqueName: \"kubernetes.io/projected/9f1334f7-9c63-429d-b0e0-b16dc4e8f690-kube-api-access-65gb9\") pod \"9f1334f7-9c63-429d-b0e0-b16dc4e8f690\" (UID: \"9f1334f7-9c63-429d-b0e0-b16dc4e8f690\") "
Oct 30 13:04:56.395988 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:56.395973    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f1334f7-9c63-429d-b0e0-b16dc4e8f690-maas-api-openshift-ca" pod "9f1334f7-9c63-429d-b0e0-b16dc4e8f690" (UID: "9f1334f7-9c63-429d-b0e0-b16dc4e8f690"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:56.396337 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:56.396319    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1334f7-9c63-429d-b0e0-b16dc4e8f690-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:04:56.397888 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:56.397862    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f1334f7-9c63-429d-b0e0-b16dc4e8f690-maas-api-xks-ca" pod "9f1334f7-9c63-429d-b0e0-b16dc4e8f690" (UID: "9f1334f7-9c63-429d-b0e0-b16dc4e8f690"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:04:56.398013 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:56.397993    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f1334f7-9c63-429d-b0e0-b16dc4e8f690-kube-api-access-65gb9" pod "9f1334f7-9c63-429d-b0e0-b16dc4e8f690" (UID: "9f1334f7-9c63-429d-b0e0-b16dc4e8f690"). InnerVolumeSpecName "kube-api-access-65gb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:56.497782 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:56.497696    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1334f7-9c63-429d-b0e0-b16dc4e8f690-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:04:56.497782 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:56.497727    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65gb9\" (UniqueName: \"kubernetes.io/projected/9f1334f7-9c63-429d-b0e0-b16dc4e8f690-kube-api-access-65gb9\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:04:56.543107 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:56.543067    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:57.182885 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:57.182853    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65256d4-89rzm"
Oct 30 13:04:57.183355 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:57.182855    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65256d4-89rzm" event={"ID":"9f1334f7-9c63-429d-b0e0-b16dc4e8f690","Type":"ContainerDied","Data":"eeeee0e53e8a855132ad027f9b04fd43c439859aaec680587fc44947fd087c90"}
Oct 30 13:04:57.183355 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:57.183018    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eeeee0e53e8a855132ad027f9b04fd43c439859aaec680587fc44947fd087c90"
Oct 30 13:04:57.659646 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:57.659612    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0a2cb4b-pt66h"]
Oct 30 13:04:57.660327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:57.660312    4544 cpu_manager.go:414] "containerMap: removing container" podUID="9f1334f7-9c63-429d-b0e0-b16dc4e8f690" containerName="revoke-keys"
Oct 30 13:04:57.660385 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:57.660328    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f1334f7-9c63-429d-b0e0-b16dc4e8f690" containerName="revoke-keys"
Oct 30 13:04:57.660466 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:57.660457    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f1334f7-9c63-429d-b0e0-b16dc4e8f690" containerName="revoke-keys"
Oct 30 13:04:57.664571 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:57.664550    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a2cb4b-pt66h"
Oct 30 13:04:57.716156 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:57.716121    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/012f6e7f-8187-4fe2-a2d1-1ae080efed14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0a2cb4b-pt66h\" (UID: \"012f6e7f-8187-4fe2-a2d1-1ae080efed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a2cb4b-pt66h"
Oct 30 13:04:57.716377 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:57.716267    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhh5j\" (UniqueName: \"kubernetes.io/projected/012f6e7f-8187-4fe2-a2d1-1ae080efed14-kube-api-access-rhh5j\") pod \"maas-api-revoke-sub-b0a2cb4b-pt66h\" (UID: \"012f6e7f-8187-4fe2-a2d1-1ae080efed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a2cb4b-pt66h"
Oct 30 13:04:57.716377 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:57.716360    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/012f6e7f-8187-4fe2-a2d1-1ae080efed14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0a2cb4b-pt66h\" (UID: \"012f6e7f-8187-4fe2-a2d1-1ae080efed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a2cb4b-pt66h"
Oct 30 13:04:57.716623 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:57.716601    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0a2cb4b-pt66h"]
Oct 30 13:04:57.718732 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:57.718705    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:04:57.718732 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:57.718712    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:04:57.817340 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:57.817303    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/012f6e7f-8187-4fe2-a2d1-1ae080efed14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0a2cb4b-pt66h\" (UID: \"012f6e7f-8187-4fe2-a2d1-1ae080efed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a2cb4b-pt66h"
Oct 30 13:04:57.817535 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:57.817405    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhh5j\" (UniqueName: \"kubernetes.io/projected/012f6e7f-8187-4fe2-a2d1-1ae080efed14-kube-api-access-rhh5j\") pod \"maas-api-revoke-sub-b0a2cb4b-pt66h\" (UID: \"012f6e7f-8187-4fe2-a2d1-1ae080efed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a2cb4b-pt66h"
Oct 30 13:04:57.817535 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:57.817499    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/012f6e7f-8187-4fe2-a2d1-1ae080efed14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0a2cb4b-pt66h\" (UID: \"012f6e7f-8187-4fe2-a2d1-1ae080efed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a2cb4b-pt66h"
Oct 30 13:04:57.818028 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:57.818008    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/012f6e7f-8187-4fe2-a2d1-1ae080efed14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0a2cb4b-pt66h\" (UID: \"012f6e7f-8187-4fe2-a2d1-1ae080efed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a2cb4b-pt66h"
Oct 30 13:04:57.819862 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:57.819839    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/012f6e7f-8187-4fe2-a2d1-1ae080efed14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0a2cb4b-pt66h\" (UID: \"012f6e7f-8187-4fe2-a2d1-1ae080efed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a2cb4b-pt66h"
Oct 30 13:04:57.848418 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:57.848384    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhh5j\" (UniqueName: \"kubernetes.io/projected/012f6e7f-8187-4fe2-a2d1-1ae080efed14-kube-api-access-rhh5j\") pod \"maas-api-revoke-sub-b0a2cb4b-pt66h\" (UID: \"012f6e7f-8187-4fe2-a2d1-1ae080efed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a2cb4b-pt66h"
Oct 30 13:04:57.978554 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:57.978471    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a2cb4b-pt66h"
Oct 30 13:04:58.312032 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:58.312003    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0a2cb4b-pt66h"]
Oct 30 13:04:58.314478 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:04:58.314447    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod012f6e7f_8187_4fe2_a2d1_1ae080efed14.slice/crio-3d722da5e39e4fbfd5a4fdaab10bc75f8191184d862751fb5b7bd471a01ce28d WatchSource:0}: Error finding container 3d722da5e39e4fbfd5a4fdaab10bc75f8191184d862751fb5b7bd471a01ce28d: Status 404 returned error can't find the container with id 3d722da5e39e4fbfd5a4fdaab10bc75f8191184d862751fb5b7bd471a01ce28d
Oct 30 13:04:58.485773 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:58.485738    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:58.590281 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:58.590209    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p"]
Oct 30 13:04:58.596491 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:58.596457    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p"
Oct 30 13:04:58.614547 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:58.614508    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-752d3a58\""
Oct 30 13:04:58.627155 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:58.627116    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p"]
Oct 30 13:04:58.630352 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:58.630305    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsprd\" (UniqueName: \"kubernetes.io/projected/53b7be9d-ff23-452d-b121-12500f7d6787-kube-api-access-dsprd\") pod \"payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p\" (UID: \"53b7be9d-ff23-452d-b121-12500f7d6787\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p"
Oct 30 13:04:58.630565 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:58.630388    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53b7be9d-ff23-452d-b121-12500f7d6787-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p\" (UID: \"53b7be9d-ff23-452d-b121-12500f7d6787\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p"
Oct 30 13:04:58.675054 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:58.675011    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x"]
Oct 30 13:04:58.681611 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:58.681582    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x"
Oct 30 13:04:58.706028 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:58.705997    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-752d3a58-dockercfg-xp544\""
Oct 30 13:04:58.706563 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:58.706536    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x"]
Oct 30 13:04:58.731749 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:58.731691    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsprd\" (UniqueName: \"kubernetes.io/projected/53b7be9d-ff23-452d-b121-12500f7d6787-kube-api-access-dsprd\") pod \"payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p\" (UID: \"53b7be9d-ff23-452d-b121-12500f7d6787\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p"
Oct 30 13:04:58.732029 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:58.731758    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a68695-2f4b-4a06-938e-56e656c5c445-plugins-config-volume\") pod \"payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x\" (UID: \"e4a68695-2f4b-4a06-938e-56e656c5c445\") " pod="openshift-ingress/payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x"
Oct 30 13:04:58.732029 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:58.731842    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53b7be9d-ff23-452d-b121-12500f7d6787-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p\" (UID: \"53b7be9d-ff23-452d-b121-12500f7d6787\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p"
Oct 30 13:04:58.732328 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:58.732084    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7fw8\" (UniqueName: \"kubernetes.io/projected/e4a68695-2f4b-4a06-938e-56e656c5c445-kube-api-access-h7fw8\") pod \"payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x\" (UID: \"e4a68695-2f4b-4a06-938e-56e656c5c445\") " pod="openshift-ingress/payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x"
Oct 30 13:04:58.733730 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:58.733702    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53b7be9d-ff23-452d-b121-12500f7d6787-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p\" (UID: \"53b7be9d-ff23-452d-b121-12500f7d6787\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p"
Oct 30 13:04:58.869544 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:58.833072    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsprd\" (UniqueName: \"kubernetes.io/projected/53b7be9d-ff23-452d-b121-12500f7d6787-kube-api-access-dsprd\") pod \"payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p\" (UID: \"53b7be9d-ff23-452d-b121-12500f7d6787\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p"
Oct 30 13:04:58.869544 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:58.856741    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7fw8\" (UniqueName: \"kubernetes.io/projected/e4a68695-2f4b-4a06-938e-56e656c5c445-kube-api-access-h7fw8\") pod \"payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x\" (UID: \"e4a68695-2f4b-4a06-938e-56e656c5c445\") " pod="openshift-ingress/payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x"
Oct 30 13:04:58.869544 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:58.856842    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a68695-2f4b-4a06-938e-56e656c5c445-plugins-config-volume\") pod \"payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x\" (UID: \"e4a68695-2f4b-4a06-938e-56e656c5c445\") " pod="openshift-ingress/payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x"
Oct 30 13:04:58.869544 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:58.859297    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a68695-2f4b-4a06-938e-56e656c5c445-plugins-config-volume\") pod \"payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x\" (UID: \"e4a68695-2f4b-4a06-938e-56e656c5c445\") " pod="openshift-ingress/payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x"
Oct 30 13:04:58.869544 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:58.869305    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:58.924888 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:58.924523    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p"
Oct 30 13:04:59.051055 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:59.029260    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7fw8\" (UniqueName: \"kubernetes.io/projected/e4a68695-2f4b-4a06-938e-56e656c5c445-kube-api-access-h7fw8\") pod \"payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x\" (UID: \"e4a68695-2f4b-4a06-938e-56e656c5c445\") " pod="openshift-ingress/payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x"
Oct 30 13:04:59.166709 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:04:59.166628    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:04:59.173565 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:59.173500    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:04:59.209434 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:59.208034    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a2cb4b-pt66h" event={"ID":"012f6e7f-8187-4fe2-a2d1-1ae080efed14","Type":"ContainerStarted","Data":"e0cdd19d36d34ea35df556a8c71170cf6d9d0640f987d06c3ffb5c1b2f33f4dd"}
Oct 30 13:04:59.209434 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:59.208106    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a2cb4b-pt66h" event={"ID":"012f6e7f-8187-4fe2-a2d1-1ae080efed14","Type":"ContainerStarted","Data":"3d722da5e39e4fbfd5a4fdaab10bc75f8191184d862751fb5b7bd471a01ce28d"}
Oct 30 13:04:59.266570 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:59.266512    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a2cb4b-pt66h" podStartSLOduration=2.266497629 podStartE2EDuration="2.266497629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:59.261098564 +0000 UTC m=+1194.308232518" watchObservedRunningTime="2026-10-30 13:04:59.266497629 +0000 UTC m=+1194.313631582"
Oct 30 13:04:59.298287 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:59.298242    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x"
Oct 30 13:04:59.492370 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:04:59.492334    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53b7be9d_ff23_452d_b121_12500f7d6787.slice/crio-8d572d7a7f2378cee869889a669f3fa4196ed607d814def5caa4a7c236ae05bd WatchSource:0}: Error finding container 8d572d7a7f2378cee869889a669f3fa4196ed607d814def5caa4a7c236ae05bd: Status 404 returned error can't find the container with id 8d572d7a7f2378cee869889a669f3fa4196ed607d814def5caa4a7c236ae05bd
Oct 30 13:04:59.492710 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:59.492684    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p"]
Oct 30 13:04:59.622679 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:04:59.622652    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4a68695_2f4b_4a06_938e_56e656c5c445.slice/crio-42aa7bbe7fda9262ee35147bc6db6ea6f33674bfe28fbd20759405a79d89d6dd WatchSource:0}: Error finding container 42aa7bbe7fda9262ee35147bc6db6ea6f33674bfe28fbd20759405a79d89d6dd: Status 404 returned error can't find the container with id 42aa7bbe7fda9262ee35147bc6db6ea6f33674bfe28fbd20759405a79d89d6dd
Oct 30 13:04:59.623499 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:04:59.623472    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x"]
Oct 30 13:05:00.219811 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:00.219706    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x" event={"ID":"e4a68695-2f4b-4a06-938e-56e656c5c445","Type":"ContainerStarted","Data":"7128f05c6b3fe7f1c4bd1a7e30b5fee27ac37d4fdcbbe481a874d1e1e75edf21"}
Oct 30 13:05:00.219811 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:00.219776    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x" event={"ID":"e4a68695-2f4b-4a06-938e-56e656c5c445","Type":"ContainerStarted","Data":"42aa7bbe7fda9262ee35147bc6db6ea6f33674bfe28fbd20759405a79d89d6dd"}
Oct 30 13:05:00.219811 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:00.219802    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x"
Oct 30 13:05:00.221530 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:00.221505    4544 generic.go:356] "Generic (PLEG): container finished" podID="012f6e7f-8187-4fe2-a2d1-1ae080efed14" containerID="e0cdd19d36d34ea35df556a8c71170cf6d9d0640f987d06c3ffb5c1b2f33f4dd" exitCode=0
Oct 30 13:05:00.221636 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:00.221580    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a2cb4b-pt66h" event={"ID":"012f6e7f-8187-4fe2-a2d1-1ae080efed14","Type":"ContainerDied","Data":"e0cdd19d36d34ea35df556a8c71170cf6d9d0640f987d06c3ffb5c1b2f33f4dd"}
Oct 30 13:05:00.223432 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:00.223407    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p" event={"ID":"53b7be9d-ff23-452d-b121-12500f7d6787","Type":"ContainerStarted","Data":"1dae6ab7ec66e79def5da0ba16d54d583ed9b92aae6db49e3d49ff6093fd4def"}
Oct 30 13:05:00.223567 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:00.223450    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p" event={"ID":"53b7be9d-ff23-452d-b121-12500f7d6787","Type":"ContainerStarted","Data":"8d572d7a7f2378cee869889a669f3fa4196ed607d814def5caa4a7c236ae05bd"}
Oct 30 13:05:00.223567 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:00.223507    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p"
Oct 30 13:05:00.298357 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:00.298299    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x" podStartSLOduration=2.298284151 podStartE2EDuration="2.298284151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:00.291266726 +0000 UTC m=+1195.338400680" watchObservedRunningTime="2026-10-30 13:05:00.298284151 +0000 UTC m=+1195.345418104"
Oct 30 13:05:00.395509 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:00.395453    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p" podStartSLOduration=2.395436353 podStartE2EDuration="2.395436353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:00.392059131 +0000 UTC m=+1195.439193084" watchObservedRunningTime="2026-10-30 13:05:00.395436353 +0000 UTC m=+1195.442570362"
Oct 30 13:05:01.242117 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:01.242088    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:01.427960 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:01.427925    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a2cb4b-pt66h"
Oct 30 13:05:01.452664 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:01.452631    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/012f6e7f-8187-4fe2-a2d1-1ae080efed14-kube-api-access-rhh5j\" (UniqueName: \"kubernetes.io/projected/012f6e7f-8187-4fe2-a2d1-1ae080efed14-kube-api-access-rhh5j\") pod \"012f6e7f-8187-4fe2-a2d1-1ae080efed14\" (UID: \"012f6e7f-8187-4fe2-a2d1-1ae080efed14\") "
Oct 30 13:05:01.452836 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:01.452806    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/012f6e7f-8187-4fe2-a2d1-1ae080efed14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/012f6e7f-8187-4fe2-a2d1-1ae080efed14-maas-api-openshift-ca\") pod \"012f6e7f-8187-4fe2-a2d1-1ae080efed14\" (UID: \"012f6e7f-8187-4fe2-a2d1-1ae080efed14\") "
Oct 30 13:05:01.452874 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:01.452853    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/012f6e7f-8187-4fe2-a2d1-1ae080efed14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/012f6e7f-8187-4fe2-a2d1-1ae080efed14-maas-api-xks-ca\") pod \"012f6e7f-8187-4fe2-a2d1-1ae080efed14\" (UID: \"012f6e7f-8187-4fe2-a2d1-1ae080efed14\") "
Oct 30 13:05:01.453193 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:01.453162    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/012f6e7f-8187-4fe2-a2d1-1ae080efed14-maas-api-openshift-ca" pod "012f6e7f-8187-4fe2-a2d1-1ae080efed14" (UID: "012f6e7f-8187-4fe2-a2d1-1ae080efed14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:01.454811 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:01.454784    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012f6e7f-8187-4fe2-a2d1-1ae080efed14-kube-api-access-rhh5j" pod "012f6e7f-8187-4fe2-a2d1-1ae080efed14" (UID: "012f6e7f-8187-4fe2-a2d1-1ae080efed14"). InnerVolumeSpecName "kube-api-access-rhh5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:01.454900 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:01.454812    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/012f6e7f-8187-4fe2-a2d1-1ae080efed14-maas-api-xks-ca" pod "012f6e7f-8187-4fe2-a2d1-1ae080efed14" (UID: "012f6e7f-8187-4fe2-a2d1-1ae080efed14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:05:01.555400 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:01.555300    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhh5j\" (UniqueName: \"kubernetes.io/projected/012f6e7f-8187-4fe2-a2d1-1ae080efed14-kube-api-access-rhh5j\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:05:01.555400 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:01.555338    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/012f6e7f-8187-4fe2-a2d1-1ae080efed14-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:05:01.555400 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:01.555353    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/012f6e7f-8187-4fe2-a2d1-1ae080efed14-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:05:02.248330 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:02.248291    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a2cb4b-pt66h" event={"ID":"012f6e7f-8187-4fe2-a2d1-1ae080efed14","Type":"ContainerDied","Data":"3d722da5e39e4fbfd5a4fdaab10bc75f8191184d862751fb5b7bd471a01ce28d"}
Oct 30 13:05:02.248330 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:02.248313    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a2cb4b-pt66h"
Oct 30 13:05:02.248763 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:02.248347    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d722da5e39e4fbfd5a4fdaab10bc75f8191184d862751fb5b7bd471a01ce28d"
Oct 30 13:05:02.811340 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:02.811308    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-096c8434-8xk57"]
Oct 30 13:05:02.812051 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:02.812035    4544 cpu_manager.go:414] "containerMap: removing container" podUID="012f6e7f-8187-4fe2-a2d1-1ae080efed14" containerName="revoke-keys"
Oct 30 13:05:02.812051 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:02.812051    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="012f6e7f-8187-4fe2-a2d1-1ae080efed14" containerName="revoke-keys"
Oct 30 13:05:02.812169 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:02.812159    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="012f6e7f-8187-4fe2-a2d1-1ae080efed14" containerName="revoke-keys"
Oct 30 13:05:02.816275 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:02.816253    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-096c8434-8xk57"
Oct 30 13:05:02.836210 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:02.836185    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:05:02.846880 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:02.846857    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:05:02.876265 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:02.876209    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmk6n\" (UniqueName: \"kubernetes.io/projected/47a963c9-3243-4afd-a05d-e35e024ee5b6-kube-api-access-vmk6n\") pod \"maas-api-revoke-sub-096c8434-8xk57\" (UID: \"47a963c9-3243-4afd-a05d-e35e024ee5b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096c8434-8xk57"
Oct 30 13:05:02.876433 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:02.876313    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47a963c9-3243-4afd-a05d-e35e024ee5b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-096c8434-8xk57\" (UID: \"47a963c9-3243-4afd-a05d-e35e024ee5b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096c8434-8xk57"
Oct 30 13:05:02.876562 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:02.876542    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47a963c9-3243-4afd-a05d-e35e024ee5b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-096c8434-8xk57\" (UID: \"47a963c9-3243-4afd-a05d-e35e024ee5b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096c8434-8xk57"
Oct 30 13:05:02.937243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:02.936519    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-096c8434-8xk57"]
Oct 30 13:05:02.978297 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:02.978254    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmk6n\" (UniqueName: \"kubernetes.io/projected/47a963c9-3243-4afd-a05d-e35e024ee5b6-kube-api-access-vmk6n\") pod \"maas-api-revoke-sub-096c8434-8xk57\" (UID: \"47a963c9-3243-4afd-a05d-e35e024ee5b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096c8434-8xk57"
Oct 30 13:05:02.978297 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:02.978298    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47a963c9-3243-4afd-a05d-e35e024ee5b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-096c8434-8xk57\" (UID: \"47a963c9-3243-4afd-a05d-e35e024ee5b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096c8434-8xk57"
Oct 30 13:05:02.978526 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:02.978514    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47a963c9-3243-4afd-a05d-e35e024ee5b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-096c8434-8xk57\" (UID: \"47a963c9-3243-4afd-a05d-e35e024ee5b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096c8434-8xk57"
Oct 30 13:05:02.979385 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:02.979358    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47a963c9-3243-4afd-a05d-e35e024ee5b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-096c8434-8xk57\" (UID: \"47a963c9-3243-4afd-a05d-e35e024ee5b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096c8434-8xk57"
Oct 30 13:05:02.980780 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:02.980752    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47a963c9-3243-4afd-a05d-e35e024ee5b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-096c8434-8xk57\" (UID: \"47a963c9-3243-4afd-a05d-e35e024ee5b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096c8434-8xk57"
Oct 30 13:05:03.004494 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:03.004468    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmk6n\" (UniqueName: \"kubernetes.io/projected/47a963c9-3243-4afd-a05d-e35e024ee5b6-kube-api-access-vmk6n\") pod \"maas-api-revoke-sub-096c8434-8xk57\" (UID: \"47a963c9-3243-4afd-a05d-e35e024ee5b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096c8434-8xk57"
Oct 30 13:05:03.131590 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:03.131555    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-096c8434-8xk57"
Oct 30 13:05:03.263399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:03.263286    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:03.479369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:03.479341    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-096c8434-8xk57"]
Oct 30 13:05:03.481167 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:05:03.481133    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47a963c9_3243_4afd_a05d_e35e024ee5b6.slice/crio-a1d9927edc323d2462e09caee8b99937b790bb3bf66512cc0d2c7992ae02611b WatchSource:0}: Error finding container a1d9927edc323d2462e09caee8b99937b790bb3bf66512cc0d2c7992ae02611b: Status 404 returned error can't find the container with id a1d9927edc323d2462e09caee8b99937b790bb3bf66512cc0d2c7992ae02611b
Oct 30 13:05:04.398851 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:04.398792    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-096c8434-8xk57" event={"ID":"47a963c9-3243-4afd-a05d-e35e024ee5b6","Type":"ContainerStarted","Data":"0f3b461daf9ac3ec80b1668f1ba2e968d4a9b92fe608fe1811096f1644d26c34"}
Oct 30 13:05:04.404384 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:04.404341    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-096c8434-8xk57" event={"ID":"47a963c9-3243-4afd-a05d-e35e024ee5b6","Type":"ContainerStarted","Data":"a1d9927edc323d2462e09caee8b99937b790bb3bf66512cc0d2c7992ae02611b"}
Oct 30 13:05:05.548251 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:05.544697    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn"
Oct 30 13:05:05.787711 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:05.772288    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-096c8434-8xk57" podStartSLOduration=3.7722637580000002 podStartE2EDuration="3.772263758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:04.479988846 +0000 UTC m=+1199.527122796" watchObservedRunningTime="2026-10-30 13:05:05.772263758 +0000 UTC m=+1200.819397716"
Oct 30 13:05:05.996254 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:05.992778    4544 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:05:05.996254 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:05.992851    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt" podUID="f8eb30f8-aa71-4121-9542-4f65aff15344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:06.339670 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:06.339611    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:06.465261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:06.439698    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:06.511262 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:06.489505    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:06.511262 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:06.507450    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb"
Oct 30 13:05:06.583248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:06.579945    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:06.976447 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:06.948513    4544 generic.go:356] "Generic (PLEG): container finished" podID="47a963c9-3243-4afd-a05d-e35e024ee5b6" containerID="0f3b461daf9ac3ec80b1668f1ba2e968d4a9b92fe608fe1811096f1644d26c34" exitCode=0
Oct 30 13:05:06.976447 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:06.948848    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-096c8434-8xk57" event={"ID":"47a963c9-3243-4afd-a05d-e35e024ee5b6","Type":"ContainerDied","Data":"0f3b461daf9ac3ec80b1668f1ba2e968d4a9b92fe608fe1811096f1644d26c34"}
Oct 30 13:05:07.122400 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:07.042870    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:07.122400 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:07.043709    4544 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:07.465118 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:07.465069    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 13:05:07.585862 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:07.564553    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:05:07.641536 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:07.635734    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:07.642838 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:07.642058    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:05:07.973451 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:07.973402    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:08.043535 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:08.043475    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:08.045200 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:08.045170    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:05:08.269498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:08.269374    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:08.636315 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:08.490625    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 13:05:08.673399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:08.638897    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 13:05:08.741333 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:08.741294    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:08.984085 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:08.983970    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:09.008801 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:09.008746    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 13:05:09.044734 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:09.044682    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:09.861397 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:09.813809    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:09.929245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:09.814112    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 13:05:10.051244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:10.047496    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:10.095248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:10.067062    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:10.095610 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:10.095058    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 13:05:10.699608 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:10.699561    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 13:05:10.761720 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:05:10.761623    4544 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.645s"
Oct 30 13:05:10.762029 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:10.762010    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:05:10.762200 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:10.762189    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:05:10.790684 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:10.790519    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:05:11.055983 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:11.055939    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:11.111278 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:11.059333    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:05:11.111278 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:11.097992    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:11.576244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:11.566996    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p"
Oct 30 13:05:11.576244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:11.567121    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x"
Oct 30 13:05:11.747624 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:11.747542    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:11.937510 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:11.935131    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:11.938099 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:11.938071    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-096c8434-8xk57"
Oct 30 13:05:11.940416 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:11.940376    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:12.072811 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:12.070530    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:12.083806 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:12.083755    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47a963c9-3243-4afd-a05d-e35e024ee5b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47a963c9-3243-4afd-a05d-e35e024ee5b6-maas-api-openshift-ca\") pod \"47a963c9-3243-4afd-a05d-e35e024ee5b6\" (UID: \"47a963c9-3243-4afd-a05d-e35e024ee5b6\") "
Oct 30 13:05:12.084140 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:12.084115    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47a963c9-3243-4afd-a05d-e35e024ee5b6-kube-api-access-vmk6n\" (UniqueName: \"kubernetes.io/projected/47a963c9-3243-4afd-a05d-e35e024ee5b6-kube-api-access-vmk6n\") pod \"47a963c9-3243-4afd-a05d-e35e024ee5b6\" (UID: \"47a963c9-3243-4afd-a05d-e35e024ee5b6\") "
Oct 30 13:05:12.084492 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:12.084470    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47a963c9-3243-4afd-a05d-e35e024ee5b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47a963c9-3243-4afd-a05d-e35e024ee5b6-maas-api-xks-ca\") pod \"47a963c9-3243-4afd-a05d-e35e024ee5b6\" (UID: \"47a963c9-3243-4afd-a05d-e35e024ee5b6\") "
Oct 30 13:05:12.100638 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:12.100593    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:12.176093 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:12.129958    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:05:12.348296 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:12.321358    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 13:05:12.348296 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:12.345534    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47a963c9-3243-4afd-a05d-e35e024ee5b6-maas-api-openshift-ca" pod "47a963c9-3243-4afd-a05d-e35e024ee5b6" (UID: "47a963c9-3243-4afd-a05d-e35e024ee5b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:12.355259 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:12.351685    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:12.423258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:12.355824    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47a963c9-3243-4afd-a05d-e35e024ee5b6-kube-api-access-vmk6n" pod "47a963c9-3243-4afd-a05d-e35e024ee5b6" (UID: "47a963c9-3243-4afd-a05d-e35e024ee5b6"). InnerVolumeSpecName "kube-api-access-vmk6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:12.423258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:12.384265    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-096c8434-8xk57" event={"ID":"47a963c9-3243-4afd-a05d-e35e024ee5b6","Type":"ContainerDied","Data":"a1d9927edc323d2462e09caee8b99937b790bb3bf66512cc0d2c7992ae02611b"}
Oct 30 13:05:12.423258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:12.384546    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-096c8434-8xk57"
Oct 30 13:05:12.423561 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:12.422832    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1d9927edc323d2462e09caee8b99937b790bb3bf66512cc0d2c7992ae02611b"
Oct 30 13:05:12.450631 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:12.450598    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47a963c9-3243-4afd-a05d-e35e024ee5b6-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:05:12.450868 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:12.450846    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmk6n\" (UniqueName: \"kubernetes.io/projected/47a963c9-3243-4afd-a05d-e35e024ee5b6-kube-api-access-vmk6n\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:05:12.467327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:12.466167    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47a963c9-3243-4afd-a05d-e35e024ee5b6-maas-api-xks-ca" pod "47a963c9-3243-4afd-a05d-e35e024ee5b6" (UID: "47a963c9-3243-4afd-a05d-e35e024ee5b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:05:12.467327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:12.467058    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:05:12.588612 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:12.571941    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47a963c9-3243-4afd-a05d-e35e024ee5b6-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:05:13.511758 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:13.511721    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:14.225070 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:14.225032    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:15.404897 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:15.404864    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:16.062347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:16.062309    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-68573cef-6d977f448c-rn2qv"]
Oct 30 13:05:16.063092 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:16.063077    4544 cpu_manager.go:414] "containerMap: removing container" podUID="47a963c9-3243-4afd-a05d-e35e024ee5b6" containerName="revoke-keys"
Oct 30 13:05:16.063159 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:16.063093    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47a963c9-3243-4afd-a05d-e35e024ee5b6" containerName="revoke-keys"
Oct 30 13:05:16.063278 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:16.063267    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="47a963c9-3243-4afd-a05d-e35e024ee5b6" containerName="revoke-keys"
Oct 30 13:05:16.073610 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:16.073582    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68573cef-6d977f448c-rn2qv"
Oct 30 13:05:16.091761 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:16.091731    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-68573cef-serving-cert\""
Oct 30 13:05:16.097859 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:16.097821    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-68573cef-6d977f448c-rn2qv"]
Oct 30 13:05:16.128066 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:16.128025    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:16.231510 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:16.231470    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51d6cd93-1695-4201-bbe3-9eb7f3da0597-openshift-service-ca\") pod \"maas-api-e2e-ait-68573cef-6d977f448c-rn2qv\" (UID: \"51d6cd93-1695-4201-bbe3-9eb7f3da0597\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68573cef-6d977f448c-rn2qv"
Oct 30 13:05:16.231684 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:16.231589    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mjpx\" (UniqueName: \"kubernetes.io/projected/51d6cd93-1695-4201-bbe3-9eb7f3da0597-kube-api-access-8mjpx\") pod \"maas-api-e2e-ait-68573cef-6d977f448c-rn2qv\" (UID: \"51d6cd93-1695-4201-bbe3-9eb7f3da0597\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68573cef-6d977f448c-rn2qv"
Oct 30 13:05:16.231684 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:16.231616    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51d6cd93-1695-4201-bbe3-9eb7f3da0597-metrics-certs\") pod \"maas-api-e2e-ait-68573cef-6d977f448c-rn2qv\" (UID: \"51d6cd93-1695-4201-bbe3-9eb7f3da0597\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68573cef-6d977f448c-rn2qv"
Oct 30 13:05:16.231782 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:16.231757    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51d6cd93-1695-4201-bbe3-9eb7f3da0597-maas-api-tls\") pod \"maas-api-e2e-ait-68573cef-6d977f448c-rn2qv\" (UID: \"51d6cd93-1695-4201-bbe3-9eb7f3da0597\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68573cef-6d977f448c-rn2qv"
Oct 30 13:05:16.244546 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:16.244509    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:16.344251 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:16.337648    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51d6cd93-1695-4201-bbe3-9eb7f3da0597-openshift-service-ca\") pod \"maas-api-e2e-ait-68573cef-6d977f448c-rn2qv\" (UID: \"51d6cd93-1695-4201-bbe3-9eb7f3da0597\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68573cef-6d977f448c-rn2qv"
Oct 30 13:05:16.344251 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:16.337736    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mjpx\" (UniqueName: \"kubernetes.io/projected/51d6cd93-1695-4201-bbe3-9eb7f3da0597-kube-api-access-8mjpx\") pod \"maas-api-e2e-ait-68573cef-6d977f448c-rn2qv\" (UID: \"51d6cd93-1695-4201-bbe3-9eb7f3da0597\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68573cef-6d977f448c-rn2qv"
Oct 30 13:05:16.344251 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:16.337772    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51d6cd93-1695-4201-bbe3-9eb7f3da0597-metrics-certs\") pod \"maas-api-e2e-ait-68573cef-6d977f448c-rn2qv\" (UID: \"51d6cd93-1695-4201-bbe3-9eb7f3da0597\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68573cef-6d977f448c-rn2qv"
Oct 30 13:05:16.344251 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:16.337853    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51d6cd93-1695-4201-bbe3-9eb7f3da0597-maas-api-tls\") pod \"maas-api-e2e-ait-68573cef-6d977f448c-rn2qv\" (UID: \"51d6cd93-1695-4201-bbe3-9eb7f3da0597\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68573cef-6d977f448c-rn2qv"
Oct 30 13:05:16.344251 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:16.342827    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51d6cd93-1695-4201-bbe3-9eb7f3da0597-openshift-service-ca\") pod \"maas-api-e2e-ait-68573cef-6d977f448c-rn2qv\" (UID: \"51d6cd93-1695-4201-bbe3-9eb7f3da0597\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68573cef-6d977f448c-rn2qv"
Oct 30 13:05:16.354971 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:16.354931    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51d6cd93-1695-4201-bbe3-9eb7f3da0597-maas-api-tls\") pod \"maas-api-e2e-ait-68573cef-6d977f448c-rn2qv\" (UID: \"51d6cd93-1695-4201-bbe3-9eb7f3da0597\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68573cef-6d977f448c-rn2qv"
Oct 30 13:05:16.355853 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:16.355821    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51d6cd93-1695-4201-bbe3-9eb7f3da0597-metrics-certs\") pod \"maas-api-e2e-ait-68573cef-6d977f448c-rn2qv\" (UID: \"51d6cd93-1695-4201-bbe3-9eb7f3da0597\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68573cef-6d977f448c-rn2qv"
Oct 30 13:05:16.391248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:16.379474    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mjpx\" (UniqueName: \"kubernetes.io/projected/51d6cd93-1695-4201-bbe3-9eb7f3da0597-kube-api-access-8mjpx\") pod \"maas-api-e2e-ait-68573cef-6d977f448c-rn2qv\" (UID: \"51d6cd93-1695-4201-bbe3-9eb7f3da0597\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68573cef-6d977f448c-rn2qv"
Oct 30 13:05:16.391518 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:16.387740    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:16.415201 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:16.415153    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68573cef-6d977f448c-rn2qv"
Oct 30 13:05:16.773566 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:16.773539    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-68573cef-6d977f448c-rn2qv"]
Oct 30 13:05:16.773769 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:05:16.773745    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51d6cd93_1695_4201_bbe3_9eb7f3da0597.slice/crio-aca8f80c49917634d1007d26d580ddeb6c19a0dd6a2cf8b40b9a9fd8e16bf8d7 WatchSource:0}: Error finding container aca8f80c49917634d1007d26d580ddeb6c19a0dd6a2cf8b40b9a9fd8e16bf8d7: Status 404 returned error can't find the container with id aca8f80c49917634d1007d26d580ddeb6c19a0dd6a2cf8b40b9a9fd8e16bf8d7
Oct 30 13:05:17.505247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:17.503478    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68573cef-6d977f448c-rn2qv" event={"ID":"51d6cd93-1695-4201-bbe3-9eb7f3da0597","Type":"ContainerStarted","Data":"077d95b832f19ac56f7bfaa1f5a79ca9dd77eab4e30ef8302d2475938c063911"}
Oct 30 13:05:17.505247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:17.503551    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68573cef-6d977f448c-rn2qv" event={"ID":"51d6cd93-1695-4201-bbe3-9eb7f3da0597","Type":"ContainerStarted","Data":"aca8f80c49917634d1007d26d580ddeb6c19a0dd6a2cf8b40b9a9fd8e16bf8d7"}
Oct 30 13:05:17.505247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:17.504821    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68573cef-6d977f448c-rn2qv"
Oct 30 13:05:17.565178 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:17.559839    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68573cef-6d977f448c-rn2qv" podStartSLOduration=1.037758689 podStartE2EDuration="1.559816457s" totalImagesPullingTime="522.057768ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:17.5548604 +0000 UTC m=+1212.601994354" watchObservedRunningTime="2026-10-30 13:05:17.559816457 +0000 UTC m=+1212.606950410"
Oct 30 13:05:18.199101 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:18.199039    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:19.791353 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:19.791302    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8"]
Oct 30 13:05:19.796723 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:19.796693    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8"
Oct 30 13:05:19.817658 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:19.817626    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-752d3a58-serving-cert\""
Oct 30 13:05:19.819587 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:19.819528    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpq8j\" (UniqueName: \"kubernetes.io/projected/6e34292a-71f4-4414-9a2d-4d69729c535c-kube-api-access-mpq8j\") pod \"maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8\" (UID: \"6e34292a-71f4-4414-9a2d-4d69729c535c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8"
Oct 30 13:05:19.819820 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:19.819684    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e34292a-71f4-4414-9a2d-4d69729c535c-metrics-certs\") pod \"maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8\" (UID: \"6e34292a-71f4-4414-9a2d-4d69729c535c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8"
Oct 30 13:05:19.819820 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:19.819730    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e34292a-71f4-4414-9a2d-4d69729c535c-maas-api-tls\") pod \"maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8\" (UID: \"6e34292a-71f4-4414-9a2d-4d69729c535c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8"
Oct 30 13:05:19.819907 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:19.819884    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e34292a-71f4-4414-9a2d-4d69729c535c-openshift-service-ca\") pod \"maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8\" (UID: \"6e34292a-71f4-4414-9a2d-4d69729c535c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8"
Oct 30 13:05:19.835710 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:19.835674    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8"]
Oct 30 13:05:19.901945 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:19.901906    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:19.920996 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:19.920947    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e34292a-71f4-4414-9a2d-4d69729c535c-openshift-service-ca\") pod \"maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8\" (UID: \"6e34292a-71f4-4414-9a2d-4d69729c535c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8"
Oct 30 13:05:19.921290 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:19.921264    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpq8j\" (UniqueName: \"kubernetes.io/projected/6e34292a-71f4-4414-9a2d-4d69729c535c-kube-api-access-mpq8j\") pod \"maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8\" (UID: \"6e34292a-71f4-4414-9a2d-4d69729c535c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8"
Oct 30 13:05:19.921378 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:19.921337    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e34292a-71f4-4414-9a2d-4d69729c535c-metrics-certs\") pod \"maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8\" (UID: \"6e34292a-71f4-4414-9a2d-4d69729c535c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8"
Oct 30 13:05:19.921378 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:19.921368    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e34292a-71f4-4414-9a2d-4d69729c535c-maas-api-tls\") pod \"maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8\" (UID: \"6e34292a-71f4-4414-9a2d-4d69729c535c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8"
Oct 30 13:05:19.921780 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:19.921755    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e34292a-71f4-4414-9a2d-4d69729c535c-openshift-service-ca\") pod \"maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8\" (UID: \"6e34292a-71f4-4414-9a2d-4d69729c535c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8"
Oct 30 13:05:19.924050 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:19.924021    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e34292a-71f4-4414-9a2d-4d69729c535c-maas-api-tls\") pod \"maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8\" (UID: \"6e34292a-71f4-4414-9a2d-4d69729c535c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8"
Oct 30 13:05:19.924155 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:19.924070    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e34292a-71f4-4414-9a2d-4d69729c535c-metrics-certs\") pod \"maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8\" (UID: \"6e34292a-71f4-4414-9a2d-4d69729c535c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8"
Oct 30 13:05:19.952141 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:19.952113    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpq8j\" (UniqueName: \"kubernetes.io/projected/6e34292a-71f4-4414-9a2d-4d69729c535c-kube-api-access-mpq8j\") pod \"maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8\" (UID: \"6e34292a-71f4-4414-9a2d-4d69729c535c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8"
Oct 30 13:05:20.046345 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:20.036978    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:20.118122 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:20.116265    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8"
Oct 30 13:05:20.507007 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:05:20.506965    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e34292a_71f4_4414_9a2d_4d69729c535c.slice/crio-76872ad67a557d4ef08eb6837dbbc3dd4e023f1250b5eff66ed72d94ebc8b0c6 WatchSource:0}: Error finding container 76872ad67a557d4ef08eb6837dbbc3dd4e023f1250b5eff66ed72d94ebc8b0c6: Status 404 returned error can't find the container with id 76872ad67a557d4ef08eb6837dbbc3dd4e023f1250b5eff66ed72d94ebc8b0c6
Oct 30 13:05:20.507263 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:20.507243    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8"]
Oct 30 13:05:20.529786 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:20.529753    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8" event={"ID":"6e34292a-71f4-4414-9a2d-4d69729c535c","Type":"ContainerStarted","Data":"76872ad67a557d4ef08eb6837dbbc3dd4e023f1250b5eff66ed72d94ebc8b0c6"}
Oct 30 13:05:21.541130 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:21.541092    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8" event={"ID":"6e34292a-71f4-4414-9a2d-4d69729c535c","Type":"ContainerStarted","Data":"528369278b00d236c4d6646499136326fdf39104e6ef893e9fa107107b8229db"}
Oct 30 13:05:21.541584 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:21.541174    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8"
Oct 30 13:05:21.600692 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:21.600633    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8" podStartSLOduration=2.228638285 podStartE2EDuration="2.600618241s" totalImagesPullingTime="371.979956ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:21.597850582 +0000 UTC m=+1216.644984517" watchObservedRunningTime="2026-10-30 13:05:21.600618241 +0000 UTC m=+1216.647752194"
Oct 30 13:05:22.211662 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:22.211626    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:23.059194 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:23.059144    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bbaf346-j2c57"]
Oct 30 13:05:23.071286 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:23.071248    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bbaf346-j2c57"]
Oct 30 13:05:23.089466 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:23.089430    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c" path="/var/lib/kubelet/pods/6aeaad1a-ebfb-4bf0-8c6f-d281d2cc6f7c/volumes"
Oct 30 13:05:24.003447 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:24.003399    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:24.526509 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:24.526479    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68573cef-6d977f448c-rn2qv"
Oct 30 13:05:25.113985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.113947    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:25.172777 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.172740    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"]
Oct 30 13:05:25.177369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.177340    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:05:25.208552 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.208516    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"]
Oct 30 13:05:25.267542 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.267499    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:25.277041 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.277003    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh5gg\" (UniqueName: \"kubernetes.io/projected/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-kube-api-access-lh5gg\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:05:25.277308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.277062    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-workload-certs\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:05:25.277308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.277182    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-workload-socket\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:05:25.277308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.277240    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istiod-ca-cert\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:05:25.277511 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.277348    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istio-podinfo\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:05:25.277511 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.277417    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-credential-socket\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:05:25.277511 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.277496    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istio-data\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:05:25.277674 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.277540    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istio-token\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:05:25.277732 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.277672    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istio-envoy\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:05:25.378933 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.378841    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-workload-socket\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:05:25.378933 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.378891    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istiod-ca-cert\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:05:25.379148 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.378954    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istio-podinfo\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:05:25.379148 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.379072    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-credential-socket\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:05:25.379264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.379149    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istio-data\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:05:25.379264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.379199    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istio-token\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:05:25.379385 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.379347    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-workload-socket\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:05:25.379452 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.379399    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-credential-socket\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:05:25.379518 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.379500    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istio-envoy\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:05:25.379566 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.379547    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh5gg\" (UniqueName: \"kubernetes.io/projected/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-kube-api-access-lh5gg\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:05:25.379614 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.379597    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-workload-certs\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:05:25.379842 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.379818    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istiod-ca-cert\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:05:25.380131 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.380106    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istio-data\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:05:25.380246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.380183    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-workload-certs\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:05:25.382052 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.382031    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istio-podinfo\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:05:25.382052 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.382061    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istio-envoy\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:05:25.406344 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.406308    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istio-token\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:05:25.408626 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.408598    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh5gg\" (UniqueName: \"kubernetes.io/projected/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-kube-api-access-lh5gg\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:05:25.476330 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.476295    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"]
Oct 30 13:05:25.482357 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.482327    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:05:25.498320 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.498291    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:05:25.527277 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.518891    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"]
Oct 30 13:05:25.584619 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.582534    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-istio-envoy\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:05:25.584619 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.582710    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bdb6b8f-2389-407e-a9e7-a98907722803-istiod-ca-cert\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:05:25.584619 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.582781    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bdb6b8f-2389-407e-a9e7-a98907722803-istio-podinfo\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:05:25.584619 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.582832    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-workload-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:05:25.584619 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.582929    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-workload-certs\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:05:25.584619 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.582973    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bdb6b8f-2389-407e-a9e7-a98907722803-istio-token\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:05:25.584619 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.583145    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-istio-data\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:05:25.584619 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.583317    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdrwd\" (UniqueName: \"kubernetes.io/projected/6bdb6b8f-2389-407e-a9e7-a98907722803-kube-api-access-jdrwd\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:05:25.584619 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.583394    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-credential-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:05:25.685483 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.685393    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bdb6b8f-2389-407e-a9e7-a98907722803-istio-podinfo\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:05:25.685483 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.685434    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-workload-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:05:25.685483 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.685464    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-workload-certs\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:05:25.685789 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.685498    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bdb6b8f-2389-407e-a9e7-a98907722803-istio-token\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:05:25.685789 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.685568    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-istio-data\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:05:25.685789 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.685661    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdrwd\" (UniqueName: \"kubernetes.io/projected/6bdb6b8f-2389-407e-a9e7-a98907722803-kube-api-access-jdrwd\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:05:25.685789 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.685692    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-credential-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:05:25.685789 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.685784    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-istio-envoy\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:05:25.686038 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.685830    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bdb6b8f-2389-407e-a9e7-a98907722803-istiod-ca-cert\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:05:25.687117 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.686491    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-workload-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:05:25.687117 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.686610    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bdb6b8f-2389-407e-a9e7-a98907722803-istiod-ca-cert\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:05:25.687117 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.686794    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-workload-certs\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:05:25.687117 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.687060    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-istio-data\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:05:25.687435 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.687340    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-credential-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:05:25.688736 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.688708    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bdb6b8f-2389-407e-a9e7-a98907722803-istio-podinfo\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:05:25.689556 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.689533    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-istio-envoy\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:05:25.709091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.709058    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bdb6b8f-2389-407e-a9e7-a98907722803-istio-token\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:05:25.710006 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.709976    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdrwd\" (UniqueName: \"kubernetes.io/projected/6bdb6b8f-2389-407e-a9e7-a98907722803-kube-api-access-jdrwd\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:05:25.799305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.799264    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:05:25.893423 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.893378    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"]
Oct 30 13:05:25.898190 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.898146    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:05:25.898328 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.898254    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:05:25.898328 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:25.898301    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:05:26.138437 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:05:26.138404    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bdb6b8f_2389_407e_a9e7_a98907722803.slice/crio-804e73fa9b9a665c74a422b7321789f13703bd3166c935e1bf4e1b1319354303 WatchSource:0}: Error finding container 804e73fa9b9a665c74a422b7321789f13703bd3166c935e1bf4e1b1319354303: Status 404 returned error can't find the container with id 804e73fa9b9a665c74a422b7321789f13703bd3166c935e1bf4e1b1319354303
Oct 30 13:05:26.140150 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:26.140124    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"]
Oct 30 13:05:26.142300 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:26.142260    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:05:26.142470 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:26.142347    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:05:26.142470 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:26.142429    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:05:26.143886 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:26.143865    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:26.268247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:26.265500    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:26.602467 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:26.602425    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt" event={"ID":"6bdb6b8f-2389-407e-a9e7-a98907722803","Type":"ContainerStarted","Data":"8bb90c57c4d8bbe1cfc564e75719afa4077ce0c9589faf4f635e001c421bf2f8"}
Oct 30 13:05:26.602994 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:26.602495    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt" event={"ID":"6bdb6b8f-2389-407e-a9e7-a98907722803","Type":"ContainerStarted","Data":"804e73fa9b9a665c74a422b7321789f13703bd3166c935e1bf4e1b1319354303"}
Oct 30 13:05:26.604465 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:26.604433    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf" event={"ID":"1fc6a6f0-9e93-41c0-a0db-eee647f4082b","Type":"ContainerStarted","Data":"446a5bf8d94854172fa8207889d6670f4db80fb8c94de7a6f4fa35442805f1ca"}
Oct 30 13:05:26.604595 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:26.604490    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf" event={"ID":"1fc6a6f0-9e93-41c0-a0db-eee647f4082b","Type":"ContainerStarted","Data":"ab6e42f87c1b83722edc02593d78c75a50f1cb7636d489ea6b2fa2a2e9c8399f"}
Oct 30 13:05:26.652898 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:26.652845    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt" podStartSLOduration=1.652829372 podStartE2EDuration="1.652829372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:26.650600038 +0000 UTC m=+1221.697733994" watchObservedRunningTime="2026-10-30 13:05:26.652829372 +0000 UTC m=+1221.699963324"
Oct 30 13:05:26.696837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:26.696784    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf" podStartSLOduration=1.696768895 podStartE2EDuration="1.696768895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:26.695504776 +0000 UTC m=+1221.742638730" watchObservedRunningTime="2026-10-30 13:05:26.696768895 +0000 UTC m=+1221.743902847"
Oct 30 13:05:26.801459 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:26.801401    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:05:27.499383 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:27.499348    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:05:27.501164 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:27.501136    4544 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 13:05:27.501304 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:27.501187    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf" podUID="1fc6a6f0-9e93-41c0-a0db-eee647f4082b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 13:05:27.570963 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:27.570931    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8"
Oct 30 13:05:27.801421 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:27.801327    4544 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 13:05:27.801421 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:27.801384    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt" podUID="6bdb6b8f-2389-407e-a9e7-a98907722803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 13:05:28.236245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:28.235835    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:28.503434 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:28.501291    4544 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 13:05:28.503434 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:28.501354    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf" podUID="1fc6a6f0-9e93-41c0-a0db-eee647f4082b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 13:05:28.799957 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:28.799856    4544 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 13:05:28.799957 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:28.799933    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt" podUID="6bdb6b8f-2389-407e-a9e7-a98907722803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 13:05:29.008455 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:05:29.008418    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:05:29.512254 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:29.502760    4544 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 13:05:29.512254 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:29.502822    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf" podUID="1fc6a6f0-9e93-41c0-a0db-eee647f4082b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 13:05:29.832315 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:29.822709    4544 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 13:05:29.884484 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:29.884434    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt" podUID="6bdb6b8f-2389-407e-a9e7-a98907722803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 13:05:30.499517 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:30.499483    4544 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 13:05:30.499907 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:30.499561    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf" podUID="1fc6a6f0-9e93-41c0-a0db-eee647f4082b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 13:05:30.508950 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:30.508805    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:30.801718 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:30.801623    4544 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 13:05:30.801997 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:30.801970    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt" podUID="6bdb6b8f-2389-407e-a9e7-a98907722803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 13:05:31.509246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:31.503138    4544 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 13:05:31.509246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:31.503214    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf" podUID="1fc6a6f0-9e93-41c0-a0db-eee647f4082b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 13:05:31.809300 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:31.802105    4544 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 13:05:31.809300 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:31.802187    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt" podUID="6bdb6b8f-2389-407e-a9e7-a98907722803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 13:05:31.923380 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:31.923324    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:32.051249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:32.049320    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:32.384412 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:32.384360    4544 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:32.505740 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:32.505700    4544 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 13:05:32.506748 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:32.506709    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf" podUID="1fc6a6f0-9e93-41c0-a0db-eee647f4082b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 13:05:32.863311 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:32.817650    4544 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 13:05:32.863311 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:32.817718    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt" podUID="6bdb6b8f-2389-407e-a9e7-a98907722803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 13:05:33.095264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:33.061949    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:33.095264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:33.062441    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:05:33.095264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:33.064522    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:33.375306 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:33.260810    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:33.375306 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:33.365340    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6nbzr" podUID="8c41830a-e18f-4b80-a625-d013bc6f6f5a" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.21:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:33.556274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:33.534036    4544 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 13:05:33.556274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:33.534100    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf" podUID="1fc6a6f0-9e93-41c0-a0db-eee647f4082b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 13:05:33.910603 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:33.910557    4544 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 13:05:33.934829 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:33.934783    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt" podUID="6bdb6b8f-2389-407e-a9e7-a98907722803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 13:05:33.943732 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:33.943683    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 13:05:33.957463 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:33.943993    4544 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:34.076800 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:34.076752    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 13:05:34.104259 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:34.078846    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:05:34.106212 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:34.106176    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 13:05:34.134429 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:34.134380    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 13:05:34.268451 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:34.268344    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:34.292247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:34.272756    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:05:34.329389 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:34.329341    4544 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6nbzr" podUID="8c41830a-e18f-4b80-a625-d013bc6f6f5a" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.21:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:34.626559 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:34.626506    4544 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 13:05:34.663422 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:34.663362    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf" podUID="1fc6a6f0-9e93-41c0-a0db-eee647f4082b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 13:05:34.697258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:34.677380    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-w27kj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:05:34.697654 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:34.697609    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj" podUID="287ea8cd-7fc1-4967-a291-adf6973f5967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:34.902305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:34.901692    4544 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 13:05:34.902305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:34.901781    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt" podUID="6bdb6b8f-2389-407e-a9e7-a98907722803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 13:05:34.960908 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:34.960837    4544 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:05:34.961609 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:34.961559    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz" podUID="1bf8120f-fb85-451c-922f-7e1cafe165df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:35.261304 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:35.260717    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 13:05:35.273260 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:35.261707    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:05:35.403248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:35.394206    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 13:05:35.442497 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:35.424582    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 13:05:35.451885 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:35.451820    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:05:35.670785 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:35.670734    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 13:05:36.101247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:36.085510    4544 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:05:36.206269 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:36.196382    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn" podUID="7f93aa79-27fa-4f24-9b20-b6da0dee9ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:36.206585 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:36.204892    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj" podUID="52424dad-1898-48d3-9529-3dc7f16cb1e5" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Oct 30 13:05:36.206734 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:36.101446    4544 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": context deadline exceeded"
Oct 30 13:05:36.206859 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:36.158958    4544 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 13:05:36.207102 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:36.207078    4544 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 13:05:36.207281 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:36.207249    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf" podUID="1fc6a6f0-9e93-41c0-a0db-eee647f4082b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 13:05:36.252489 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:36.252380    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:36.430306 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:36.424505    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:36.430306 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:36.424765    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj" podUID="3c724287-666b-47b4-8c6b-428a20ff1164" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.57:50052\" within 1s: context deadline exceeded"
Oct 30 13:05:36.430306 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:36.425019    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:36.460461 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:36.460316    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:36.525247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:36.205970    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6" podUID="4a23bc66-d828-423f-a2a1-e803aef031d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:36.533805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:36.199328    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88" podUID="62b23320-6cd9-4cbb-b58e-cf8de8a931a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:36.576504 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:05:36.576466    4544 logging.go:55] [core] [Channel #1526 SubChannel #1527]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.57:50052", ServerName: "10.133.0.57:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: failed to write client preface: write tcp 10.133.0.2:48760->10.133.0.57:50052: use of closed network connection"
Oct 30 13:05:36.689567 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:36.689460    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:36.706925 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:36.706883    4544 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 13:05:36.707213 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:36.707183    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf" podUID="1fc6a6f0-9e93-41c0-a0db-eee647f4082b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 13:05:36.817315 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:36.485680    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:36.871640 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:36.827488    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt" podUID="6bdb6b8f-2389-407e-a9e7-a98907722803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 13:05:36.872566 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:36.872535    4544 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 13:05:36.872789 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:36.872761    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt" podUID="6bdb6b8f-2389-407e-a9e7-a98907722803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 13:05:37.465249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:37.464155    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:37.545248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:37.538396    4544 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 13:05:37.545248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:37.538472    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf" podUID="1fc6a6f0-9e93-41c0-a0db-eee647f4082b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 13:05:37.624481 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:37.624442    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7462m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:05:37.624786 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:37.624756    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m" podUID="3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:37.648920 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:05:37.625897    4544 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.228s"
Oct 30 13:05:37.649273 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:37.649249    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:05:37.717271 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:37.670798    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:37.728828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:37.691472    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:37.731432 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:37.690361    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:37.783067 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:37.783006    4544 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:37.922269 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:37.900288    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:38.084674 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:38.084637    4544 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 13:05:38.526159 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:38.525761    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt" podUID="6bdb6b8f-2389-407e-a9e7-a98907722803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 13:05:38.737744 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:38.719011    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:38.768682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:38.738613    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:38.769264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:38.738649    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 13:05:38.981247 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:05:38.948574    4544 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.244s"
Oct 30 13:05:38.996750 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:38.996719    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:05:39.001332 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:39.001298    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f4c3482-lvwn4"]
Oct 30 13:05:39.001623 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:39.001590    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f4c3482-lvwn4"]
Oct 30 13:05:39.042527 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:39.042481    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 13:05:39.104712 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:39.060565    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:05:39.105055 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:39.103795    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:39.395974 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:39.395930    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:39.804320 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:39.790539    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:39.805085 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:39.805040    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 13:05:39.836780 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:39.836717    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 13:05:39.852173 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:39.852128    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:39.890427 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:39.890385    4544 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:05:39.890723 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:39.890693    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt" podUID="6bdb6b8f-2389-407e-a9e7-a98907722803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:39.899281 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:39.893751    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb59d65b-955e-49b3-9bfb-ae2fa7f98fda" path="/var/lib/kubelet/pods/eb59d65b-955e-49b3-9bfb-ae2fa7f98fda/volumes"
Oct 30 13:05:40.112294 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:40.108486    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:40.134261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:40.113177    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 13:05:40.209044 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:40.209008    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:05:40.210395 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:40.210356    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:05:40.210677 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:40.210660    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:05:40.742744 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:40.742710    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:05:40.791076 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:40.790966    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:40.808399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:40.806344    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:40.838281 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:40.835385    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:05:40.838281 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:40.837044    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:05:40.838281 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:40.837139    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 13:05:40.862454 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:40.846489    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:05:41.054072 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.053979    4544 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": context deadline exceeded"
Oct 30 13:05:41.221244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.205276    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:41.501653 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.486211    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 13:05:41.501653 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.489338    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"]
Oct 30 13:05:41.501653 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.501203    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"]
Oct 30 13:05:41.501653 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.501328    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:05:41.507189 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.507164    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:05:41.508338 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.508316    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:05:41.512654 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.512631    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:05:41.687673 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.687639    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-credential-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:05:41.688316 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.688102    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20cef20e-1c1b-4d57-b260-019f2de104a1-istio-podinfo\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:05:41.688982 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.688958    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-workload-certs\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:05:41.689209 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.689174    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-istio-data\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:05:41.689526 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.689502    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20cef20e-1c1b-4d57-b260-019f2de104a1-istiod-ca-cert\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:05:41.690091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.690064    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-istio-envoy\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:05:41.690202 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.690126    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20cef20e-1c1b-4d57-b260-019f2de104a1-istio-token\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:05:41.690636 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.690608    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56frj\" (UniqueName: \"kubernetes.io/projected/20cef20e-1c1b-4d57-b260-019f2de104a1-kube-api-access-56frj\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:05:41.690731 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.690714    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-workload-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:05:41.794278 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.794169    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20cef20e-1c1b-4d57-b260-019f2de104a1-istio-podinfo\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:05:41.794278 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.794261    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-workload-certs\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:05:41.794500 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.794296    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-istio-data\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:05:41.794500 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.794371    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20cef20e-1c1b-4d57-b260-019f2de104a1-istiod-ca-cert\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:05:41.794500 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.794421    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-istio-envoy\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:05:41.794500 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.794466    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20cef20e-1c1b-4d57-b260-019f2de104a1-istio-token\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:05:41.794700 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.794583    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56frj\" (UniqueName: \"kubernetes.io/projected/20cef20e-1c1b-4d57-b260-019f2de104a1-kube-api-access-56frj\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:05:41.794700 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.794644    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-workload-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:05:41.794797 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.794766    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-credential-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:05:41.795168 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.795143    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-credential-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:05:41.796015 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.795986    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-workload-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:05:41.796395 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.796367    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-istio-data\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:05:41.796603 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.796572    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-workload-certs\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:05:41.796775 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.796749    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20cef20e-1c1b-4d57-b260-019f2de104a1-istiod-ca-cert\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:05:41.798930 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.798907    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20cef20e-1c1b-4d57-b260-019f2de104a1-istio-podinfo\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:05:41.800155 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.800137    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-istio-envoy\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:05:41.868637 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.864000    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20cef20e-1c1b-4d57-b260-019f2de104a1-istio-token\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:05:41.909719 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:41.909685    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56frj\" (UniqueName: \"kubernetes.io/projected/20cef20e-1c1b-4d57-b260-019f2de104a1-kube-api-access-56frj\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:05:42.188088 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:42.187827    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:05:42.375446 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:42.375388    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:42.794606 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:42.794574    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"]
Oct 30 13:05:42.797462 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:42.797424    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:05:42.797565 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:42.797500    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:05:42.797565 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:42.797532    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:05:42.994967 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:42.994920    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp" event={"ID":"20cef20e-1c1b-4d57-b260-019f2de104a1","Type":"ContainerStarted","Data":"5754c51bb6a2522144bcd347ef34b240b0bde3a9f0ceb8c543ab3a55a1921e1f"}
Oct 30 13:05:44.008664 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:44.008555    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp" event={"ID":"20cef20e-1c1b-4d57-b260-019f2de104a1","Type":"ContainerStarted","Data":"0815d1389d8b9b41e98b4b4bb3b047315250308667ec1c516be9e83a569ffea6"}
Oct 30 13:05:44.067989 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:44.067922    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp" podStartSLOduration=4.067902955 podStartE2EDuration="4.067902955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:44.061234758 +0000 UTC m=+1239.108368707" watchObservedRunningTime="2026-10-30 13:05:44.067902955 +0000 UTC m=+1239.115036908"
Oct 30 13:05:44.222020 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:44.218636    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:05:44.241269 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:44.222351    4544 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 30 13:05:44.241269 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:44.222417    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp" podUID="20cef20e-1c1b-4d57-b260-019f2de104a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 30 13:05:44.744246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:44.735158    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:45.188914 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:45.188878    4544 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 30 13:05:45.189352 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:45.188939    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp" podUID="20cef20e-1c1b-4d57-b260-019f2de104a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 30 13:05:46.189728 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:46.189689    4544 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 30 13:05:46.190143 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:46.189759    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp" podUID="20cef20e-1c1b-4d57-b260-019f2de104a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 30 13:05:46.888203 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:46.888166    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:47.188969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:47.188879    4544 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 30 13:05:47.188969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:47.188935    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp" podUID="20cef20e-1c1b-4d57-b260-019f2de104a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 30 13:05:48.189607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:48.189566    4544 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 30 13:05:48.189977 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:48.189640    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp" podUID="20cef20e-1c1b-4d57-b260-019f2de104a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 30 13:05:49.069404 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:49.069367    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:49.213363 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:49.210684    4544 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 30 13:05:49.213363 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:49.210746    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp" podUID="20cef20e-1c1b-4d57-b260-019f2de104a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 30 13:05:50.010122 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:50.010086    4544 patch_prober.go:29] interesting pod/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": read tcp 10.133.0.2:55170->10.133.0.120:15021: read: connection reset by peer" start-of-body=
Oct 30 13:05:50.010348 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:50.010158    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s" podUID="78230db8-1d2b-4d35-b025-c54111d5bfd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": read tcp 10.133.0.2:55170->10.133.0.120:15021: read: connection reset by peer"
Oct 30 13:05:50.188960 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:50.188851    4544 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 30 13:05:50.188960 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:50.188915    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp" podUID="20cef20e-1c1b-4d57-b260-019f2de104a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 30 13:05:50.247854 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:50.247822    4544 generic.go:356] "Generic (PLEG): container finished" podID="78230db8-1d2b-4d35-b025-c54111d5bfd2" containerID="7114a1bbff5bec10c67ee27609e2050975014ad8c423d168677467f74811f98f" exitCode=0
Oct 30 13:05:50.248339 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:50.247862    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s" event={"ID":"78230db8-1d2b-4d35-b025-c54111d5bfd2","Type":"ContainerDied","Data":"7114a1bbff5bec10c67ee27609e2050975014ad8c423d168677467f74811f98f"}
Oct 30 13:05:50.248431 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:50.248413    4544 scope.go:122] "RemoveContainer" containerID="7114a1bbff5bec10c67ee27609e2050975014ad8c423d168677467f74811f98f"
Oct 30 13:05:50.250406 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:50.250386    4544 generic.go:356] "Generic (PLEG): container finished" podID="5d0c65ab-624c-40ab-9529-327c79d6567a" containerID="a099f064cb205f4a7ec204b4cda78f73f91353fdcf3caaea5bb9e26fd02db3ab" exitCode=0
Oct 30 13:05:50.250516 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:50.250421    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz" event={"ID":"5d0c65ab-624c-40ab-9529-327c79d6567a","Type":"ContainerDied","Data":"a099f064cb205f4a7ec204b4cda78f73f91353fdcf3caaea5bb9e26fd02db3ab"}
Oct 30 13:05:50.250571 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:50.250537    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:05:50.250660 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:50.250637    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:05:50.250714 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:50.250697    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:05:50.250775 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:50.250762    4544 scope.go:122] "RemoveContainer" containerID="a099f064cb205f4a7ec204b4cda78f73f91353fdcf3caaea5bb9e26fd02db3ab"
Oct 30 13:05:50.252445 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:50.252410    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:05:50.252555 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:50.252498    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:05:50.252555 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:50.252545    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:05:51.167610 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:51.167572    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:51.202245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:51.199184    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:05:51.202245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:51.200902    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:05:51.209288 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:51.203138    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:05:51.412261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:51.406896    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz" event={"ID":"5d0c65ab-624c-40ab-9529-327c79d6567a","Type":"ContainerStarted","Data":"c1d1c97d72b74201dadc51e3c35547c2dba1c9752d4a4f674d097fe8c44c5aea"}
Oct 30 13:05:51.426350 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:51.416527    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s" event={"ID":"78230db8-1d2b-4d35-b025-c54111d5bfd2","Type":"ContainerStarted","Data":"28f1f39085a632e9b5cbf74eff37292ff5a3e87cfc396c33d042c33e543f101f"}
Oct 30 13:05:52.057454 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:52.057413    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23911506-vdb5f"]
Oct 30 13:05:52.068253 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:52.062979    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23911506-vdb5f"]
Oct 30 13:05:52.070572 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:52.070529    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:05:52.070933 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:52.070907    4544 patch_prober.go:29] interesting pod/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 13:05:52.071093 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:52.071067    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s" podUID="78230db8-1d2b-4d35-b025-c54111d5bfd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 13:05:52.355250 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:52.346801    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:05:52.355250 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:52.347000    4544 patch_prober.go:29] interesting pod/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 13:05:52.355250 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:52.347047    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz" podUID="5d0c65ab-624c-40ab-9529-327c79d6567a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 13:05:52.659307 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:52.655249    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:53.071308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:53.070777    4544 patch_prober.go:29] interesting pod/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 13:05:53.071308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:53.070877    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s" podUID="78230db8-1d2b-4d35-b025-c54111d5bfd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 13:05:53.097266 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:53.095698    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37d468c2-42fa-4cf6-903b-167b4780e5b2" path="/var/lib/kubelet/pods/37d468c2-42fa-4cf6-903b-167b4780e5b2/volumes"
Oct 30 13:05:53.366245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:53.366059    4544 patch_prober.go:29] interesting pod/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 13:05:53.366245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:53.366124    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz" podUID="5d0c65ab-624c-40ab-9529-327c79d6567a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 13:05:53.367629 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:53.367602    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:53.836244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:53.832749    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:54.072105 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:54.072060    4544 patch_prober.go:29] interesting pod/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 13:05:54.072320 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:54.072128    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s" podUID="78230db8-1d2b-4d35-b025-c54111d5bfd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 13:05:54.191802 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:54.191704    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:54.339703 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:54.339663    4544 patch_prober.go:29] interesting pod/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 13:05:54.339989 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:54.339959    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz" podUID="5d0c65ab-624c-40ab-9529-327c79d6567a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 13:05:55.070300 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:55.070259    4544 patch_prober.go:29] interesting pod/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 13:05:55.070817 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:55.070336    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s" podUID="78230db8-1d2b-4d35-b025-c54111d5bfd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 13:05:55.337799 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:55.337759    4544 patch_prober.go:29] interesting pod/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 13:05:55.338132 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:55.338095    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz" podUID="5d0c65ab-624c-40ab-9529-327c79d6567a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 13:05:55.868858 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:55.868793    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:56.072007 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:56.071963    4544 patch_prober.go:29] interesting pod/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 13:05:56.073624 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:56.073582    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s" podUID="78230db8-1d2b-4d35-b025-c54111d5bfd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 13:05:56.104121 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:56.104084    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3838e9d-pdzg7"]
Oct 30 13:05:56.111372 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:56.111111    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3838e9d-pdzg7"]
Oct 30 13:05:56.174334 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:56.173284    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:05:56.366783 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:56.366744    4544 patch_prober.go:29] interesting pod/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 13:05:56.367001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:56.366820    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz" podUID="5d0c65ab-624c-40ab-9529-327c79d6567a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 13:05:57.180582 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:57.180526    4544 patch_prober.go:29] interesting pod/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 13:05:57.182064 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:57.182023    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s" podUID="78230db8-1d2b-4d35-b025-c54111d5bfd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 13:05:57.305364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:57.272992    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76d4505d-2bd3-491d-99c2-e4e3e6942327" path="/var/lib/kubelet/pods/76d4505d-2bd3-491d-99c2-e4e3e6942327/volumes"
Oct 30 13:05:57.364036 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:57.363286    4544 patch_prober.go:29] interesting pod/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 13:05:57.383378 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:57.383321    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz" podUID="5d0c65ab-624c-40ab-9529-327c79d6567a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 13:05:58.173120 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:58.173079    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:58.339243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:58.338648    4544 patch_prober.go:29] interesting pod/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 13:05:58.339243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:58.338724    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz" podUID="5d0c65ab-624c-40ab-9529-327c79d6567a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 13:05:58.453627 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:58.453542    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:05:58.456282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:58.456092    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:05:58.465662 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:58.465574    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:05:58.708303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:58.699329    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:58.930252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:58.904560    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 13:05:58.975934 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:58.964712    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:05:59.363556 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:59.363519    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:05:59.378322 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:59.370880    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:05:59.450880 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:05:59.450811    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:05:59.459403 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:59.458307    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:59.728348 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:59.716328    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:59.736266 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:59.728918    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:05:59.767369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:05:59.767303    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:06:00.025333 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:00.012208    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:06:00.769500 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:00.769462    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:01.418829 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:01.418780    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn"]
Oct 30 13:06:01.467834 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:01.467789    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb"]
Oct 30 13:06:01.575120 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:01.575069    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:01.688965 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:01.688872    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:02.728966 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:02.728931    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:03.605869 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:03.605817    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x"]
Oct 30 13:06:03.639914 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:03.639879    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p"]
Oct 30 13:06:03.759055 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:03.759017    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:03.915021 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:03.914928    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:04.034441 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:04.034397    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7cac70f-v85z2"]
Oct 30 13:06:04.040799 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:04.040756    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7cac70f-v85z2"]
Oct 30 13:06:04.230523 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:04.230434    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-68573cef-6d977f448c-rn2qv"]
Oct 30 13:06:04.230776 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:04.230720    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68573cef-6d977f448c-rn2qv" podUID="51d6cd93-1695-4201-bbe3-9eb7f3da0597" containerName="maas-api" containerID="cri-o://077d95b832f19ac56f7bfaa1f5a79ca9dd77eab4e30ef8302d2475938c063911" gracePeriod=30
Oct 30 13:06:04.295261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:04.295191    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:04.699059 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:04.698665    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68573cef-6d977f448c-rn2qv"
Oct 30 13:06:04.902505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:04.902114    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51d6cd93-1695-4201-bbe3-9eb7f3da0597-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51d6cd93-1695-4201-bbe3-9eb7f3da0597-openshift-service-ca\") pod \"51d6cd93-1695-4201-bbe3-9eb7f3da0597\" (UID: \"51d6cd93-1695-4201-bbe3-9eb7f3da0597\") "
Oct 30 13:06:04.902505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:04.902183    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51d6cd93-1695-4201-bbe3-9eb7f3da0597-kube-api-access-8mjpx\" (UniqueName: \"kubernetes.io/projected/51d6cd93-1695-4201-bbe3-9eb7f3da0597-kube-api-access-8mjpx\") pod \"51d6cd93-1695-4201-bbe3-9eb7f3da0597\" (UID: \"51d6cd93-1695-4201-bbe3-9eb7f3da0597\") "
Oct 30 13:06:04.902505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:04.902328    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51d6cd93-1695-4201-bbe3-9eb7f3da0597-metrics-certs\" (UniqueName: \"kubernetes.io/secret/51d6cd93-1695-4201-bbe3-9eb7f3da0597-metrics-certs\") pod \"51d6cd93-1695-4201-bbe3-9eb7f3da0597\" (UID: \"51d6cd93-1695-4201-bbe3-9eb7f3da0597\") "
Oct 30 13:06:04.903738 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:04.903292    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51d6cd93-1695-4201-bbe3-9eb7f3da0597-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51d6cd93-1695-4201-bbe3-9eb7f3da0597-maas-api-tls\") pod \"51d6cd93-1695-4201-bbe3-9eb7f3da0597\" (UID: \"51d6cd93-1695-4201-bbe3-9eb7f3da0597\") "
Oct 30 13:06:04.906690 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:04.906245    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51d6cd93-1695-4201-bbe3-9eb7f3da0597-openshift-service-ca" pod "51d6cd93-1695-4201-bbe3-9eb7f3da0597" (UID: "51d6cd93-1695-4201-bbe3-9eb7f3da0597"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:04.931172 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:04.931109    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51d6cd93-1695-4201-bbe3-9eb7f3da0597-maas-api-tls" pod "51d6cd93-1695-4201-bbe3-9eb7f3da0597" (UID: "51d6cd93-1695-4201-bbe3-9eb7f3da0597"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:06:04.965304 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:04.964275    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51d6cd93-1695-4201-bbe3-9eb7f3da0597-metrics-certs" pod "51d6cd93-1695-4201-bbe3-9eb7f3da0597" (UID: "51d6cd93-1695-4201-bbe3-9eb7f3da0597"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:06:04.965304 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:04.964843    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51d6cd93-1695-4201-bbe3-9eb7f3da0597-kube-api-access-8mjpx" pod "51d6cd93-1695-4201-bbe3-9eb7f3da0597" (UID: "51d6cd93-1695-4201-bbe3-9eb7f3da0597"). InnerVolumeSpecName "kube-api-access-8mjpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:05.043263 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:05.006756    4544 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51d6cd93-1695-4201-bbe3-9eb7f3da0597-openshift-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:06:05.043263 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:05.006800    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mjpx\" (UniqueName: \"kubernetes.io/projected/51d6cd93-1695-4201-bbe3-9eb7f3da0597-kube-api-access-8mjpx\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:06:05.043263 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:05.006817    4544 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51d6cd93-1695-4201-bbe3-9eb7f3da0597-metrics-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:06:05.043263 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:05.006831    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51d6cd93-1695-4201-bbe3-9eb7f3da0597-maas-api-tls\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:06:05.228718 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:05.228623    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:05.229528 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:05.229499    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="184b9e84-a59d-49f2-8e76-0af679c4f148" path="/var/lib/kubelet/pods/184b9e84-a59d-49f2-8e76-0af679c4f148/volumes"
Oct 30 13:06:05.230353 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:05.230328    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6347ffc7-tzb98"]
Oct 30 13:06:05.231393 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:05.231365    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6347ffc7-tzb98"]
Oct 30 13:06:05.347782 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:05.347745    4544 generic.go:356] "Generic (PLEG): container finished" podID="51d6cd93-1695-4201-bbe3-9eb7f3da0597" containerID="077d95b832f19ac56f7bfaa1f5a79ca9dd77eab4e30ef8302d2475938c063911" exitCode=0
Oct 30 13:06:05.347989 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:05.347839    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68573cef-6d977f448c-rn2qv"
Oct 30 13:06:05.348055 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:05.348009    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68573cef-6d977f448c-rn2qv" event={"ID":"51d6cd93-1695-4201-bbe3-9eb7f3da0597","Type":"ContainerDied","Data":"077d95b832f19ac56f7bfaa1f5a79ca9dd77eab4e30ef8302d2475938c063911"}
Oct 30 13:06:05.348122 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:05.348067    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68573cef-6d977f448c-rn2qv" event={"ID":"51d6cd93-1695-4201-bbe3-9eb7f3da0597","Type":"ContainerDied","Data":"aca8f80c49917634d1007d26d580ddeb6c19a0dd6a2cf8b40b9a9fd8e16bf8d7"}
Oct 30 13:06:05.348122 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:05.348102    4544 scope.go:122] "RemoveContainer" containerID="077d95b832f19ac56f7bfaa1f5a79ca9dd77eab4e30ef8302d2475938c063911"
Oct 30 13:06:05.361747 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:05.361721    4544 scope.go:122] "RemoveContainer" containerID="077d95b832f19ac56f7bfaa1f5a79ca9dd77eab4e30ef8302d2475938c063911"
Oct 30 13:06:05.362133 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:06:05.362113    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"077d95b832f19ac56f7bfaa1f5a79ca9dd77eab4e30ef8302d2475938c063911\": container with ID starting with 077d95b832f19ac56f7bfaa1f5a79ca9dd77eab4e30ef8302d2475938c063911 not found: ID does not exist" containerID="077d95b832f19ac56f7bfaa1f5a79ca9dd77eab4e30ef8302d2475938c063911"
Oct 30 13:06:05.362182 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:05.362143    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"077d95b832f19ac56f7bfaa1f5a79ca9dd77eab4e30ef8302d2475938c063911"} err="failed to get container status \"077d95b832f19ac56f7bfaa1f5a79ca9dd77eab4e30ef8302d2475938c063911\": rpc error: code = NotFound desc = could not find container \"077d95b832f19ac56f7bfaa1f5a79ca9dd77eab4e30ef8302d2475938c063911\": container with ID starting with 077d95b832f19ac56f7bfaa1f5a79ca9dd77eab4e30ef8302d2475938c063911 not found: ID does not exist"
Oct 30 13:06:05.394341 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:05.394311    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:05.402306 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:05.402270    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-68573cef-6d977f448c-rn2qv"]
Oct 30 13:06:05.408264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:05.408209    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-68573cef-6d977f448c-rn2qv"]
Oct 30 13:06:05.544255 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:05.544149    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:06.035237 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:06.035182    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-332f4078-zltrf"]
Oct 30 13:06:06.040695 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:06.040660    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-332f4078-zltrf"]
Oct 30 13:06:06.419539 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:06.419493    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn" podUID="b46598ca-1b3b-45b6-9f6d-3ba5edcc539d" containerName="payload-processing" containerID="cri-o://1732ef6b1eee696f7205ddf69b7ce54454b9289f64fc376bd7be427c087da148" gracePeriod=25
Oct 30 13:06:06.468068 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:06.468025    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb" podUID="489f7620-9d5f-40e9-9225-1aee1181fb5c" containerName="payload-pre-processing" containerID="cri-o://4eba388de30e57ec5b13b7d61caf3ebaf5b7547aa275ba2302d58a70d5dd1bce" gracePeriod=25
Oct 30 13:06:06.794975 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:06.794943    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn"
Oct 30 13:06:06.823043 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:06.823013    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb"
Oct 30 13:06:06.906311 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:06.906274    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/489f7620-9d5f-40e9-9225-1aee1181fb5c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489f7620-9d5f-40e9-9225-1aee1181fb5c-plugins-config-volume\") pod \"489f7620-9d5f-40e9-9225-1aee1181fb5c\" (UID: \"489f7620-9d5f-40e9-9225-1aee1181fb5c\") "
Oct 30 13:06:06.906506 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:06.906370    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/489f7620-9d5f-40e9-9225-1aee1181fb5c-kube-api-access-wbf8b\" (UniqueName: \"kubernetes.io/projected/489f7620-9d5f-40e9-9225-1aee1181fb5c-kube-api-access-wbf8b\") pod \"489f7620-9d5f-40e9-9225-1aee1181fb5c\" (UID: \"489f7620-9d5f-40e9-9225-1aee1181fb5c\") "
Oct 30 13:06:06.906771 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:06.906736    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/489f7620-9d5f-40e9-9225-1aee1181fb5c-plugins-config-volume" pod "489f7620-9d5f-40e9-9225-1aee1181fb5c" (UID: "489f7620-9d5f-40e9-9225-1aee1181fb5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:06.906917 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:06.906885    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b46598ca-1b3b-45b6-9f6d-3ba5edcc539d-kube-api-access-48gmr\" (UniqueName: \"kubernetes.io/projected/b46598ca-1b3b-45b6-9f6d-3ba5edcc539d-kube-api-access-48gmr\") pod \"b46598ca-1b3b-45b6-9f6d-3ba5edcc539d\" (UID: \"b46598ca-1b3b-45b6-9f6d-3ba5edcc539d\") "
Oct 30 13:06:06.906990 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:06.906979    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b46598ca-1b3b-45b6-9f6d-3ba5edcc539d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b46598ca-1b3b-45b6-9f6d-3ba5edcc539d-plugins-config-volume\") pod \"b46598ca-1b3b-45b6-9f6d-3ba5edcc539d\" (UID: \"b46598ca-1b3b-45b6-9f6d-3ba5edcc539d\") "
Oct 30 13:06:06.907721 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:06.907696    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b46598ca-1b3b-45b6-9f6d-3ba5edcc539d-plugins-config-volume" pod "b46598ca-1b3b-45b6-9f6d-3ba5edcc539d" (UID: "b46598ca-1b3b-45b6-9f6d-3ba5edcc539d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:06.907894 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:06.907878    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489f7620-9d5f-40e9-9225-1aee1181fb5c-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:06:06.907962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:06.907901    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b46598ca-1b3b-45b6-9f6d-3ba5edcc539d-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:06:06.909028 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:06.909004    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/489f7620-9d5f-40e9-9225-1aee1181fb5c-kube-api-access-wbf8b" pod "489f7620-9d5f-40e9-9225-1aee1181fb5c" (UID: "489f7620-9d5f-40e9-9225-1aee1181fb5c"). InnerVolumeSpecName "kube-api-access-wbf8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:06.909438 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:06.909413    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b46598ca-1b3b-45b6-9f6d-3ba5edcc539d-kube-api-access-48gmr" pod "b46598ca-1b3b-45b6-9f6d-3ba5edcc539d" (UID: "b46598ca-1b3b-45b6-9f6d-3ba5edcc539d"). InnerVolumeSpecName "kube-api-access-48gmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:07.010086 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:07.009998    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbf8b\" (UniqueName: \"kubernetes.io/projected/489f7620-9d5f-40e9-9225-1aee1181fb5c-kube-api-access-wbf8b\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:06:07.010086 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:07.010033    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48gmr\" (UniqueName: \"kubernetes.io/projected/b46598ca-1b3b-45b6-9f6d-3ba5edcc539d-kube-api-access-48gmr\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:06:07.092604 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:07.092560    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bca9a20-c06b-4047-bfa1-accd6355ae87" path="/var/lib/kubelet/pods/0bca9a20-c06b-4047-bfa1-accd6355ae87/volumes"
Oct 30 13:06:07.093725 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:07.093371    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a0441eb-217b-49ff-a9f5-33c5372c0442" path="/var/lib/kubelet/pods/2a0441eb-217b-49ff-a9f5-33c5372c0442/volumes"
Oct 30 13:06:07.094160 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:07.094099    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51d6cd93-1695-4201-bbe3-9eb7f3da0597" path="/var/lib/kubelet/pods/51d6cd93-1695-4201-bbe3-9eb7f3da0597/volumes"
Oct 30 13:06:07.129568 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:07.129535    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"]
Oct 30 13:06:07.129835 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:07.129803    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s" podUID="78230db8-1d2b-4d35-b025-c54111d5bfd2" containerName="istio-proxy" containerID="cri-o://28f1f39085a632e9b5cbf74eff37292ff5a3e87cfc396c33d042c33e543f101f" gracePeriod=30
Oct 30 13:06:07.183569 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:07.183531    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:07.205834 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:07.205794    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8"]
Oct 30 13:06:07.206070 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:07.206047    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8" podUID="6e34292a-71f4-4414-9a2d-4d69729c535c" containerName="maas-api" containerID="cri-o://528369278b00d236c4d6646499136326fdf39104e6ef893e9fa107107b8229db" gracePeriod=30
Oct 30 13:06:07.332492 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:07.332461    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:07.376097 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:07.376065    4544 generic.go:356] "Generic (PLEG): container finished" podID="489f7620-9d5f-40e9-9225-1aee1181fb5c" containerID="4eba388de30e57ec5b13b7d61caf3ebaf5b7547aa275ba2302d58a70d5dd1bce" exitCode=0
Oct 30 13:06:07.376310 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:07.376112    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb"
Oct 30 13:06:07.376310 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:07.376152    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb" event={"ID":"489f7620-9d5f-40e9-9225-1aee1181fb5c","Type":"ContainerDied","Data":"4eba388de30e57ec5b13b7d61caf3ebaf5b7547aa275ba2302d58a70d5dd1bce"}
Oct 30 13:06:07.376310 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:07.376234    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb" event={"ID":"489f7620-9d5f-40e9-9225-1aee1181fb5c","Type":"ContainerDied","Data":"61431b1246c05f8add3aeae0c9bbad0a4591c3d3ae48a230595d9104e85f8c4d"}
Oct 30 13:06:07.376310 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:07.376269    4544 scope.go:122] "RemoveContainer" containerID="4eba388de30e57ec5b13b7d61caf3ebaf5b7547aa275ba2302d58a70d5dd1bce"
Oct 30 13:06:07.378454 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:07.378427    4544 generic.go:356] "Generic (PLEG): container finished" podID="6e34292a-71f4-4414-9a2d-4d69729c535c" containerID="528369278b00d236c4d6646499136326fdf39104e6ef893e9fa107107b8229db" exitCode=0
Oct 30 13:06:07.378602 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:07.378584    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8" event={"ID":"6e34292a-71f4-4414-9a2d-4d69729c535c","Type":"ContainerDied","Data":"528369278b00d236c4d6646499136326fdf39104e6ef893e9fa107107b8229db"}
Oct 30 13:06:07.380142 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:07.380121    4544 generic.go:356] "Generic (PLEG): container finished" podID="b46598ca-1b3b-45b6-9f6d-3ba5edcc539d" containerID="1732ef6b1eee696f7205ddf69b7ce54454b9289f64fc376bd7be427c087da148" exitCode=0
Oct 30 13:06:07.380250 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:07.380187    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn"
Oct 30 13:06:07.380250 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:07.380201    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn" event={"ID":"b46598ca-1b3b-45b6-9f6d-3ba5edcc539d","Type":"ContainerDied","Data":"1732ef6b1eee696f7205ddf69b7ce54454b9289f64fc376bd7be427c087da148"}
Oct 30 13:06:07.380344 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:07.380255    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn" event={"ID":"b46598ca-1b3b-45b6-9f6d-3ba5edcc539d","Type":"ContainerDied","Data":"2928d550205a8b1acd15ccf111e834e7480937d831535380d0a3dbbc29e970c1"}
Oct 30 13:06:07.391026 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:07.391001    4544 scope.go:122] "RemoveContainer" containerID="4eba388de30e57ec5b13b7d61caf3ebaf5b7547aa275ba2302d58a70d5dd1bce"
Oct 30 13:06:07.391584 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:06:07.391400    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eba388de30e57ec5b13b7d61caf3ebaf5b7547aa275ba2302d58a70d5dd1bce\": container with ID starting with 4eba388de30e57ec5b13b7d61caf3ebaf5b7547aa275ba2302d58a70d5dd1bce not found: ID does not exist" containerID="4eba388de30e57ec5b13b7d61caf3ebaf5b7547aa275ba2302d58a70d5dd1bce"
Oct 30 13:06:07.391584 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:07.391442    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eba388de30e57ec5b13b7d61caf3ebaf5b7547aa275ba2302d58a70d5dd1bce"} err="failed to get container status \"4eba388de30e57ec5b13b7d61caf3ebaf5b7547aa275ba2302d58a70d5dd1bce\": rpc error: code = NotFound desc = could not find container \"4eba388de30e57ec5b13b7d61caf3ebaf5b7547aa275ba2302d58a70d5dd1bce\": container with ID starting with 4eba388de30e57ec5b13b7d61caf3ebaf5b7547aa275ba2302d58a70d5dd1bce not found: ID does not exist"
Oct 30 13:06:07.391584 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:07.391468    4544 scope.go:122] "RemoveContainer" containerID="1732ef6b1eee696f7205ddf69b7ce54454b9289f64fc376bd7be427c087da148"
Oct 30 13:06:07.407635 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:07.407610    4544 scope.go:122] "RemoveContainer" containerID="1732ef6b1eee696f7205ddf69b7ce54454b9289f64fc376bd7be427c087da148"
Oct 30 13:06:07.407940 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:06:07.407917    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1732ef6b1eee696f7205ddf69b7ce54454b9289f64fc376bd7be427c087da148\": container with ID starting with 1732ef6b1eee696f7205ddf69b7ce54454b9289f64fc376bd7be427c087da148 not found: ID does not exist" containerID="1732ef6b1eee696f7205ddf69b7ce54454b9289f64fc376bd7be427c087da148"
Oct 30 13:06:07.408008 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:07.407947    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1732ef6b1eee696f7205ddf69b7ce54454b9289f64fc376bd7be427c087da148"} err="failed to get container status \"1732ef6b1eee696f7205ddf69b7ce54454b9289f64fc376bd7be427c087da148\": rpc error: code = NotFound desc = could not find container \"1732ef6b1eee696f7205ddf69b7ce54454b9289f64fc376bd7be427c087da148\": container with ID starting with 1732ef6b1eee696f7205ddf69b7ce54454b9289f64fc376bd7be427c087da148 not found: ID does not exist"
Oct 30 13:06:07.439960 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:07.439886    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn"]
Oct 30 13:06:07.445418 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:07.445393    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn"]
Oct 30 13:06:07.556266 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:07.551294    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb"]
Oct 30 13:06:07.613253 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:07.604827    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68573cef-6d977f448c-rn2qv" podUID="51d6cd93-1695-4201-bbe3-9eb7f3da0597" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.129:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:06:07.725781 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:07.725744    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb"]
Oct 30 13:06:07.852210 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:07.852146    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:07.978271 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:07.973419    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8"
Oct 30 13:06:08.105365 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.098789    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:08.133276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.129182    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e34292a-71f4-4414-9a2d-4d69729c535c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e34292a-71f4-4414-9a2d-4d69729c535c-maas-api-tls\") pod \"6e34292a-71f4-4414-9a2d-4d69729c535c\" (UID: \"6e34292a-71f4-4414-9a2d-4d69729c535c\") "
Oct 30 13:06:08.133276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.129292    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e34292a-71f4-4414-9a2d-4d69729c535c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e34292a-71f4-4414-9a2d-4d69729c535c-openshift-service-ca\") pod \"6e34292a-71f4-4414-9a2d-4d69729c535c\" (UID: \"6e34292a-71f4-4414-9a2d-4d69729c535c\") "
Oct 30 13:06:08.133276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.129525    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e34292a-71f4-4414-9a2d-4d69729c535c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e34292a-71f4-4414-9a2d-4d69729c535c-metrics-certs\") pod \"6e34292a-71f4-4414-9a2d-4d69729c535c\" (UID: \"6e34292a-71f4-4414-9a2d-4d69729c535c\") "
Oct 30 13:06:08.133276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.129588    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e34292a-71f4-4414-9a2d-4d69729c535c-kube-api-access-mpq8j\" (UniqueName: \"kubernetes.io/projected/6e34292a-71f4-4414-9a2d-4d69729c535c-kube-api-access-mpq8j\") pod \"6e34292a-71f4-4414-9a2d-4d69729c535c\" (UID: \"6e34292a-71f4-4414-9a2d-4d69729c535c\") "
Oct 30 13:06:08.135885 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.135848    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e34292a-71f4-4414-9a2d-4d69729c535c-kube-api-access-mpq8j" pod "6e34292a-71f4-4414-9a2d-4d69729c535c" (UID: "6e34292a-71f4-4414-9a2d-4d69729c535c"). InnerVolumeSpecName "kube-api-access-mpq8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:08.160671 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.160630    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e34292a-71f4-4414-9a2d-4d69729c535c-openshift-service-ca" pod "6e34292a-71f4-4414-9a2d-4d69729c535c" (UID: "6e34292a-71f4-4414-9a2d-4d69729c535c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:08.197290 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.195022    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e34292a-71f4-4414-9a2d-4d69729c535c-metrics-certs" pod "6e34292a-71f4-4414-9a2d-4d69729c535c" (UID: "6e34292a-71f4-4414-9a2d-4d69729c535c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:06:08.206382 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.206341    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e34292a-71f4-4414-9a2d-4d69729c535c-maas-api-tls" pod "6e34292a-71f4-4414-9a2d-4d69729c535c" (UID: "6e34292a-71f4-4414-9a2d-4d69729c535c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:06:08.232297 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.232258    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e34292a-71f4-4414-9a2d-4d69729c535c-maas-api-tls\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:06:08.232297 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.232297    4544 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e34292a-71f4-4414-9a2d-4d69729c535c-openshift-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:06:08.232533 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.232335    4544 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e34292a-71f4-4414-9a2d-4d69729c535c-metrics-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:06:08.232533 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.232355    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpq8j\" (UniqueName: \"kubernetes.io/projected/6e34292a-71f4-4414-9a2d-4d69729c535c-kube-api-access-mpq8j\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:06:08.321017 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.320966    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:08.371126 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.371040    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"]
Oct 30 13:06:08.373007 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.372894    4544 cpu_manager.go:414] "containerMap: removing container" podUID="b46598ca-1b3b-45b6-9f6d-3ba5edcc539d" containerName="payload-processing"
Oct 30 13:06:08.373169 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.373152    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b46598ca-1b3b-45b6-9f6d-3ba5edcc539d" containerName="payload-processing"
Oct 30 13:06:08.373297 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.373284    4544 cpu_manager.go:414] "containerMap: removing container" podUID="6e34292a-71f4-4414-9a2d-4d69729c535c" containerName="maas-api"
Oct 30 13:06:08.373383 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.373372    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e34292a-71f4-4414-9a2d-4d69729c535c" containerName="maas-api"
Oct 30 13:06:08.373534 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.373522    4544 cpu_manager.go:414] "containerMap: removing container" podUID="489f7620-9d5f-40e9-9225-1aee1181fb5c" containerName="payload-pre-processing"
Oct 30 13:06:08.373615 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.373604    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="489f7620-9d5f-40e9-9225-1aee1181fb5c" containerName="payload-pre-processing"
Oct 30 13:06:08.373716 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.373705    4544 cpu_manager.go:414] "containerMap: removing container" podUID="51d6cd93-1695-4201-bbe3-9eb7f3da0597" containerName="maas-api"
Oct 30 13:06:08.373838 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.373816    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51d6cd93-1695-4201-bbe3-9eb7f3da0597" containerName="maas-api"
Oct 30 13:06:08.374238 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.374205    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="489f7620-9d5f-40e9-9225-1aee1181fb5c" containerName="payload-pre-processing"
Oct 30 13:06:08.374360 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.374348    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="51d6cd93-1695-4201-bbe3-9eb7f3da0597" containerName="maas-api"
Oct 30 13:06:08.374455 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.374444    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="b46598ca-1b3b-45b6-9f6d-3ba5edcc539d" containerName="payload-processing"
Oct 30 13:06:08.374533 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.374522    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e34292a-71f4-4414-9a2d-4d69729c535c" containerName="maas-api"
Oct 30 13:06:08.379505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.379479    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:08.402531 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.402495    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8" event={"ID":"6e34292a-71f4-4414-9a2d-4d69729c535c","Type":"ContainerDied","Data":"76872ad67a557d4ef08eb6837dbbc3dd4e023f1250b5eff66ed72d94ebc8b0c6"}
Oct 30 13:06:08.402673 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.402557    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8"
Oct 30 13:06:08.402673 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.402584    4544 scope.go:122] "RemoveContainer" containerID="528369278b00d236c4d6646499136326fdf39104e6ef893e9fa107107b8229db"
Oct 30 13:06:08.405135 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.404402    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-5af2de67-openshift-default-dockercfg-fkn4h\""
Oct 30 13:06:08.407493 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.407468    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"]
Oct 30 13:06:08.437460 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.437423    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-credential-socket\") pod \"e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") " pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:08.437669 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.437578    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq5v5\" (UniqueName: \"kubernetes.io/projected/4b940dbd-f31b-4620-84e8-f32fc7feef3e-kube-api-access-rq5v5\") pod \"e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") " pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:08.437669 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.437626    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-workload-certs\") pod \"e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") " pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:08.437789 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.437704    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istio-envoy\") pod \"e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") " pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:08.437789 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.437768    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istio-data\") pod \"e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") " pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:08.437968 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.437841    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-workload-socket\") pod \"e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") " pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:08.437968 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.437952    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istio-token\") pod \"e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") " pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:08.438073 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.438005    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istio-podinfo\") pod \"e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") " pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:08.438150 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.438132    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istiod-ca-cert\") pod \"e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") " pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:08.444124 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.444092    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:08.493121 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.493088    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8"]
Oct 30 13:06:08.503707 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.503669    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8"]
Oct 30 13:06:08.515546 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.515507    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"]
Oct 30 13:06:08.515830 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.515795    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz" podUID="5d0c65ab-624c-40ab-9529-327c79d6567a" containerName="istio-proxy" containerID="cri-o://c1d1c97d72b74201dadc51e3c35547c2dba1c9752d4a4f674d097fe8c44c5aea" gracePeriod=30
Oct 30 13:06:08.539560 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.539524    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istio-token\") pod \"e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") " pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:08.539740 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.539591    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istio-podinfo\") pod \"e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") " pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:08.539740 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.539700    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istiod-ca-cert\") pod \"e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") " pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:08.539740 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.539733    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-credential-socket\") pod \"e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") " pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:08.539929 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.539815    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq5v5\" (UniqueName: \"kubernetes.io/projected/4b940dbd-f31b-4620-84e8-f32fc7feef3e-kube-api-access-rq5v5\") pod \"e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") " pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:08.539929 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.539852    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-workload-certs\") pod \"e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") " pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:08.540031 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.539934    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istio-envoy\") pod \"e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") " pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:08.540031 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.539998    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istio-data\") pod \"e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") " pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:08.540133 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.540061    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-workload-socket\") pod \"e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") " pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:08.540568 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.540538    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-workload-socket\") pod \"e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") " pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:08.541912 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.541679    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-credential-socket\") pod \"e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") " pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:08.541912 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.541876    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-workload-certs\") pod \"e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") " pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:08.542373 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.542333    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istio-data\") pod \"e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") " pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:08.543338 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.543319    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istiod-ca-cert\") pod \"e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") " pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:08.555264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.555187    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istio-envoy\") pod \"e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") " pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:08.555996 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.555976    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istio-podinfo\") pod \"e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") " pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:08.563981 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.563958    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istio-token\") pod \"e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") " pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:08.564325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.564303    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq5v5\" (UniqueName: \"kubernetes.io/projected/4b940dbd-f31b-4620-84e8-f32fc7feef3e-kube-api-access-rq5v5\") pod \"e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") " pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:08.574594 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.574571    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:08.607118 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.606294    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x" podUID="e4a68695-2f4b-4a06-938e-56e656c5c445" containerName="payload-processing" containerID="cri-o://7128f05c6b3fe7f1c4bd1a7e30b5fee27ac37d4fdcbbe481a874d1e1e75edf21" gracePeriod=25
Oct 30 13:06:08.674327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.645035    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p" podUID="53b7be9d-ff23-452d-b121-12500f7d6787" containerName="payload-pre-processing" containerID="cri-o://1dae6ab7ec66e79def5da0ba16d54d583ed9b92aae6db49e3d49ff6093fd4def" gracePeriod=25
Oct 30 13:06:08.710560 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:08.710522    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:09.091756 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.091719    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="489f7620-9d5f-40e9-9225-1aee1181fb5c" path="/var/lib/kubelet/pods/489f7620-9d5f-40e9-9225-1aee1181fb5c/volumes"
Oct 30 13:06:09.092186 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.092170    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e34292a-71f4-4414-9a2d-4d69729c535c" path="/var/lib/kubelet/pods/6e34292a-71f4-4414-9a2d-4d69729c535c/volumes"
Oct 30 13:06:09.092602 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.092585    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b46598ca-1b3b-45b6-9f6d-3ba5edcc539d" path="/var/lib/kubelet/pods/b46598ca-1b3b-45b6-9f6d-3ba5edcc539d/volumes"
Oct 30 13:06:09.188265 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.188180    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x"
Oct 30 13:06:09.225605 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.225578    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p"
Oct 30 13:06:09.292356 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.292321    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4a68695-2f4b-4a06-938e-56e656c5c445-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a68695-2f4b-4a06-938e-56e656c5c445-plugins-config-volume\") pod \"e4a68695-2f4b-4a06-938e-56e656c5c445\" (UID: \"e4a68695-2f4b-4a06-938e-56e656c5c445\") "
Oct 30 13:06:09.292562 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.292544    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4a68695-2f4b-4a06-938e-56e656c5c445-kube-api-access-h7fw8\" (UniqueName: \"kubernetes.io/projected/e4a68695-2f4b-4a06-938e-56e656c5c445-kube-api-access-h7fw8\") pod \"e4a68695-2f4b-4a06-938e-56e656c5c445\" (UID: \"e4a68695-2f4b-4a06-938e-56e656c5c445\") "
Oct 30 13:06:09.293125 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.293081    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4a68695-2f4b-4a06-938e-56e656c5c445-plugins-config-volume" pod "e4a68695-2f4b-4a06-938e-56e656c5c445" (UID: "e4a68695-2f4b-4a06-938e-56e656c5c445"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:09.295084 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.295060    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4a68695-2f4b-4a06-938e-56e656c5c445-kube-api-access-h7fw8" pod "e4a68695-2f4b-4a06-938e-56e656c5c445" (UID: "e4a68695-2f4b-4a06-938e-56e656c5c445"). InnerVolumeSpecName "kube-api-access-h7fw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:09.298517 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:06:09.298476    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b940dbd_f31b_4620_84e8_f32fc7feef3e.slice/crio-dbc7151097c51d110e5897179f53809bae0da3d50a8d0c81e41c07ac6cc2738c WatchSource:0}: Error finding container dbc7151097c51d110e5897179f53809bae0da3d50a8d0c81e41c07ac6cc2738c: Status 404 returned error can't find the container with id dbc7151097c51d110e5897179f53809bae0da3d50a8d0c81e41c07ac6cc2738c
Oct 30 13:06:09.299143 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.299114    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"]
Oct 30 13:06:09.301369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.301336    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:06:09.301456 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.301402    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:06:09.301456 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.301439    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:06:09.394403 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.394371    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53b7be9d-ff23-452d-b121-12500f7d6787-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53b7be9d-ff23-452d-b121-12500f7d6787-plugins-config-volume\") pod \"53b7be9d-ff23-452d-b121-12500f7d6787\" (UID: \"53b7be9d-ff23-452d-b121-12500f7d6787\") "
Oct 30 13:06:09.394736 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.394716    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53b7be9d-ff23-452d-b121-12500f7d6787-kube-api-access-dsprd\" (UniqueName: \"kubernetes.io/projected/53b7be9d-ff23-452d-b121-12500f7d6787-kube-api-access-dsprd\") pod \"53b7be9d-ff23-452d-b121-12500f7d6787\" (UID: \"53b7be9d-ff23-452d-b121-12500f7d6787\") "
Oct 30 13:06:09.394856 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.394715    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53b7be9d-ff23-452d-b121-12500f7d6787-plugins-config-volume" pod "53b7be9d-ff23-452d-b121-12500f7d6787" (UID: "53b7be9d-ff23-452d-b121-12500f7d6787"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:09.395342 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.395320    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7fw8\" (UniqueName: \"kubernetes.io/projected/e4a68695-2f4b-4a06-938e-56e656c5c445-kube-api-access-h7fw8\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:06:09.395342 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.395340    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a68695-2f4b-4a06-938e-56e656c5c445-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:06:09.395531 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.395350    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53b7be9d-ff23-452d-b121-12500f7d6787-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:06:09.396809 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.396783    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53b7be9d-ff23-452d-b121-12500f7d6787-kube-api-access-dsprd" pod "53b7be9d-ff23-452d-b121-12500f7d6787" (UID: "53b7be9d-ff23-452d-b121-12500f7d6787"). InnerVolumeSpecName "kube-api-access-dsprd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:09.415409 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.415378    4544 generic.go:356] "Generic (PLEG): container finished" podID="e4a68695-2f4b-4a06-938e-56e656c5c445" containerID="7128f05c6b3fe7f1c4bd1a7e30b5fee27ac37d4fdcbbe481a874d1e1e75edf21" exitCode=0
Oct 30 13:06:09.415594 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.415444    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x"
Oct 30 13:06:09.415594 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.415461    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x" event={"ID":"e4a68695-2f4b-4a06-938e-56e656c5c445","Type":"ContainerDied","Data":"7128f05c6b3fe7f1c4bd1a7e30b5fee27ac37d4fdcbbe481a874d1e1e75edf21"}
Oct 30 13:06:09.415594 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.415525    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x" event={"ID":"e4a68695-2f4b-4a06-938e-56e656c5c445","Type":"ContainerDied","Data":"42aa7bbe7fda9262ee35147bc6db6ea6f33674bfe28fbd20759405a79d89d6dd"}
Oct 30 13:06:09.415594 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.415556    4544 scope.go:122] "RemoveContainer" containerID="7128f05c6b3fe7f1c4bd1a7e30b5fee27ac37d4fdcbbe481a874d1e1e75edf21"
Oct 30 13:06:09.419152 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.419111    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9" event={"ID":"4b940dbd-f31b-4620-84e8-f32fc7feef3e","Type":"ContainerStarted","Data":"2b0a772caa410bddc4a83bdb14a6d6415ab43bf8b561824b8f14ba86d8f75adc"}
Oct 30 13:06:09.419285 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.419183    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9" event={"ID":"4b940dbd-f31b-4620-84e8-f32fc7feef3e","Type":"ContainerStarted","Data":"dbc7151097c51d110e5897179f53809bae0da3d50a8d0c81e41c07ac6cc2738c"}
Oct 30 13:06:09.420964 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.420785    4544 generic.go:356] "Generic (PLEG): container finished" podID="53b7be9d-ff23-452d-b121-12500f7d6787" containerID="1dae6ab7ec66e79def5da0ba16d54d583ed9b92aae6db49e3d49ff6093fd4def" exitCode=0
Oct 30 13:06:09.420964 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.420859    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p"
Oct 30 13:06:09.420964 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.420893    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p" event={"ID":"53b7be9d-ff23-452d-b121-12500f7d6787","Type":"ContainerDied","Data":"1dae6ab7ec66e79def5da0ba16d54d583ed9b92aae6db49e3d49ff6093fd4def"}
Oct 30 13:06:09.420964 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.420934    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p" event={"ID":"53b7be9d-ff23-452d-b121-12500f7d6787","Type":"ContainerDied","Data":"8d572d7a7f2378cee869889a669f3fa4196ed607d814def5caa4a7c236ae05bd"}
Oct 30 13:06:09.433464 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.433439    4544 scope.go:122] "RemoveContainer" containerID="7128f05c6b3fe7f1c4bd1a7e30b5fee27ac37d4fdcbbe481a874d1e1e75edf21"
Oct 30 13:06:09.435077 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:06:09.434861    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7128f05c6b3fe7f1c4bd1a7e30b5fee27ac37d4fdcbbe481a874d1e1e75edf21\": container with ID starting with 7128f05c6b3fe7f1c4bd1a7e30b5fee27ac37d4fdcbbe481a874d1e1e75edf21 not found: ID does not exist" containerID="7128f05c6b3fe7f1c4bd1a7e30b5fee27ac37d4fdcbbe481a874d1e1e75edf21"
Oct 30 13:06:09.435077 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.434910    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7128f05c6b3fe7f1c4bd1a7e30b5fee27ac37d4fdcbbe481a874d1e1e75edf21"} err="failed to get container status \"7128f05c6b3fe7f1c4bd1a7e30b5fee27ac37d4fdcbbe481a874d1e1e75edf21\": rpc error: code = NotFound desc = could not find container \"7128f05c6b3fe7f1c4bd1a7e30b5fee27ac37d4fdcbbe481a874d1e1e75edf21\": container with ID starting with 7128f05c6b3fe7f1c4bd1a7e30b5fee27ac37d4fdcbbe481a874d1e1e75edf21 not found: ID does not exist"
Oct 30 13:06:09.435077 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.434949    4544 scope.go:122] "RemoveContainer" containerID="1dae6ab7ec66e79def5da0ba16d54d583ed9b92aae6db49e3d49ff6093fd4def"
Oct 30 13:06:09.479067 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.479042    4544 scope.go:122] "RemoveContainer" containerID="1dae6ab7ec66e79def5da0ba16d54d583ed9b92aae6db49e3d49ff6093fd4def"
Oct 30 13:06:09.479538 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:06:09.479499    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dae6ab7ec66e79def5da0ba16d54d583ed9b92aae6db49e3d49ff6093fd4def\": container with ID starting with 1dae6ab7ec66e79def5da0ba16d54d583ed9b92aae6db49e3d49ff6093fd4def not found: ID does not exist" containerID="1dae6ab7ec66e79def5da0ba16d54d583ed9b92aae6db49e3d49ff6093fd4def"
Oct 30 13:06:09.479673 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.479538    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dae6ab7ec66e79def5da0ba16d54d583ed9b92aae6db49e3d49ff6093fd4def"} err="failed to get container status \"1dae6ab7ec66e79def5da0ba16d54d583ed9b92aae6db49e3d49ff6093fd4def\": rpc error: code = NotFound desc = could not find container \"1dae6ab7ec66e79def5da0ba16d54d583ed9b92aae6db49e3d49ff6093fd4def\": container with ID starting with 1dae6ab7ec66e79def5da0ba16d54d583ed9b92aae6db49e3d49ff6093fd4def not found: ID does not exist"
Oct 30 13:06:09.496985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.496950    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsprd\" (UniqueName: \"kubernetes.io/projected/53b7be9d-ff23-452d-b121-12500f7d6787-kube-api-access-dsprd\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:06:09.513855 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.513793    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9" podStartSLOduration=1.5137782039999998 podStartE2EDuration="1.513778204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:09.507828234 +0000 UTC m=+1264.554962210" watchObservedRunningTime="2026-10-30 13:06:09.513778204 +0000 UTC m=+1264.560912157"
Oct 30 13:06:09.534379 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.534347    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:09.554973 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.554934    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p"]
Oct 30 13:06:09.561638 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.561607    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p"]
Oct 30 13:06:09.602053 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.602017    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x"]
Oct 30 13:06:09.613403 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.613368    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x"]
Oct 30 13:06:09.675475 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.675439    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:09.719301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:09.713381    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:10.552466 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:10.552416    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-752d3a58-b595d49bc-vd9v8" podUID="6e34292a-71f4-4414-9a2d-4d69729c535c" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.130:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:06:10.718328 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:10.718291    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:10.990236 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:10.990175    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:11.096088 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:11.095977    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53b7be9d-ff23-452d-b121-12500f7d6787" path="/var/lib/kubelet/pods/53b7be9d-ff23-452d-b121-12500f7d6787/volumes"
Oct 30 13:06:11.097170 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:11.097137    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4a68695-2f4b-4a06-938e-56e656c5c445" path="/var/lib/kubelet/pods/e4a68695-2f4b-4a06-938e-56e656c5c445/volumes"
Oct 30 13:06:11.161137 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:11.160838    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:11.291299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:11.291197    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:11.440010 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:11.439979    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:11.441069 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:11.441046    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:12.044147 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.044111    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c46fc70e-wwlhp"]
Oct 30 13:06:12.055325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.055293    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c46fc70e-wwlhp"]
Oct 30 13:06:12.427545 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.427519    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:06:12.451347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.451314    4544 generic.go:356] "Generic (PLEG): container finished" podID="78230db8-1d2b-4d35-b025-c54111d5bfd2" containerID="28f1f39085a632e9b5cbf74eff37292ff5a3e87cfc396c33d042c33e543f101f" exitCode=0
Oct 30 13:06:12.451549 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.451377    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"
Oct 30 13:06:12.451549 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.451402    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s" event={"ID":"78230db8-1d2b-4d35-b025-c54111d5bfd2","Type":"ContainerDied","Data":"28f1f39085a632e9b5cbf74eff37292ff5a3e87cfc396c33d042c33e543f101f"}
Oct 30 13:06:12.451549 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.451466    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s" event={"ID":"78230db8-1d2b-4d35-b025-c54111d5bfd2","Type":"ContainerDied","Data":"b9b2f2b78571d21414f7dcfbb77bcf5768d3662d25843da90f35d1d038cd4ea0"}
Oct 30 13:06:12.451549 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.451502    4544 scope.go:122] "RemoveContainer" containerID="28f1f39085a632e9b5cbf74eff37292ff5a3e87cfc396c33d042c33e543f101f"
Oct 30 13:06:12.464366 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.464338    4544 scope.go:122] "RemoveContainer" containerID="7114a1bbff5bec10c67ee27609e2050975014ad8c423d168677467f74811f98f"
Oct 30 13:06:12.476491 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.476462    4544 scope.go:122] "RemoveContainer" containerID="28f1f39085a632e9b5cbf74eff37292ff5a3e87cfc396c33d042c33e543f101f"
Oct 30 13:06:12.476807 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:06:12.476780    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28f1f39085a632e9b5cbf74eff37292ff5a3e87cfc396c33d042c33e543f101f\": container with ID starting with 28f1f39085a632e9b5cbf74eff37292ff5a3e87cfc396c33d042c33e543f101f not found: ID does not exist" containerID="28f1f39085a632e9b5cbf74eff37292ff5a3e87cfc396c33d042c33e543f101f"
Oct 30 13:06:12.476864 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.476818    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28f1f39085a632e9b5cbf74eff37292ff5a3e87cfc396c33d042c33e543f101f"} err="failed to get container status \"28f1f39085a632e9b5cbf74eff37292ff5a3e87cfc396c33d042c33e543f101f\": rpc error: code = NotFound desc = could not find container \"28f1f39085a632e9b5cbf74eff37292ff5a3e87cfc396c33d042c33e543f101f\": container with ID starting with 28f1f39085a632e9b5cbf74eff37292ff5a3e87cfc396c33d042c33e543f101f not found: ID does not exist"
Oct 30 13:06:12.476864 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.476838    4544 scope.go:122] "RemoveContainer" containerID="7114a1bbff5bec10c67ee27609e2050975014ad8c423d168677467f74811f98f"
Oct 30 13:06:12.477135 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:06:12.477114    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7114a1bbff5bec10c67ee27609e2050975014ad8c423d168677467f74811f98f\": container with ID starting with 7114a1bbff5bec10c67ee27609e2050975014ad8c423d168677467f74811f98f not found: ID does not exist" containerID="7114a1bbff5bec10c67ee27609e2050975014ad8c423d168677467f74811f98f"
Oct 30 13:06:12.477210 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.477146    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7114a1bbff5bec10c67ee27609e2050975014ad8c423d168677467f74811f98f"} err="failed to get container status \"7114a1bbff5bec10c67ee27609e2050975014ad8c423d168677467f74811f98f\": rpc error: code = NotFound desc = could not find container \"7114a1bbff5bec10c67ee27609e2050975014ad8c423d168677467f74811f98f\": container with ID starting with 7114a1bbff5bec10c67ee27609e2050975014ad8c423d168677467f74811f98f not found: ID does not exist"
Oct 30 13:06:12.571760 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.571674    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-istio-data\") pod \"78230db8-1d2b-4d35-b025-c54111d5bfd2\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") "
Oct 30 13:06:12.571760 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.571743    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78230db8-1d2b-4d35-b025-c54111d5bfd2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78230db8-1d2b-4d35-b025-c54111d5bfd2-istiod-ca-cert\") pod \"78230db8-1d2b-4d35-b025-c54111d5bfd2\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") "
Oct 30 13:06:12.572063 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.571774    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78230db8-1d2b-4d35-b025-c54111d5bfd2-istio-token\" (UniqueName: \"kubernetes.io/projected/78230db8-1d2b-4d35-b025-c54111d5bfd2-istio-token\") pod \"78230db8-1d2b-4d35-b025-c54111d5bfd2\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") "
Oct 30 13:06:12.572063 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.571801    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78230db8-1d2b-4d35-b025-c54111d5bfd2-kube-api-access-cgkf8\" (UniqueName: \"kubernetes.io/projected/78230db8-1d2b-4d35-b025-c54111d5bfd2-kube-api-access-cgkf8\") pod \"78230db8-1d2b-4d35-b025-c54111d5bfd2\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") "
Oct 30 13:06:12.572063 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.571887    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-istio-envoy\") pod \"78230db8-1d2b-4d35-b025-c54111d5bfd2\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") "
Oct 30 13:06:12.572063 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.571948    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-istio-data" pod "78230db8-1d2b-4d35-b025-c54111d5bfd2" (UID: "78230db8-1d2b-4d35-b025-c54111d5bfd2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:12.572063 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.571992    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-workload-socket\") pod \"78230db8-1d2b-4d35-b025-c54111d5bfd2\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") "
Oct 30 13:06:12.572063 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.572031    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-credential-socket\") pod \"78230db8-1d2b-4d35-b025-c54111d5bfd2\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") "
Oct 30 13:06:12.572405 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.572104    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-workload-certs\") pod \"78230db8-1d2b-4d35-b025-c54111d5bfd2\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") "
Oct 30 13:06:12.572405 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.572174    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78230db8-1d2b-4d35-b025-c54111d5bfd2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78230db8-1d2b-4d35-b025-c54111d5bfd2-istio-podinfo\") pod \"78230db8-1d2b-4d35-b025-c54111d5bfd2\" (UID: \"78230db8-1d2b-4d35-b025-c54111d5bfd2\") "
Oct 30 13:06:12.572405 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.572282    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-workload-socket" pod "78230db8-1d2b-4d35-b025-c54111d5bfd2" (UID: "78230db8-1d2b-4d35-b025-c54111d5bfd2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:12.572405 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.572322    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78230db8-1d2b-4d35-b025-c54111d5bfd2-istiod-ca-cert" pod "78230db8-1d2b-4d35-b025-c54111d5bfd2" (UID: "78230db8-1d2b-4d35-b025-c54111d5bfd2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:12.572564 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.572437    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-credential-socket" pod "78230db8-1d2b-4d35-b025-c54111d5bfd2" (UID: "78230db8-1d2b-4d35-b025-c54111d5bfd2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:12.572564 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.572490    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-workload-certs" pod "78230db8-1d2b-4d35-b025-c54111d5bfd2" (UID: "78230db8-1d2b-4d35-b025-c54111d5bfd2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:12.573054 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.573029    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:06:12.573167 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.573057    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:06:12.573167 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.573072    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78230db8-1d2b-4d35-b025-c54111d5bfd2-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:06:12.573167 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.573084    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:06:12.573167 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.573097    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:06:12.574704 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.574671    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-istio-envoy" pod "78230db8-1d2b-4d35-b025-c54111d5bfd2" (UID: "78230db8-1d2b-4d35-b025-c54111d5bfd2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:12.574704 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.574683    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78230db8-1d2b-4d35-b025-c54111d5bfd2-istio-podinfo" pod "78230db8-1d2b-4d35-b025-c54111d5bfd2" (UID: "78230db8-1d2b-4d35-b025-c54111d5bfd2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:06:12.574896 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.574881    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78230db8-1d2b-4d35-b025-c54111d5bfd2-kube-api-access-cgkf8" pod "78230db8-1d2b-4d35-b025-c54111d5bfd2" (UID: "78230db8-1d2b-4d35-b025-c54111d5bfd2"). InnerVolumeSpecName "kube-api-access-cgkf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:12.574950 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.574903    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78230db8-1d2b-4d35-b025-c54111d5bfd2-istio-token" pod "78230db8-1d2b-4d35-b025-c54111d5bfd2" (UID: "78230db8-1d2b-4d35-b025-c54111d5bfd2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:12.674830 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.674788    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgkf8\" (UniqueName: \"kubernetes.io/projected/78230db8-1d2b-4d35-b025-c54111d5bfd2-kube-api-access-cgkf8\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:06:12.674830 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.674823    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78230db8-1d2b-4d35-b025-c54111d5bfd2-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:06:12.674830 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.674833    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78230db8-1d2b-4d35-b025-c54111d5bfd2-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:06:12.674830 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.674845    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78230db8-1d2b-4d35-b025-c54111d5bfd2-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:06:12.812907 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.812876    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"]
Oct 30 13:06:12.823017 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:12.822992    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s"]
Oct 30 13:06:13.087460 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.087353    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:13.089358 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.089316    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78230db8-1d2b-4d35-b025-c54111d5bfd2" path="/var/lib/kubelet/pods/78230db8-1d2b-4d35-b025-c54111d5bfd2/volumes"
Oct 30 13:06:13.090107 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.090082    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d750941a-ed7c-4a24-bfd1-de926f1b09e0" path="/var/lib/kubelet/pods/d750941a-ed7c-4a24-bfd1-de926f1b09e0/volumes"
Oct 30 13:06:13.247303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.247259    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh"]
Oct 30 13:06:13.249391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.249364    4544 cpu_manager.go:414] "containerMap: removing container" podUID="53b7be9d-ff23-452d-b121-12500f7d6787" containerName="payload-pre-processing"
Oct 30 13:06:13.249391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.249389    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53b7be9d-ff23-452d-b121-12500f7d6787" containerName="payload-pre-processing"
Oct 30 13:06:13.249565 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.249455    4544 cpu_manager.go:414] "containerMap: removing container" podUID="e4a68695-2f4b-4a06-938e-56e656c5c445" containerName="payload-processing"
Oct 30 13:06:13.249565 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.249463    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4a68695-2f4b-4a06-938e-56e656c5c445" containerName="payload-processing"
Oct 30 13:06:13.249565 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.249518    4544 cpu_manager.go:414] "containerMap: removing container" podUID="78230db8-1d2b-4d35-b025-c54111d5bfd2" containerName="istio-proxy"
Oct 30 13:06:13.249565 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.249526    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78230db8-1d2b-4d35-b025-c54111d5bfd2" containerName="istio-proxy"
Oct 30 13:06:13.249565 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.249538    4544 cpu_manager.go:414] "containerMap: removing container" podUID="78230db8-1d2b-4d35-b025-c54111d5bfd2" containerName="istio-proxy"
Oct 30 13:06:13.249565 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.249551    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78230db8-1d2b-4d35-b025-c54111d5bfd2" containerName="istio-proxy"
Oct 30 13:06:13.249916 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.249899    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="78230db8-1d2b-4d35-b025-c54111d5bfd2" containerName="istio-proxy"
Oct 30 13:06:13.249951 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.249918    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="78230db8-1d2b-4d35-b025-c54111d5bfd2" containerName="istio-proxy"
Oct 30 13:06:13.249986 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.249954    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="53b7be9d-ff23-452d-b121-12500f7d6787" containerName="payload-pre-processing"
Oct 30 13:06:13.249986 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.249967    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4a68695-2f4b-4a06-938e-56e656c5c445" containerName="payload-processing"
Oct 30 13:06:13.257556 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.257526    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh"
Oct 30 13:06:13.268392 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.268360    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:13.284879 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.284839    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-5af2de67\""
Oct 30 13:06:13.285171 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.285023    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-5af2de67-dockercfg-wjqsz\""
Oct 30 13:06:13.287874 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.287842    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh"]
Oct 30 13:06:13.347049 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.346941    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6"]
Oct 30 13:06:13.426426 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.395381    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnnjz\" (UniqueName: \"kubernetes.io/projected/5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce-kube-api-access-fnnjz\") pod \"payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh\" (UID: \"5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh"
Oct 30 13:06:13.426426 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.395579    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh\" (UID: \"5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh"
Oct 30 13:06:13.426426 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.410414    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6"
Oct 30 13:06:13.426426 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.422444    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6"]
Oct 30 13:06:13.535661 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.535619    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8qsf\" (UniqueName: \"kubernetes.io/projected/2da765c6-1b1b-47f4-856b-49166e411857-kube-api-access-f8qsf\") pod \"payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6\" (UID: \"2da765c6-1b1b-47f4-856b-49166e411857\") " pod="openshift-ingress/payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6"
Oct 30 13:06:13.535855 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.535788    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh\" (UID: \"5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh"
Oct 30 13:06:13.536505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.535914    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da765c6-1b1b-47f4-856b-49166e411857-plugins-config-volume\") pod \"payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6\" (UID: \"2da765c6-1b1b-47f4-856b-49166e411857\") " pod="openshift-ingress/payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6"
Oct 30 13:06:13.536505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.536027    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnnjz\" (UniqueName: \"kubernetes.io/projected/5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce-kube-api-access-fnnjz\") pod \"payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh\" (UID: \"5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh"
Oct 30 13:06:13.569265 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.564541    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh\" (UID: \"5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh"
Oct 30 13:06:13.656275 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.654927    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8qsf\" (UniqueName: \"kubernetes.io/projected/2da765c6-1b1b-47f4-856b-49166e411857-kube-api-access-f8qsf\") pod \"payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6\" (UID: \"2da765c6-1b1b-47f4-856b-49166e411857\") " pod="openshift-ingress/payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6"
Oct 30 13:06:13.656275 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.655294    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da765c6-1b1b-47f4-856b-49166e411857-plugins-config-volume\") pod \"payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6\" (UID: \"2da765c6-1b1b-47f4-856b-49166e411857\") " pod="openshift-ingress/payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6"
Oct 30 13:06:13.658628 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.656583    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da765c6-1b1b-47f4-856b-49166e411857-plugins-config-volume\") pod \"payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6\" (UID: \"2da765c6-1b1b-47f4-856b-49166e411857\") " pod="openshift-ingress/payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6"
Oct 30 13:06:13.718235 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.718185    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnnjz\" (UniqueName: \"kubernetes.io/projected/5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce-kube-api-access-fnnjz\") pod \"payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh\" (UID: \"5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh"
Oct 30 13:06:13.733214 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.733146    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8qsf\" (UniqueName: \"kubernetes.io/projected/2da765c6-1b1b-47f4-856b-49166e411857-kube-api-access-f8qsf\") pod \"payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6\" (UID: \"2da765c6-1b1b-47f4-856b-49166e411857\") " pod="openshift-ingress/payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6"
Oct 30 13:06:13.873465 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.873426    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh"
Oct 30 13:06:13.885553 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.885519    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6"
Oct 30 13:06:13.938999 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:13.938967    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:06:14.061289 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.061232    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d0c65ab-624c-40ab-9529-327c79d6567a-istio-token\" (UniqueName: \"kubernetes.io/projected/5d0c65ab-624c-40ab-9529-327c79d6567a-istio-token\") pod \"5d0c65ab-624c-40ab-9529-327c79d6567a\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") "
Oct 30 13:06:14.061504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.061388    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-workload-socket\") pod \"5d0c65ab-624c-40ab-9529-327c79d6567a\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") "
Oct 30 13:06:14.061504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.061472    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-workload-certs\") pod \"5d0c65ab-624c-40ab-9529-327c79d6567a\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") "
Oct 30 13:06:14.061616 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.061521    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-credential-socket\") pod \"5d0c65ab-624c-40ab-9529-327c79d6567a\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") "
Oct 30 13:06:14.061616 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.061589    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-istio-envoy\") pod \"5d0c65ab-624c-40ab-9529-327c79d6567a\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") "
Oct 30 13:06:14.061714 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.061644    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d0c65ab-624c-40ab-9529-327c79d6567a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d0c65ab-624c-40ab-9529-327c79d6567a-istiod-ca-cert\") pod \"5d0c65ab-624c-40ab-9529-327c79d6567a\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") "
Oct 30 13:06:14.061785 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.061753    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-workload-socket" pod "5d0c65ab-624c-40ab-9529-327c79d6567a" (UID: "5d0c65ab-624c-40ab-9529-327c79d6567a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:14.061785 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.061776    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d0c65ab-624c-40ab-9529-327c79d6567a-kube-api-access-ldqzn\" (UniqueName: \"kubernetes.io/projected/5d0c65ab-624c-40ab-9529-327c79d6567a-kube-api-access-ldqzn\") pod \"5d0c65ab-624c-40ab-9529-327c79d6567a\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") "
Oct 30 13:06:14.061904 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.061839    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-workload-certs" pod "5d0c65ab-624c-40ab-9529-327c79d6567a" (UID: "5d0c65ab-624c-40ab-9529-327c79d6567a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:14.061904 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.061876    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-credential-socket" pod "5d0c65ab-624c-40ab-9529-327c79d6567a" (UID: "5d0c65ab-624c-40ab-9529-327c79d6567a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:14.062017 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.061949    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-istio-data\") pod \"5d0c65ab-624c-40ab-9529-327c79d6567a\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") "
Oct 30 13:06:14.062017 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.061988    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d0c65ab-624c-40ab-9529-327c79d6567a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d0c65ab-624c-40ab-9529-327c79d6567a-istio-podinfo\") pod \"5d0c65ab-624c-40ab-9529-327c79d6567a\" (UID: \"5d0c65ab-624c-40ab-9529-327c79d6567a\") "
Oct 30 13:06:14.062442 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.062248    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d0c65ab-624c-40ab-9529-327c79d6567a-istiod-ca-cert" pod "5d0c65ab-624c-40ab-9529-327c79d6567a" (UID: "5d0c65ab-624c-40ab-9529-327c79d6567a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:14.062442 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.062406    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-istio-data" pod "5d0c65ab-624c-40ab-9529-327c79d6567a" (UID: "5d0c65ab-624c-40ab-9529-327c79d6567a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:14.062849 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.062827    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:06:14.062962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.062854    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d0c65ab-624c-40ab-9529-327c79d6567a-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:06:14.062962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.062869    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:06:14.062962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.062882    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:06:14.062962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.062896    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:06:14.064382 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.064353    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-istio-envoy" pod "5d0c65ab-624c-40ab-9529-327c79d6567a" (UID: "5d0c65ab-624c-40ab-9529-327c79d6567a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:14.064811 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.064787    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d0c65ab-624c-40ab-9529-327c79d6567a-kube-api-access-ldqzn" pod "5d0c65ab-624c-40ab-9529-327c79d6567a" (UID: "5d0c65ab-624c-40ab-9529-327c79d6567a"). InnerVolumeSpecName "kube-api-access-ldqzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:14.064975 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.064953    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d0c65ab-624c-40ab-9529-327c79d6567a-istio-podinfo" pod "5d0c65ab-624c-40ab-9529-327c79d6567a" (UID: "5d0c65ab-624c-40ab-9529-327c79d6567a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:06:14.065354 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.065326    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d0c65ab-624c-40ab-9529-327c79d6567a-istio-token" pod "5d0c65ab-624c-40ab-9529-327c79d6567a" (UID: "5d0c65ab-624c-40ab-9529-327c79d6567a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:14.164365 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.164281    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d0c65ab-624c-40ab-9529-327c79d6567a-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:06:14.164365 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.164316    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldqzn\" (UniqueName: \"kubernetes.io/projected/5d0c65ab-624c-40ab-9529-327c79d6567a-kube-api-access-ldqzn\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:06:14.164365 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.164329    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d0c65ab-624c-40ab-9529-327c79d6567a-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:06:14.164365 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.164337    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d0c65ab-624c-40ab-9529-327c79d6567a-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:06:14.200720 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.200691    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh"]
Oct 30 13:06:14.201311 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:06:14.201286    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ccbdf68_0a6e_44a7_8acf_1cb0632c78ce.slice/crio-192e4da96f2f3fa16c71ceaf3f1043ca6d8fbbdcda58915c2f65f4f2612e7a59 WatchSource:0}: Error finding container 192e4da96f2f3fa16c71ceaf3f1043ca6d8fbbdcda58915c2f65f4f2612e7a59: Status 404 returned error can't find the container with id 192e4da96f2f3fa16c71ceaf3f1043ca6d8fbbdcda58915c2f65f4f2612e7a59
Oct 30 13:06:14.216697 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.216665    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6"]
Oct 30 13:06:14.219846 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:06:14.219814    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da765c6_1b1b_47f4_856b_49166e411857.slice/crio-61c89b695d1601276f399bf1803c55ce0cfb9f23ab7b336ff17397e56908c530 WatchSource:0}: Error finding container 61c89b695d1601276f399bf1803c55ce0cfb9f23ab7b336ff17397e56908c530: Status 404 returned error can't find the container with id 61c89b695d1601276f399bf1803c55ce0cfb9f23ab7b336ff17397e56908c530
Oct 30 13:06:14.568092 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.567984    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6" event={"ID":"2da765c6-1b1b-47f4-856b-49166e411857","Type":"ContainerStarted","Data":"464f3ce375a2a81f1ca37d4917031a2862ed1ce04db6ba070ccff26e45e8238e"}
Oct 30 13:06:14.568092 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.568057    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6" event={"ID":"2da765c6-1b1b-47f4-856b-49166e411857","Type":"ContainerStarted","Data":"61c89b695d1601276f399bf1803c55ce0cfb9f23ab7b336ff17397e56908c530"}
Oct 30 13:06:14.568092 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.568092    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6"
Oct 30 13:06:14.570570 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.570538    4544 generic.go:356] "Generic (PLEG): container finished" podID="5d0c65ab-624c-40ab-9529-327c79d6567a" containerID="c1d1c97d72b74201dadc51e3c35547c2dba1c9752d4a4f674d097fe8c44c5aea" exitCode=0
Oct 30 13:06:14.570716 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.570578    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"
Oct 30 13:06:14.570716 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.570573    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz" event={"ID":"5d0c65ab-624c-40ab-9529-327c79d6567a","Type":"ContainerDied","Data":"c1d1c97d72b74201dadc51e3c35547c2dba1c9752d4a4f674d097fe8c44c5aea"}
Oct 30 13:06:14.570838 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.570748    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz" event={"ID":"5d0c65ab-624c-40ab-9529-327c79d6567a","Type":"ContainerDied","Data":"c588b67a810e4152c54e735bf4816f5f5b6227682d077584cc631cc23e2cdd42"}
Oct 30 13:06:14.570838 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.570770    4544 scope.go:122] "RemoveContainer" containerID="c1d1c97d72b74201dadc51e3c35547c2dba1c9752d4a4f674d097fe8c44c5aea"
Oct 30 13:06:14.572804 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.572780    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh" event={"ID":"5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce","Type":"ContainerStarted","Data":"31d9923c6ab6a8f631dd89a473813a0bd606f0903adc6b88f8059c0e9c9edb25"}
Oct 30 13:06:14.572931 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.572820    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh" event={"ID":"5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce","Type":"ContainerStarted","Data":"192e4da96f2f3fa16c71ceaf3f1043ca6d8fbbdcda58915c2f65f4f2612e7a59"}
Oct 30 13:06:14.572931 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.572855    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh"
Oct 30 13:06:14.582859 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.582838    4544 scope.go:122] "RemoveContainer" containerID="a099f064cb205f4a7ec204b4cda78f73f91353fdcf3caaea5bb9e26fd02db3ab"
Oct 30 13:06:14.593850 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.593821    4544 scope.go:122] "RemoveContainer" containerID="c1d1c97d72b74201dadc51e3c35547c2dba1c9752d4a4f674d097fe8c44c5aea"
Oct 30 13:06:14.594172 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:06:14.594149    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1d1c97d72b74201dadc51e3c35547c2dba1c9752d4a4f674d097fe8c44c5aea\": container with ID starting with c1d1c97d72b74201dadc51e3c35547c2dba1c9752d4a4f674d097fe8c44c5aea not found: ID does not exist" containerID="c1d1c97d72b74201dadc51e3c35547c2dba1c9752d4a4f674d097fe8c44c5aea"
Oct 30 13:06:14.594303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.594179    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1d1c97d72b74201dadc51e3c35547c2dba1c9752d4a4f674d097fe8c44c5aea"} err="failed to get container status \"c1d1c97d72b74201dadc51e3c35547c2dba1c9752d4a4f674d097fe8c44c5aea\": rpc error: code = NotFound desc = could not find container \"c1d1c97d72b74201dadc51e3c35547c2dba1c9752d4a4f674d097fe8c44c5aea\": container with ID starting with c1d1c97d72b74201dadc51e3c35547c2dba1c9752d4a4f674d097fe8c44c5aea not found: ID does not exist"
Oct 30 13:06:14.594303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.594197    4544 scope.go:122] "RemoveContainer" containerID="a099f064cb205f4a7ec204b4cda78f73f91353fdcf3caaea5bb9e26fd02db3ab"
Oct 30 13:06:14.594533 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:06:14.594512    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a099f064cb205f4a7ec204b4cda78f73f91353fdcf3caaea5bb9e26fd02db3ab\": container with ID starting with a099f064cb205f4a7ec204b4cda78f73f91353fdcf3caaea5bb9e26fd02db3ab not found: ID does not exist" containerID="a099f064cb205f4a7ec204b4cda78f73f91353fdcf3caaea5bb9e26fd02db3ab"
Oct 30 13:06:14.594586 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.594544    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a099f064cb205f4a7ec204b4cda78f73f91353fdcf3caaea5bb9e26fd02db3ab"} err="failed to get container status \"a099f064cb205f4a7ec204b4cda78f73f91353fdcf3caaea5bb9e26fd02db3ab\": rpc error: code = NotFound desc = could not find container \"a099f064cb205f4a7ec204b4cda78f73f91353fdcf3caaea5bb9e26fd02db3ab\": container with ID starting with a099f064cb205f4a7ec204b4cda78f73f91353fdcf3caaea5bb9e26fd02db3ab not found: ID does not exist"
Oct 30 13:06:14.625188 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.625135    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6" podStartSLOduration=1.625117381 podStartE2EDuration="1.625117381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:14.620294252 +0000 UTC m=+1269.667428205" watchObservedRunningTime="2026-10-30 13:06:14.625117381 +0000 UTC m=+1269.672251367"
Oct 30 13:06:14.704539 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.704481    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh" podStartSLOduration=1.7044644629999999 podStartE2EDuration="1.704464463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:14.70202238 +0000 UTC m=+1269.749156333" watchObservedRunningTime="2026-10-30 13:06:14.704464463 +0000 UTC m=+1269.751598420"
Oct 30 13:06:14.752299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.752236    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"]
Oct 30 13:06:14.760551 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:14.760514    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz"]
Oct 30 13:06:15.090145 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:15.090109    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d0c65ab-624c-40ab-9529-327c79d6567a" path="/var/lib/kubelet/pods/5d0c65ab-624c-40ab-9529-327c79d6567a/volumes"
Oct 30 13:06:15.121430 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:15.121398    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:16.947422 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:16.947373    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:18.030410 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:18.030375    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ec1adf9-zql4m"]
Oct 30 13:06:18.041416 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:18.041389    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ec1adf9-zql4m"]
Oct 30 13:06:18.745903 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:18.745869    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:19.096092 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:19.096049    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f370f314-d33a-4ae4-b2b2-498533ee66b4" path="/var/lib/kubelet/pods/f370f314-d33a-4ae4-b2b2-498533ee66b4/volumes"
Oct 30 13:06:20.695390 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:20.695353    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:22.454542 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:22.454503    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:24.233899 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:24.233863    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:25.640576 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:25.638834    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6"
Oct 30 13:06:25.642307 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:25.641355    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh"
Oct 30 13:06:25.973895 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:25.973795    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:26.802860 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:26.802818    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5af2de67-684b7684df-d8j6j"]
Oct 30 13:06:26.804726 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:26.804693    4544 cpu_manager.go:414] "containerMap: removing container" podUID="5d0c65ab-624c-40ab-9529-327c79d6567a" containerName="istio-proxy"
Oct 30 13:06:26.804892 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:26.804878    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d0c65ab-624c-40ab-9529-327c79d6567a" containerName="istio-proxy"
Oct 30 13:06:26.805178 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:26.805161    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d0c65ab-624c-40ab-9529-327c79d6567a" containerName="istio-proxy"
Oct 30 13:06:26.805331 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:26.805318    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d0c65ab-624c-40ab-9529-327c79d6567a" containerName="istio-proxy"
Oct 30 13:06:26.810170 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:26.810139    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5af2de67-684b7684df-d8j6j"
Oct 30 13:06:26.822521 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:26.822477    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5af2de67-684b7684df-d8j6j"]
Oct 30 13:06:26.830516 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:26.827396    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-5af2de67-serving-cert\""
Oct 30 13:06:26.948344 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:26.947835    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:27.060264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:27.060127    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/456ae1e2-ef5b-427c-a4a3-fba864a91b55-maas-api-tls\") pod \"maas-api-e2e-derive-5af2de67-684b7684df-d8j6j\" (UID: \"456ae1e2-ef5b-427c-a4a3-fba864a91b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5af2de67-684b7684df-d8j6j"
Oct 30 13:06:27.060504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:27.060479    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/456ae1e2-ef5b-427c-a4a3-fba864a91b55-openshift-service-ca\") pod \"maas-api-e2e-derive-5af2de67-684b7684df-d8j6j\" (UID: \"456ae1e2-ef5b-427c-a4a3-fba864a91b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5af2de67-684b7684df-d8j6j"
Oct 30 13:06:27.060713 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:27.060699    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwvd2\" (UniqueName: \"kubernetes.io/projected/456ae1e2-ef5b-427c-a4a3-fba864a91b55-kube-api-access-nwvd2\") pod \"maas-api-e2e-derive-5af2de67-684b7684df-d8j6j\" (UID: \"456ae1e2-ef5b-427c-a4a3-fba864a91b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5af2de67-684b7684df-d8j6j"
Oct 30 13:06:27.060931 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:27.060914    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/456ae1e2-ef5b-427c-a4a3-fba864a91b55-metrics-certs\") pod \"maas-api-e2e-derive-5af2de67-684b7684df-d8j6j\" (UID: \"456ae1e2-ef5b-427c-a4a3-fba864a91b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5af2de67-684b7684df-d8j6j"
Oct 30 13:06:27.063100 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:27.063073    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:27.177246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:27.173348    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/456ae1e2-ef5b-427c-a4a3-fba864a91b55-metrics-certs\") pod \"maas-api-e2e-derive-5af2de67-684b7684df-d8j6j\" (UID: \"456ae1e2-ef5b-427c-a4a3-fba864a91b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5af2de67-684b7684df-d8j6j"
Oct 30 13:06:27.177246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:27.173563    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/456ae1e2-ef5b-427c-a4a3-fba864a91b55-maas-api-tls\") pod \"maas-api-e2e-derive-5af2de67-684b7684df-d8j6j\" (UID: \"456ae1e2-ef5b-427c-a4a3-fba864a91b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5af2de67-684b7684df-d8j6j"
Oct 30 13:06:27.177246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:27.173590    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/456ae1e2-ef5b-427c-a4a3-fba864a91b55-openshift-service-ca\") pod \"maas-api-e2e-derive-5af2de67-684b7684df-d8j6j\" (UID: \"456ae1e2-ef5b-427c-a4a3-fba864a91b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5af2de67-684b7684df-d8j6j"
Oct 30 13:06:27.177246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:27.173724    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwvd2\" (UniqueName: \"kubernetes.io/projected/456ae1e2-ef5b-427c-a4a3-fba864a91b55-kube-api-access-nwvd2\") pod \"maas-api-e2e-derive-5af2de67-684b7684df-d8j6j\" (UID: \"456ae1e2-ef5b-427c-a4a3-fba864a91b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5af2de67-684b7684df-d8j6j"
Oct 30 13:06:27.211579 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:27.211539    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/456ae1e2-ef5b-427c-a4a3-fba864a91b55-openshift-service-ca\") pod \"maas-api-e2e-derive-5af2de67-684b7684df-d8j6j\" (UID: \"456ae1e2-ef5b-427c-a4a3-fba864a91b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5af2de67-684b7684df-d8j6j"
Oct 30 13:06:27.220936 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:27.220889    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:27.268292 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:27.256799    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/456ae1e2-ef5b-427c-a4a3-fba864a91b55-maas-api-tls\") pod \"maas-api-e2e-derive-5af2de67-684b7684df-d8j6j\" (UID: \"456ae1e2-ef5b-427c-a4a3-fba864a91b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5af2de67-684b7684df-d8j6j"
Oct 30 13:06:27.268292 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:27.267035    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/456ae1e2-ef5b-427c-a4a3-fba864a91b55-metrics-certs\") pod \"maas-api-e2e-derive-5af2de67-684b7684df-d8j6j\" (UID: \"456ae1e2-ef5b-427c-a4a3-fba864a91b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5af2de67-684b7684df-d8j6j"
Oct 30 13:06:27.268292 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:27.267752    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwvd2\" (UniqueName: \"kubernetes.io/projected/456ae1e2-ef5b-427c-a4a3-fba864a91b55-kube-api-access-nwvd2\") pod \"maas-api-e2e-derive-5af2de67-684b7684df-d8j6j\" (UID: \"456ae1e2-ef5b-427c-a4a3-fba864a91b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5af2de67-684b7684df-d8j6j"
Oct 30 13:06:27.557881 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:27.557523    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5af2de67-684b7684df-d8j6j"
Oct 30 13:06:28.046252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:28.044819    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55463289-2mj2g"]
Oct 30 13:06:28.055801 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:28.055765    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55463289-2mj2g"]
Oct 30 13:06:28.190347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:28.190308    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5af2de67-684b7684df-d8j6j"]
Oct 30 13:06:28.194600 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:06:28.194553    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456ae1e2_ef5b_427c_a4a3_fba864a91b55.slice/crio-0468b301d6c735a12783a76eec71bf18a460bd53494e5bd816f148c854e9021a WatchSource:0}: Error finding container 0468b301d6c735a12783a76eec71bf18a460bd53494e5bd816f148c854e9021a: Status 404 returned error can't find the container with id 0468b301d6c735a12783a76eec71bf18a460bd53494e5bd816f148c854e9021a
Oct 30 13:06:28.771331 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:28.771278    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5af2de67-684b7684df-d8j6j" event={"ID":"456ae1e2-ef5b-427c-a4a3-fba864a91b55","Type":"ContainerStarted","Data":"ad709b24144d870fcafc47d97dd53fb0d9f0d84a71d7824bcefb62f47a8c89e4"}
Oct 30 13:06:28.771556 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:28.771348    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5af2de67-684b7684df-d8j6j" event={"ID":"456ae1e2-ef5b-427c-a4a3-fba864a91b55","Type":"ContainerStarted","Data":"0468b301d6c735a12783a76eec71bf18a460bd53494e5bd816f148c854e9021a"}
Oct 30 13:06:28.771556 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:28.771504    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5af2de67-684b7684df-d8j6j"
Oct 30 13:06:28.819283 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:28.819158    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5af2de67-684b7684df-d8j6j" podStartSLOduration=2.458510273 podStartE2EDuration="2.819135672s" totalImagesPullingTime="360.625399ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:28.814178619 +0000 UTC m=+1283.861312577" watchObservedRunningTime="2026-10-30 13:06:28.819135672 +0000 UTC m=+1283.866269629"
Oct 30 13:06:29.086529 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:06:29.086490    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:06:29.096521 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:29.096485    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d1d21c0-897a-4e80-9fd2-b774bda9b260" path="/var/lib/kubelet/pods/3d1d21c0-897a-4e80-9fd2-b774bda9b260/volumes"
Oct 30 13:06:29.098351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:29.098322    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:30.845400 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:30.845360    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:32.666003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:32.665967    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:32.920886 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:32.920770    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:34.405148 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:34.405099    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:34.796479 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:34.796399    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5af2de67-684b7684df-d8j6j"
Oct 30 13:06:36.290074 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:36.290038    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:36.423619 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:36.423559    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:38.249260 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:38.249202    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:40.358145 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:40.358070    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:41.053074 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:41.053041    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:41.091166 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:41.091130    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4521c558-fjdcg"]
Oct 30 13:06:41.091166 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:41.091175    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4521c558-fjdcg"]
Oct 30 13:06:41.305700 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:41.305609    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:41.461992 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:41.461950    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:41.811291 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:41.811256    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"]
Oct 30 13:06:41.812032 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:41.812016    4544 cpu_manager.go:414] "containerMap: removing container" podUID="5d0c65ab-624c-40ab-9529-327c79d6567a" containerName="istio-proxy"
Oct 30 13:06:41.812085 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:41.812033    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d0c65ab-624c-40ab-9529-327c79d6567a" containerName="istio-proxy"
Oct 30 13:06:41.856504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:41.856469    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"]
Oct 30 13:06:41.856741 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:41.856502    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:06:41.873233 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:41.873190    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:41.874833 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:41.874808    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-85be4bb9-gw-openshift-default-dockercfg-g5b77\""
Oct 30 13:06:42.132089 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:42.131867    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-workload-certs\") pod \"e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") " pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:06:42.134433 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:42.134399    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istiod-ca-cert\") pod \"e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") " pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:06:42.279855 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:42.279815    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-workload-socket\") pod \"e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") " pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:06:42.280135 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:42.280115    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istio-podinfo\") pod \"e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") " pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:06:42.280400 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:42.280383    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-credential-socket\") pod \"e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") " pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:06:42.280644 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:42.280623    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istio-envoy\") pod \"e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") " pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:06:42.280876 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:42.280858    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istio-token\") pod \"e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") " pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:06:42.284324 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:42.281102    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjtrv\" (UniqueName: \"kubernetes.io/projected/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-kube-api-access-kjtrv\") pod \"e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") " pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:06:42.284553 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:42.284532    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istio-data\") pod \"e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") " pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:06:42.344183 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:42.344142    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:42.394744 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:42.394630    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-credential-socket\") pod \"e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") " pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:06:42.407245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:42.402492    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istio-envoy\") pod \"e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") " pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:06:42.407245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:42.402676    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istio-token\") pod \"e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") " pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:06:42.407245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:42.402817    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjtrv\" (UniqueName: \"kubernetes.io/projected/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-kube-api-access-kjtrv\") pod \"e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") " pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:06:42.407245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:42.402857    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istio-data\") pod \"e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") " pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:06:42.407245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:42.402983    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-workload-certs\") pod \"e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") " pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:06:42.407245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:42.403068    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istiod-ca-cert\") pod \"e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") " pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:06:42.407245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:42.403112    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-workload-socket\") pod \"e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") " pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:06:42.407245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:42.403153    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istio-podinfo\") pod \"e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") " pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:06:42.425246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:42.398861    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-credential-socket\") pod \"e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") " pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:06:42.435337 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:42.422710    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-workload-certs\") pod \"e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") " pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:06:42.439243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:42.422992    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istio-data\") pod \"e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") " pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:06:42.445340 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:42.424014    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istiod-ca-cert\") pod \"e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") " pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:06:42.456538 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:42.456444    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-workload-socket\") pod \"e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") " pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:06:42.484331 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:42.483986    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjtrv\" (UniqueName: \"kubernetes.io/projected/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-kube-api-access-kjtrv\") pod \"e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") " pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:06:42.484331 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:42.484274    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istio-envoy\") pod \"e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") " pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:06:42.502937 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:42.502899    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istio-podinfo\") pod \"e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") " pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:06:42.518244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:42.505697    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istio-token\") pod \"e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") " pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:06:42.782194 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:42.782101    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:06:43.044438 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:43.044345    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:43.244089 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:43.243176    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:06:43.244089 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:43.243305    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:06:43.244089 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:43.243357    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:06:43.256207 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:43.256154    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4e25784-2cf6-432d-8924-e855d1c88155" path="/var/lib/kubelet/pods/e4e25784-2cf6-432d-8924-e855d1c88155/volumes"
Oct 30 13:06:43.267827 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:43.267784    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"]
Oct 30 13:06:43.303973 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:43.303942    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx" event={"ID":"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f","Type":"ContainerStarted","Data":"ad585926ec6fc89caa9701b197185717178ec91caa1149771c9b02ac3cc8d4f6"}
Oct 30 13:06:44.339710 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:44.339674    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx" event={"ID":"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f","Type":"ContainerStarted","Data":"11cf2aa8460bc44631b9a50a36c260adcfafc7702591893ffccd29d607a766d6"}
Oct 30 13:06:44.783294 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:44.783176    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:06:44.789668 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:44.789634    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:06:44.837868 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:44.837798    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx" podStartSLOduration=3.837778208 podStartE2EDuration="3.837778208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:44.409293178 +0000 UTC m=+1299.456427131" watchObservedRunningTime="2026-10-30 13:06:44.837778208 +0000 UTC m=+1299.884912162"
Oct 30 13:06:45.047631 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:45.047524    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:45.349365 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:45.349337    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:06:45.350429 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:45.350408    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:06:46.301113 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:46.301071    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:46.438946 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:46.438911    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:47.283411 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:47.282786    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:47.466847 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:47.466813    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:47.503121 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:47.503086    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx"]
Oct 30 13:06:47.509262 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:47.509216    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx"
Oct 30 13:06:47.530492 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:47.530458    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt"]
Oct 30 13:06:47.536438 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:47.536363    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt"
Oct 30 13:06:47.536994 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:47.536966    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx"]
Oct 30 13:06:47.558261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:47.558170    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-85be4bb9-dockercfg-jl7ml\""
Oct 30 13:06:47.559053 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:47.559015    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-85be4bb9\""
Oct 30 13:06:47.574908 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:47.574868    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt"]
Oct 30 13:06:47.584162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:47.584126    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqzh7\" (UniqueName: \"kubernetes.io/projected/cadd0f75-a378-43fa-8bdd-3182071fa2c1-kube-api-access-wqzh7\") pod \"payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx\" (UID: \"cadd0f75-a378-43fa-8bdd-3182071fa2c1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx"
Oct 30 13:06:47.584540 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:47.584513    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cadd0f75-a378-43fa-8bdd-3182071fa2c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx\" (UID: \"cadd0f75-a378-43fa-8bdd-3182071fa2c1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx"
Oct 30 13:06:47.584819 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:47.584797    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7tnh\" (UniqueName: \"kubernetes.io/projected/13c0f5bf-8735-4702-8d62-86810366b1f9-kube-api-access-l7tnh\") pod \"payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt\" (UID: \"13c0f5bf-8735-4702-8d62-86810366b1f9\") " pod="openshift-ingress/payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt"
Oct 30 13:06:47.585338 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:47.585303    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13c0f5bf-8735-4702-8d62-86810366b1f9-plugins-config-volume\") pod \"payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt\" (UID: \"13c0f5bf-8735-4702-8d62-86810366b1f9\") " pod="openshift-ingress/payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt"
Oct 30 13:06:47.618015 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:47.617979    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:47.799301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:47.718538    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13c0f5bf-8735-4702-8d62-86810366b1f9-plugins-config-volume\") pod \"payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt\" (UID: \"13c0f5bf-8735-4702-8d62-86810366b1f9\") " pod="openshift-ingress/payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt"
Oct 30 13:06:47.799301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:47.719151    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13c0f5bf-8735-4702-8d62-86810366b1f9-plugins-config-volume\") pod \"payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt\" (UID: \"13c0f5bf-8735-4702-8d62-86810366b1f9\") " pod="openshift-ingress/payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt"
Oct 30 13:06:47.799301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:47.719501    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqzh7\" (UniqueName: \"kubernetes.io/projected/cadd0f75-a378-43fa-8bdd-3182071fa2c1-kube-api-access-wqzh7\") pod \"payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx\" (UID: \"cadd0f75-a378-43fa-8bdd-3182071fa2c1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx"
Oct 30 13:06:47.799301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:47.719538    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cadd0f75-a378-43fa-8bdd-3182071fa2c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx\" (UID: \"cadd0f75-a378-43fa-8bdd-3182071fa2c1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx"
Oct 30 13:06:47.799301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:47.719685    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7tnh\" (UniqueName: \"kubernetes.io/projected/13c0f5bf-8735-4702-8d62-86810366b1f9-kube-api-access-l7tnh\") pod \"payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt\" (UID: \"13c0f5bf-8735-4702-8d62-86810366b1f9\") " pod="openshift-ingress/payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt"
Oct 30 13:06:47.799301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:47.750799    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cadd0f75-a378-43fa-8bdd-3182071fa2c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx\" (UID: \"cadd0f75-a378-43fa-8bdd-3182071fa2c1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx"
Oct 30 13:06:47.799301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:47.772736    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqzh7\" (UniqueName: \"kubernetes.io/projected/cadd0f75-a378-43fa-8bdd-3182071fa2c1-kube-api-access-wqzh7\") pod \"payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx\" (UID: \"cadd0f75-a378-43fa-8bdd-3182071fa2c1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx"
Oct 30 13:06:47.834247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:47.813674    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7tnh\" (UniqueName: \"kubernetes.io/projected/13c0f5bf-8735-4702-8d62-86810366b1f9-kube-api-access-l7tnh\") pod \"payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt\" (UID: \"13c0f5bf-8735-4702-8d62-86810366b1f9\") " pod="openshift-ingress/payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt"
Oct 30 13:06:47.854455 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:47.854422    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx"
Oct 30 13:06:47.881889 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:47.881520    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt"
Oct 30 13:06:48.349554 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:48.349514    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx"]
Oct 30 13:06:48.360544 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:48.360499    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt"]
Oct 30 13:06:48.363862 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:06:48.363832    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13c0f5bf_8735_4702_8d62_86810366b1f9.slice/crio-0d9e36c82fb3ead8f467a340a78652aa1ef523111f71bd11b718698371ea57c8 WatchSource:0}: Error finding container 0d9e36c82fb3ead8f467a340a78652aa1ef523111f71bd11b718698371ea57c8: Status 404 returned error can't find the container with id 0d9e36c82fb3ead8f467a340a78652aa1ef523111f71bd11b718698371ea57c8
Oct 30 13:06:48.380002 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:48.379972    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt" event={"ID":"13c0f5bf-8735-4702-8d62-86810366b1f9","Type":"ContainerStarted","Data":"0d9e36c82fb3ead8f467a340a78652aa1ef523111f71bd11b718698371ea57c8"}
Oct 30 13:06:48.381593 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:48.381561    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx" event={"ID":"cadd0f75-a378-43fa-8bdd-3182071fa2c1","Type":"ContainerStarted","Data":"ebae8a1eb9c428fdd0f01f3d434e0b3f8541bda70d684a59d7bb6f2c64ac3afe"}
Oct 30 13:06:49.393252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:49.393193    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt" event={"ID":"13c0f5bf-8735-4702-8d62-86810366b1f9","Type":"ContainerStarted","Data":"321d8b8100146eca43e97a75d3834cebd0aa8fa663e26b35b880f5cd9760f966"}
Oct 30 13:06:49.393688 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:49.393346    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt"
Oct 30 13:06:49.395477 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:49.395445    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx" event={"ID":"cadd0f75-a378-43fa-8bdd-3182071fa2c1","Type":"ContainerStarted","Data":"dd0ea9f05ca907bbb41a9b2b44ae3b3c47dca66a6d4c3eae785801d1e36f0408"}
Oct 30 13:06:49.395615 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:49.395506    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx"
Oct 30 13:06:49.437474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:49.437413    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt" podStartSLOduration=2.437393809 podStartE2EDuration="2.437393809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:49.432963053 +0000 UTC m=+1304.480097007" watchObservedRunningTime="2026-10-30 13:06:49.437393809 +0000 UTC m=+1304.484527761"
Oct 30 13:06:49.481833 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:49.481772    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx" podStartSLOduration=2.481754116 podStartE2EDuration="2.481754116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:49.477623428 +0000 UTC m=+1304.524757381" watchObservedRunningTime="2026-10-30 13:06:49.481754116 +0000 UTC m=+1304.528888120"
Oct 30 13:06:49.815825 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:49.815736    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:49.971616 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:49.971586    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:51.417350 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:51.417207    4544 generic.go:356] "Generic (PLEG): container finished" podID="4b940dbd-f31b-4620-84e8-f32fc7feef3e" containerID="2b0a772caa410bddc4a83bdb14a6d6415ab43bf8b561824b8f14ba86d8f75adc" exitCode=0
Oct 30 13:06:51.417350 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:51.417286    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9" event={"ID":"4b940dbd-f31b-4620-84e8-f32fc7feef3e","Type":"ContainerDied","Data":"2b0a772caa410bddc4a83bdb14a6d6415ab43bf8b561824b8f14ba86d8f75adc"}
Oct 30 13:06:51.417826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:51.417786    4544 scope.go:122] "RemoveContainer" containerID="2b0a772caa410bddc4a83bdb14a6d6415ab43bf8b561824b8f14ba86d8f75adc"
Oct 30 13:06:51.419498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:51.419467    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:06:51.419598 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:51.419541    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:06:51.419598 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:51.419587    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:06:51.899982 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:51.899944    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:52.446974 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:52.446931    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9" event={"ID":"4b940dbd-f31b-4620-84e8-f32fc7feef3e","Type":"ContainerStarted","Data":"bcf3fa3d82eb1a4dd911c8939ad732a1c89f32dc4fa988403bd888aec96892dc"}
Oct 30 13:06:52.711077 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:52.710988    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:52.737249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:52.733394    4544 patch_prober.go:29] interesting pod/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 13:06:52.737249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:52.733472    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9" podUID="4b940dbd-f31b-4620-84e8-f32fc7feef3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 13:06:53.538725 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:53.538687    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:53.711781 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:53.711741    4544 patch_prober.go:29] interesting pod/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 13:06:53.711967 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:53.711814    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9" podUID="4b940dbd-f31b-4620-84e8-f32fc7feef3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 13:06:53.939184 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:53.939147    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:54.047541 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:54.047502    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:54.172155 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:54.172108    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:54.712049 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:54.712018    4544 patch_prober.go:29] interesting pod/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 13:06:54.712595 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:54.712081    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9" podUID="4b940dbd-f31b-4620-84e8-f32fc7feef3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 13:06:55.712009 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:55.711963    4544 patch_prober.go:29] interesting pod/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 13:06:55.712255 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:55.712079    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9" podUID="4b940dbd-f31b-4620-84e8-f32fc7feef3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 13:06:55.713390 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:55.713194    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:56.044859 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:56.044757    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fedd835-c2x48"]
Oct 30 13:06:56.055193 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:56.055152    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fedd835-c2x48"]
Oct 30 13:06:56.103675 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:56.103635    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:56.440554 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:56.440511    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:56.712088 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:56.711990    4544 patch_prober.go:29] interesting pod/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 13:06:56.712088 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:56.712053    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9" podUID="4b940dbd-f31b-4620-84e8-f32fc7feef3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 13:06:57.089291 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:57.089257    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28879ce9-05ee-4ba5-8cd2-98d514e651e1" path="/var/lib/kubelet/pods/28879ce9-05ee-4ba5-8cd2-98d514e651e1/volumes"
Oct 30 13:06:57.744255 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:57.743843    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:57.748238 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:57.747204    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:06:57.879952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:57.879883    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:59.010477 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:06:59.010400    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:06:59.710181 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:59.710124    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:06:59.825675 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:06:59.825598    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:00.409835 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:00.409801    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx"
Oct 30 13:07:00.410379 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:00.409885    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt"
Oct 30 13:07:01.502397 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:01.502348    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:02.246084 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:02.246045    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6"]
Oct 30 13:07:02.303414 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:02.303281    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh"]
Oct 30 13:07:02.369938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:02.369900    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:02.753257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:02.747948    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:03.787457 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:03.787394    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:07:04.003249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:03.978824    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:04.003822 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:04.000732    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb"]
Oct 30 13:07:04.032775 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:04.009384    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:07:04.153261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:04.123052    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:04.170672 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:04.134555    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb"
Oct 30 13:07:04.183445 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:04.183389    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb"]
Oct 30 13:07:04.205597 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:04.205559    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-85be4bb9-serving-cert\""
Oct 30 13:07:04.277789 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:04.274144    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:04.315311 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:04.307997    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4485c1ce-c862-400b-9bfa-79b1a745dcd9-maas-api-tls\") pod \"maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb\" (UID: \"4485c1ce-c862-400b-9bfa-79b1a745dcd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb"
Oct 30 13:07:04.315311 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:04.308049    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4485c1ce-c862-400b-9bfa-79b1a745dcd9-openshift-service-ca\") pod \"maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb\" (UID: \"4485c1ce-c862-400b-9bfa-79b1a745dcd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb"
Oct 30 13:07:04.315311 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:04.308307    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4485c1ce-c862-400b-9bfa-79b1a745dcd9-metrics-certs\") pod \"maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb\" (UID: \"4485c1ce-c862-400b-9bfa-79b1a745dcd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb"
Oct 30 13:07:04.315311 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:04.308840    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssw87\" (UniqueName: \"kubernetes.io/projected/4485c1ce-c862-400b-9bfa-79b1a745dcd9-kube-api-access-ssw87\") pod \"maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb\" (UID: \"4485c1ce-c862-400b-9bfa-79b1a745dcd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb"
Oct 30 13:07:04.434342 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:04.423882    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssw87\" (UniqueName: \"kubernetes.io/projected/4485c1ce-c862-400b-9bfa-79b1a745dcd9-kube-api-access-ssw87\") pod \"maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb\" (UID: \"4485c1ce-c862-400b-9bfa-79b1a745dcd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb"
Oct 30 13:07:04.434342 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:04.423988    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4485c1ce-c862-400b-9bfa-79b1a745dcd9-maas-api-tls\") pod \"maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb\" (UID: \"4485c1ce-c862-400b-9bfa-79b1a745dcd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb"
Oct 30 13:07:04.434342 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:04.424026    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4485c1ce-c862-400b-9bfa-79b1a745dcd9-openshift-service-ca\") pod \"maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb\" (UID: \"4485c1ce-c862-400b-9bfa-79b1a745dcd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb"
Oct 30 13:07:04.434342 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:04.424092    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4485c1ce-c862-400b-9bfa-79b1a745dcd9-metrics-certs\") pod \"maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb\" (UID: \"4485c1ce-c862-400b-9bfa-79b1a745dcd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb"
Oct 30 13:07:04.434342 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:04.427253    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4485c1ce-c862-400b-9bfa-79b1a745dcd9-openshift-service-ca\") pod \"maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb\" (UID: \"4485c1ce-c862-400b-9bfa-79b1a745dcd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb"
Oct 30 13:07:04.434342 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:04.427429    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7grvt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:07:04.434342 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:04.427479    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt" podUID="c16d5364-17da-4fef-993d-9ac8a77f56a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:07:04.434342 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:04.433135    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7mx6s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:07:04.434342 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:04.433194    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s" podUID="9c24a948-3812-4568-a7a0-e58641a7427f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:07:04.463943 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:04.463909    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4485c1ce-c862-400b-9bfa-79b1a745dcd9-metrics-certs\") pod \"maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb\" (UID: \"4485c1ce-c862-400b-9bfa-79b1a745dcd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb"
Oct 30 13:07:04.469240 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:04.465984    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssw87\" (UniqueName: \"kubernetes.io/projected/4485c1ce-c862-400b-9bfa-79b1a745dcd9-kube-api-access-ssw87\") pod \"maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb\" (UID: \"4485c1ce-c862-400b-9bfa-79b1a745dcd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb"
Oct 30 13:07:04.577252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:04.577188    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4485c1ce-c862-400b-9bfa-79b1a745dcd9-maas-api-tls\") pod \"maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb\" (UID: \"4485c1ce-c862-400b-9bfa-79b1a745dcd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb"
Oct 30 13:07:04.595848 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:04.595791    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb"
Oct 30 13:07:04.596840 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:04.596319    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-w27kj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:07:04.596840 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:04.596397    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj" podUID="287ea8cd-7fc1-4967-a291-adf6973f5967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:07:05.854659 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:05.854598    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5af2de67-684b7684df-d8j6j"]
Oct 30 13:07:05.855674 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:05.855597    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5af2de67-684b7684df-d8j6j" podUID="456ae1e2-ef5b-427c-a4a3-fba864a91b55" containerName="maas-api" containerID="cri-o://ad709b24144d870fcafc47d97dd53fb0d9f0d84a71d7824bcefb62f47a8c89e4" gracePeriod=30
Oct 30 13:07:05.858229 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:05.858174    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:07:05.920402 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:05.920353    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:07:05.936538 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:05.920665    4544 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:07:05.983255 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:05.983180    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:06.037545 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:06.037505    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb"]
Oct 30 13:07:06.693079 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:06.693039    4544 generic.go:356] "Generic (PLEG): container finished" podID="456ae1e2-ef5b-427c-a4a3-fba864a91b55" containerID="ad709b24144d870fcafc47d97dd53fb0d9f0d84a71d7824bcefb62f47a8c89e4" exitCode=0
Oct 30 13:07:06.693559 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:06.693524    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5af2de67-684b7684df-d8j6j" event={"ID":"456ae1e2-ef5b-427c-a4a3-fba864a91b55","Type":"ContainerDied","Data":"ad709b24144d870fcafc47d97dd53fb0d9f0d84a71d7824bcefb62f47a8c89e4"}
Oct 30 13:07:06.714677 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:06.714495    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb" event={"ID":"4485c1ce-c862-400b-9bfa-79b1a745dcd9","Type":"ContainerStarted","Data":"991c184649f9d57eff71690db53a6ad9c45a8de2f9439689ce93e1f46a1735ff"}
Oct 30 13:07:06.827352 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:06.827313    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:06.864303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:06.864264    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5af2de67-684b7684df-d8j6j"
Oct 30 13:07:06.883031 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:07:06.882986    4544 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456ae1e2_ef5b_427c_a4a3_fba864a91b55.slice/crio-ad709b24144d870fcafc47d97dd53fb0d9f0d84a71d7824bcefb62f47a8c89e4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456ae1e2_ef5b_427c_a4a3_fba864a91b55.slice/crio-conmon-ad709b24144d870fcafc47d97dd53fb0d9f0d84a71d7824bcefb62f47a8c89e4.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 13:07:06.967560 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:06.967475    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:06.988497 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:06.988450    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/456ae1e2-ef5b-427c-a4a3-fba864a91b55-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/456ae1e2-ef5b-427c-a4a3-fba864a91b55-maas-api-tls\") pod \"456ae1e2-ef5b-427c-a4a3-fba864a91b55\" (UID: \"456ae1e2-ef5b-427c-a4a3-fba864a91b55\") "
Oct 30 13:07:06.988695 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:06.988553    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/456ae1e2-ef5b-427c-a4a3-fba864a91b55-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/456ae1e2-ef5b-427c-a4a3-fba864a91b55-openshift-service-ca\") pod \"456ae1e2-ef5b-427c-a4a3-fba864a91b55\" (UID: \"456ae1e2-ef5b-427c-a4a3-fba864a91b55\") "
Oct 30 13:07:06.988748 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:06.988695    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/456ae1e2-ef5b-427c-a4a3-fba864a91b55-metrics-certs\" (UniqueName: \"kubernetes.io/secret/456ae1e2-ef5b-427c-a4a3-fba864a91b55-metrics-certs\") pod \"456ae1e2-ef5b-427c-a4a3-fba864a91b55\" (UID: \"456ae1e2-ef5b-427c-a4a3-fba864a91b55\") "
Oct 30 13:07:06.988896 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:06.988878    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/456ae1e2-ef5b-427c-a4a3-fba864a91b55-kube-api-access-nwvd2\" (UniqueName: \"kubernetes.io/projected/456ae1e2-ef5b-427c-a4a3-fba864a91b55-kube-api-access-nwvd2\") pod \"456ae1e2-ef5b-427c-a4a3-fba864a91b55\" (UID: \"456ae1e2-ef5b-427c-a4a3-fba864a91b55\") "
Oct 30 13:07:06.988986 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:06.988896    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/456ae1e2-ef5b-427c-a4a3-fba864a91b55-openshift-service-ca" pod "456ae1e2-ef5b-427c-a4a3-fba864a91b55" (UID: "456ae1e2-ef5b-427c-a4a3-fba864a91b55"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:07:06.989529 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:06.989506    4544 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/456ae1e2-ef5b-427c-a4a3-fba864a91b55-openshift-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:07:06.990774 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:06.990748    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/456ae1e2-ef5b-427c-a4a3-fba864a91b55-metrics-certs" pod "456ae1e2-ef5b-427c-a4a3-fba864a91b55" (UID: "456ae1e2-ef5b-427c-a4a3-fba864a91b55"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:07:06.990879 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:06.990808    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/456ae1e2-ef5b-427c-a4a3-fba864a91b55-maas-api-tls" pod "456ae1e2-ef5b-427c-a4a3-fba864a91b55" (UID: "456ae1e2-ef5b-427c-a4a3-fba864a91b55"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:07:06.991054 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:06.991026    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/456ae1e2-ef5b-427c-a4a3-fba864a91b55-kube-api-access-nwvd2" pod "456ae1e2-ef5b-427c-a4a3-fba864a91b55" (UID: "456ae1e2-ef5b-427c-a4a3-fba864a91b55"). InnerVolumeSpecName "kube-api-access-nwvd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:07.099612 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:07.092238    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwvd2\" (UniqueName: \"kubernetes.io/projected/456ae1e2-ef5b-427c-a4a3-fba864a91b55-kube-api-access-nwvd2\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:07:07.099612 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:07.092275    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/456ae1e2-ef5b-427c-a4a3-fba864a91b55-maas-api-tls\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:07:07.099612 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:07.092289    4544 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/456ae1e2-ef5b-427c-a4a3-fba864a91b55-metrics-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:07:07.139049 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:07.138892    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:07.250888 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:07.249373    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6" podUID="2da765c6-1b1b-47f4-856b-49166e411857" containerName="payload-processing" containerID="cri-o://464f3ce375a2a81f1ca37d4917031a2862ed1ce04db6ba070ccff26e45e8238e" gracePeriod=25
Oct 30 13:07:07.269847 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:07.252541    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"]
Oct 30 13:07:07.269847 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:07.252795    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9" podUID="4b940dbd-f31b-4620-84e8-f32fc7feef3e" containerName="istio-proxy" containerID="cri-o://bcf3fa3d82eb1a4dd911c8939ad732a1c89f32dc4fa988403bd888aec96892dc" gracePeriod=30
Oct 30 13:07:07.284508 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:07.284462    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:07.304216 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:07.304166    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh" podUID="5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce" containerName="payload-pre-processing" containerID="cri-o://31d9923c6ab6a8f631dd89a473813a0bd606f0903adc6b88f8059c0e9c9edb25" gracePeriod=25
Oct 30 13:07:07.733854 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:07.733809    4544 generic.go:356] "Generic (PLEG): container finished" podID="2da765c6-1b1b-47f4-856b-49166e411857" containerID="464f3ce375a2a81f1ca37d4917031a2862ed1ce04db6ba070ccff26e45e8238e" exitCode=0
Oct 30 13:07:07.734265 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:07.733884    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6" event={"ID":"2da765c6-1b1b-47f4-856b-49166e411857","Type":"ContainerDied","Data":"464f3ce375a2a81f1ca37d4917031a2862ed1ce04db6ba070ccff26e45e8238e"}
Oct 30 13:07:07.735979 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:07.735955    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5af2de67-684b7684df-d8j6j"
Oct 30 13:07:07.736828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:07.736011    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5af2de67-684b7684df-d8j6j" event={"ID":"456ae1e2-ef5b-427c-a4a3-fba864a91b55","Type":"ContainerDied","Data":"0468b301d6c735a12783a76eec71bf18a460bd53494e5bd816f148c854e9021a"}
Oct 30 13:07:07.736934 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:07.736878    4544 scope.go:122] "RemoveContainer" containerID="ad709b24144d870fcafc47d97dd53fb0d9f0d84a71d7824bcefb62f47a8c89e4"
Oct 30 13:07:07.738571 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:07.738547    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb" event={"ID":"4485c1ce-c862-400b-9bfa-79b1a745dcd9","Type":"ContainerStarted","Data":"ecc1e20d3a0e0747b43654feaa5d268aeb56570f8fef03e62b4db425319c8fe9"}
Oct 30 13:07:07.739557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:07.739540    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb"
Oct 30 13:07:07.741923 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:07.741897    4544 generic.go:356] "Generic (PLEG): container finished" podID="5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce" containerID="31d9923c6ab6a8f631dd89a473813a0bd606f0903adc6b88f8059c0e9c9edb25" exitCode=0
Oct 30 13:07:07.742084 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:07.742052    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh" event={"ID":"5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce","Type":"ContainerDied","Data":"31d9923c6ab6a8f631dd89a473813a0bd606f0903adc6b88f8059c0e9c9edb25"}
Oct 30 13:07:07.798647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:07.798613    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5af2de67-684b7684df-d8j6j"]
Oct 30 13:07:07.810006 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:07.809970    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5af2de67-684b7684df-d8j6j"]
Oct 30 13:07:07.844394 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:07.844338    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb" podStartSLOduration=4.293647333 podStartE2EDuration="4.84432194s" totalImagesPullingTime="550.674607ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:07.840431113 +0000 UTC m=+1322.887565069" watchObservedRunningTime="2026-10-30 13:07:07.84432194 +0000 UTC m=+1322.891455893"
Oct 30 13:07:07.870339 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:07.870310    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh"
Oct 30 13:07:07.904735 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:07.904704    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6"
Oct 30 13:07:07.978356 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:07.966398    4544 scope.go:122] "RemoveContainer" containerID="d1aaef764be4e317b873b1e5171ba6c5d048f1048ecb8e54f013dc601cb926e4"
Oct 30 13:07:07.982617 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:07.982586    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce-plugins-config-volume\") pod \"5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce\" (UID: \"5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce\") "
Oct 30 13:07:07.982961 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:07.982938    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2da765c6-1b1b-47f4-856b-49166e411857-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da765c6-1b1b-47f4-856b-49166e411857-plugins-config-volume\") pod \"2da765c6-1b1b-47f4-856b-49166e411857\" (UID: \"2da765c6-1b1b-47f4-856b-49166e411857\") "
Oct 30 13:07:07.983452 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:07.983413    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce-kube-api-access-fnnjz\" (UniqueName: \"kubernetes.io/projected/5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce-kube-api-access-fnnjz\") pod \"5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce\" (UID: \"5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce\") "
Oct 30 13:07:07.983675 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:07.983643    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2da765c6-1b1b-47f4-856b-49166e411857-kube-api-access-f8qsf\" (UniqueName: \"kubernetes.io/projected/2da765c6-1b1b-47f4-856b-49166e411857-kube-api-access-f8qsf\") pod \"2da765c6-1b1b-47f4-856b-49166e411857\" (UID: \"2da765c6-1b1b-47f4-856b-49166e411857\") "
Oct 30 13:07:07.989115 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:07.988550    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2da765c6-1b1b-47f4-856b-49166e411857-plugins-config-volume" pod "2da765c6-1b1b-47f4-856b-49166e411857" (UID: "2da765c6-1b1b-47f4-856b-49166e411857"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:07:07.991206 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:07.989515    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce-plugins-config-volume" pod "5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce" (UID: "5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:07:07.991206 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:07.990807    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da765c6-1b1b-47f4-856b-49166e411857-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:07:07.991206 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:07.990832    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:07:08.159463 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:08.159418    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce-kube-api-access-fnnjz" pod "5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce" (UID: "5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce"). InnerVolumeSpecName "kube-api-access-fnnjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:08.165335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:08.165294    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da765c6-1b1b-47f4-856b-49166e411857-kube-api-access-f8qsf" pod "2da765c6-1b1b-47f4-856b-49166e411857" (UID: "2da765c6-1b1b-47f4-856b-49166e411857"). InnerVolumeSpecName "kube-api-access-f8qsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:08.195626 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:08.195597    4544 scope.go:122] "RemoveContainer" containerID="a669b8befe9b5f1ceff6dc039e3571b920a2e9089326777d31092f346a90becf"
Oct 30 13:07:08.206875 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:08.206852    4544 scope.go:122] "RemoveContainer" containerID="7b546a120049a7ba22c8c12bf096f0674f2cef176ce24e5c00acca266d7e4467"
Oct 30 13:07:08.218144 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:08.218113    4544 scope.go:122] "RemoveContainer" containerID="d0561c8547ebad47d60986824a6c7087f70156fd49918053202e85120f5b55dc"
Oct 30 13:07:08.233465 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:08.233433    4544 scope.go:122] "RemoveContainer" containerID="474cbafe2ed3cb7d2eb894b353d88336b2321c3971603625df5a9751e916623f"
Oct 30 13:07:08.243113 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:08.243080    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnnjz\" (UniqueName: \"kubernetes.io/projected/5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce-kube-api-access-fnnjz\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:07:08.243113 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:08.243114    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8qsf\" (UniqueName: \"kubernetes.io/projected/2da765c6-1b1b-47f4-856b-49166e411857-kube-api-access-f8qsf\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:07:08.246809 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:08.246777    4544 scope.go:122] "RemoveContainer" containerID="6f93200ccff66676b5f9f32f4164c2cb92e8a855419f719eeb685de2ff402874"
Oct 30 13:07:08.258843 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:08.258812    4544 scope.go:122] "RemoveContainer" containerID="125e98e769a7e32dfd706a14f747dc1e768928561225ba31468f5328eb20143f"
Oct 30 13:07:08.319880 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:08.319849    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:08.409826 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:07:08.409725    4544 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456ae1e2_ef5b_427c_a4a3_fba864a91b55.slice/crio-ad709b24144d870fcafc47d97dd53fb0d9f0d84a71d7824bcefb62f47a8c89e4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456ae1e2_ef5b_427c_a4a3_fba864a91b55.slice/crio-conmon-ad709b24144d870fcafc47d97dd53fb0d9f0d84a71d7824bcefb62f47a8c89e4.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 13:07:08.467019 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:08.466981    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:08.700185 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:08.700077    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:08.753301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:08.753254    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6" event={"ID":"2da765c6-1b1b-47f4-856b-49166e411857","Type":"ContainerDied","Data":"61c89b695d1601276f399bf1803c55ce0cfb9f23ab7b336ff17397e56908c530"}
Oct 30 13:07:08.753515 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:08.753354    4544 scope.go:122] "RemoveContainer" containerID="464f3ce375a2a81f1ca37d4917031a2862ed1ce04db6ba070ccff26e45e8238e"
Oct 30 13:07:08.753515 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:08.753265    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6"
Oct 30 13:07:08.755145 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:08.755106    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh" event={"ID":"5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce","Type":"ContainerDied","Data":"192e4da96f2f3fa16c71ceaf3f1043ca6d8fbbdcda58915c2f65f4f2612e7a59"}
Oct 30 13:07:08.755306 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:08.755125    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh"
Oct 30 13:07:08.766920 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:08.766895    4544 scope.go:122] "RemoveContainer" containerID="31d9923c6ab6a8f631dd89a473813a0bd606f0903adc6b88f8059c0e9c9edb25"
Oct 30 13:07:08.813353 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:08.813315    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh"]
Oct 30 13:07:08.822910 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:08.822883    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh"]
Oct 30 13:07:08.868945 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:08.868905    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6"]
Oct 30 13:07:08.874583 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:08.874552    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6"]
Oct 30 13:07:09.088161 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:09.088125    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2da765c6-1b1b-47f4-856b-49166e411857" path="/var/lib/kubelet/pods/2da765c6-1b1b-47f4-856b-49166e411857/volumes"
Oct 30 13:07:09.088702 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:09.088685    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="456ae1e2-ef5b-427c-a4a3-fba864a91b55" path="/var/lib/kubelet/pods/456ae1e2-ef5b-427c-a4a3-fba864a91b55/volumes"
Oct 30 13:07:09.089060 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:09.089048    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce" path="/var/lib/kubelet/pods/5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce/volumes"
Oct 30 13:07:09.401893 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:09.401797    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:10.031049 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:10.031016    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a106d20d-hqnr5"]
Oct 30 13:07:10.042431 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:10.042402    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a106d20d-hqnr5"]
Oct 30 13:07:11.089724 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:11.089676    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d40b4984-06b7-4d0d-bc03-39491f39cf2d" path="/var/lib/kubelet/pods/d40b4984-06b7-4d0d-bc03-39491f39cf2d/volumes"
Oct 30 13:07:11.174584 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:11.174549    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:12.590182 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.590155    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:07:12.611083 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.611046    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istio-podinfo\") pod \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") "
Oct 30 13:07:12.611291 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.611191    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istiod-ca-cert\") pod \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") "
Oct 30 13:07:12.611291 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.611285    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b940dbd-f31b-4620-84e8-f32fc7feef3e-kube-api-access-rq5v5\" (UniqueName: \"kubernetes.io/projected/4b940dbd-f31b-4620-84e8-f32fc7feef3e-kube-api-access-rq5v5\") pod \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") "
Oct 30 13:07:12.611385 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.611349    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-workload-socket\") pod \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") "
Oct 30 13:07:12.611439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.611422    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-credential-socket\") pod \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") "
Oct 30 13:07:12.611511 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.611483    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istio-data\") pod \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") "
Oct 30 13:07:12.611568 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.611536    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istio-envoy\") pod \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") "
Oct 30 13:07:12.611646 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.611611    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-workload-socket" pod "4b940dbd-f31b-4620-84e8-f32fc7feef3e" (UID: "4b940dbd-f31b-4620-84e8-f32fc7feef3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:12.611706 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.611697    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-workload-certs\") pod \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") "
Oct 30 13:07:12.611763 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.611724    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-credential-socket" pod "4b940dbd-f31b-4620-84e8-f32fc7feef3e" (UID: "4b940dbd-f31b-4620-84e8-f32fc7feef3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:12.611763 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.611651    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istiod-ca-cert" pod "4b940dbd-f31b-4620-84e8-f32fc7feef3e" (UID: "4b940dbd-f31b-4620-84e8-f32fc7feef3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:07:12.611763 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.611733    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istio-token\" (UniqueName: \"kubernetes.io/projected/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istio-token\") pod \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\" (UID: \"4b940dbd-f31b-4620-84e8-f32fc7feef3e\") "
Oct 30 13:07:12.611926 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.611769    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istio-data" pod "4b940dbd-f31b-4620-84e8-f32fc7feef3e" (UID: "4b940dbd-f31b-4620-84e8-f32fc7feef3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:12.611995 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.611935    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-workload-certs" pod "4b940dbd-f31b-4620-84e8-f32fc7feef3e" (UID: "4b940dbd-f31b-4620-84e8-f32fc7feef3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:12.612598 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.612570    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:07:12.612598 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.612600    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:07:12.612785 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.612614    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:07:12.612785 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.612630    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:07:12.612785 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.612643    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:07:12.613528 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.613505    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istio-podinfo" pod "4b940dbd-f31b-4620-84e8-f32fc7feef3e" (UID: "4b940dbd-f31b-4620-84e8-f32fc7feef3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:07:12.613718 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.613697    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b940dbd-f31b-4620-84e8-f32fc7feef3e-kube-api-access-rq5v5" pod "4b940dbd-f31b-4620-84e8-f32fc7feef3e" (UID: "4b940dbd-f31b-4620-84e8-f32fc7feef3e"). InnerVolumeSpecName "kube-api-access-rq5v5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:12.613797 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.613780    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istio-envoy" pod "4b940dbd-f31b-4620-84e8-f32fc7feef3e" (UID: "4b940dbd-f31b-4620-84e8-f32fc7feef3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:12.613914 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.613895    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istio-token" pod "4b940dbd-f31b-4620-84e8-f32fc7feef3e" (UID: "4b940dbd-f31b-4620-84e8-f32fc7feef3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:12.714206 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.714158    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:07:12.714206 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.714202    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:07:12.714206 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.714216    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b940dbd-f31b-4620-84e8-f32fc7feef3e-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:07:12.714206 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.714244    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq5v5\" (UniqueName: \"kubernetes.io/projected/4b940dbd-f31b-4620-84e8-f32fc7feef3e-kube-api-access-rq5v5\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:07:12.795055 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.795019    4544 generic.go:356] "Generic (PLEG): container finished" podID="4b940dbd-f31b-4620-84e8-f32fc7feef3e" containerID="bcf3fa3d82eb1a4dd911c8939ad732a1c89f32dc4fa988403bd888aec96892dc" exitCode=0
Oct 30 13:07:12.795306 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.795079    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"
Oct 30 13:07:12.795306 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.795093    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9" event={"ID":"4b940dbd-f31b-4620-84e8-f32fc7feef3e","Type":"ContainerDied","Data":"bcf3fa3d82eb1a4dd911c8939ad732a1c89f32dc4fa988403bd888aec96892dc"}
Oct 30 13:07:12.795306 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.795147    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9" event={"ID":"4b940dbd-f31b-4620-84e8-f32fc7feef3e","Type":"ContainerDied","Data":"dbc7151097c51d110e5897179f53809bae0da3d50a8d0c81e41c07ac6cc2738c"}
Oct 30 13:07:12.795306 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.795170    4544 scope.go:122] "RemoveContainer" containerID="bcf3fa3d82eb1a4dd911c8939ad732a1c89f32dc4fa988403bd888aec96892dc"
Oct 30 13:07:12.809400 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.809348    4544 scope.go:122] "RemoveContainer" containerID="2b0a772caa410bddc4a83bdb14a6d6415ab43bf8b561824b8f14ba86d8f75adc"
Oct 30 13:07:12.824318 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.824293    4544 scope.go:122] "RemoveContainer" containerID="bcf3fa3d82eb1a4dd911c8939ad732a1c89f32dc4fa988403bd888aec96892dc"
Oct 30 13:07:12.824748 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:07:12.824719    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcf3fa3d82eb1a4dd911c8939ad732a1c89f32dc4fa988403bd888aec96892dc\": container with ID starting with bcf3fa3d82eb1a4dd911c8939ad732a1c89f32dc4fa988403bd888aec96892dc not found: ID does not exist" containerID="bcf3fa3d82eb1a4dd911c8939ad732a1c89f32dc4fa988403bd888aec96892dc"
Oct 30 13:07:12.824876 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.824759    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcf3fa3d82eb1a4dd911c8939ad732a1c89f32dc4fa988403bd888aec96892dc"} err="failed to get container status \"bcf3fa3d82eb1a4dd911c8939ad732a1c89f32dc4fa988403bd888aec96892dc\": rpc error: code = NotFound desc = could not find container \"bcf3fa3d82eb1a4dd911c8939ad732a1c89f32dc4fa988403bd888aec96892dc\": container with ID starting with bcf3fa3d82eb1a4dd911c8939ad732a1c89f32dc4fa988403bd888aec96892dc not found: ID does not exist"
Oct 30 13:07:12.824876 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.824786    4544 scope.go:122] "RemoveContainer" containerID="2b0a772caa410bddc4a83bdb14a6d6415ab43bf8b561824b8f14ba86d8f75adc"
Oct 30 13:07:12.825159 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:07:12.825128    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b0a772caa410bddc4a83bdb14a6d6415ab43bf8b561824b8f14ba86d8f75adc\": container with ID starting with 2b0a772caa410bddc4a83bdb14a6d6415ab43bf8b561824b8f14ba86d8f75adc not found: ID does not exist" containerID="2b0a772caa410bddc4a83bdb14a6d6415ab43bf8b561824b8f14ba86d8f75adc"
Oct 30 13:07:12.825272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.825168    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b0a772caa410bddc4a83bdb14a6d6415ab43bf8b561824b8f14ba86d8f75adc"} err="failed to get container status \"2b0a772caa410bddc4a83bdb14a6d6415ab43bf8b561824b8f14ba86d8f75adc\": rpc error: code = NotFound desc = could not find container \"2b0a772caa410bddc4a83bdb14a6d6415ab43bf8b561824b8f14ba86d8f75adc\": container with ID starting with 2b0a772caa410bddc4a83bdb14a6d6415ab43bf8b561824b8f14ba86d8f75adc not found: ID does not exist"
Oct 30 13:07:12.856445 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.856402    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"]
Oct 30 13:07:12.862256 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:12.862204    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9"]
Oct 30 13:07:13.061468 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:13.061377    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:13.186301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:13.184387    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b940dbd-f31b-4620-84e8-f32fc7feef3e" path="/var/lib/kubelet/pods/4b940dbd-f31b-4620-84e8-f32fc7feef3e/volumes"
Oct 30 13:07:14.772430 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:14.772399    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb"
Oct 30 13:07:15.121977 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:15.121929    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:15.988504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:15.988470    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:16.268998 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:16.268904    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:16.554524 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:16.554421    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:17.000944 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.000906    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"]
Oct 30 13:07:17.002100 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.002071    4544 cpu_manager.go:414] "containerMap: removing container" podUID="456ae1e2-ef5b-427c-a4a3-fba864a91b55" containerName="maas-api"
Oct 30 13:07:17.002100 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.002100    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="456ae1e2-ef5b-427c-a4a3-fba864a91b55" containerName="maas-api"
Oct 30 13:07:17.002364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.002120    4544 cpu_manager.go:414] "containerMap: removing container" podUID="5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce" containerName="payload-pre-processing"
Oct 30 13:07:17.002364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.002127    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce" containerName="payload-pre-processing"
Oct 30 13:07:17.002364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.002183    4544 cpu_manager.go:414] "containerMap: removing container" podUID="4b940dbd-f31b-4620-84e8-f32fc7feef3e" containerName="istio-proxy"
Oct 30 13:07:17.002364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.002191    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b940dbd-f31b-4620-84e8-f32fc7feef3e" containerName="istio-proxy"
Oct 30 13:07:17.002364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.002215    4544 cpu_manager.go:414] "containerMap: removing container" podUID="4b940dbd-f31b-4620-84e8-f32fc7feef3e" containerName="istio-proxy"
Oct 30 13:07:17.002364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.002238    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b940dbd-f31b-4620-84e8-f32fc7feef3e" containerName="istio-proxy"
Oct 30 13:07:17.002364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.002273    4544 cpu_manager.go:414] "containerMap: removing container" podUID="2da765c6-1b1b-47f4-856b-49166e411857" containerName="payload-processing"
Oct 30 13:07:17.002364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.002280    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2da765c6-1b1b-47f4-856b-49166e411857" containerName="payload-processing"
Oct 30 13:07:17.002684 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.002455    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="456ae1e2-ef5b-427c-a4a3-fba864a91b55" containerName="maas-api"
Oct 30 13:07:17.002684 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.002467    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ccbdf68-0a6e-44a7-8acf-1cb0632c78ce" containerName="payload-pre-processing"
Oct 30 13:07:17.002684 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.002485    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b940dbd-f31b-4620-84e8-f32fc7feef3e" containerName="istio-proxy"
Oct 30 13:07:17.002684 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.002501    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="2da765c6-1b1b-47f4-856b-49166e411857" containerName="payload-processing"
Oct 30 13:07:17.008949 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.008906    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:07:17.047758 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.047724    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-2afe58-openshift-default-dockercfg-dhp2v\""
Oct 30 13:07:17.068116 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.068078    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"]
Oct 30 13:07:17.093711 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.093672    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:17.163246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.162313    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-workload-certs\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:07:17.163246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.162443    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75391895-2b05-426d-98ad-87ad85d2e9c3-istio-podinfo\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:07:17.163246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.162501    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-istio-data\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:07:17.163246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.162546    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-workload-socket\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:07:17.163246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.162748    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-istio-envoy\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:07:17.370248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.162870    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-credential-socket\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:07:17.370248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.368089    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75391895-2b05-426d-98ad-87ad85d2e9c3-istio-token\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:07:17.370248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.368240    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx98c\" (UniqueName: \"kubernetes.io/projected/75391895-2b05-426d-98ad-87ad85d2e9c3-kube-api-access-cx98c\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:07:17.370248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.368335    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75391895-2b05-426d-98ad-87ad85d2e9c3-istiod-ca-cert\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:07:17.447256 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.445891    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:17.471687 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.471646    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-credential-socket\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:07:17.472045 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.472026    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75391895-2b05-426d-98ad-87ad85d2e9c3-istio-token\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:07:17.472426 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.472399    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx98c\" (UniqueName: \"kubernetes.io/projected/75391895-2b05-426d-98ad-87ad85d2e9c3-kube-api-access-cx98c\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:07:17.472646 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.472631    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75391895-2b05-426d-98ad-87ad85d2e9c3-istiod-ca-cert\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:07:17.472855 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.472839    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-workload-certs\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:07:17.472973 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.472961    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75391895-2b05-426d-98ad-87ad85d2e9c3-istio-podinfo\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:07:17.473110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.473094    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-istio-data\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:07:17.473244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.473213    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-workload-socket\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:07:17.473403 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.473391    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-istio-envoy\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:07:17.479164 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.479129    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75391895-2b05-426d-98ad-87ad85d2e9c3-istiod-ca-cert\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:07:17.494431 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.480133    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-credential-socket\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:07:17.494661 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.480842    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-workload-certs\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:07:17.494818 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.481490    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-istio-data\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:07:17.494818 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.482030    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-workload-socket\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:07:17.498446 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.498406    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-istio-envoy\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:07:17.509882 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.509848    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75391895-2b05-426d-98ad-87ad85d2e9c3-istio-token\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:07:17.513322 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.513290    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx98c\" (UniqueName: \"kubernetes.io/projected/75391895-2b05-426d-98ad-87ad85d2e9c3-kube-api-access-cx98c\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:07:17.516949 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.516910    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75391895-2b05-426d-98ad-87ad85d2e9c3-istio-podinfo\") pod \"e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") " pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:07:17.628428 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.628327    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:07:17.718588 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:17.717876    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:18.298885 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:18.298846    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"]
Oct 30 13:07:18.299894 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:07:18.299863    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75391895_2b05_426d_98ad_87ad85d2e9c3.slice/crio-34f0c24244f7b06467b0d506e09e5ba197643c0b5c3994a964b1e9481a9078aa WatchSource:0}: Error finding container 34f0c24244f7b06467b0d506e09e5ba197643c0b5c3994a964b1e9481a9078aa: Status 404 returned error can't find the container with id 34f0c24244f7b06467b0d506e09e5ba197643c0b5c3994a964b1e9481a9078aa
Oct 30 13:07:18.302687 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:18.302636    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:07:18.302787 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:18.302722    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:07:18.302787 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:18.302754    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:07:18.499169 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:07:18.499134    4544 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456ae1e2_ef5b_427c_a4a3_fba864a91b55.slice/crio-ad709b24144d870fcafc47d97dd53fb0d9f0d84a71d7824bcefb62f47a8c89e4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456ae1e2_ef5b_427c_a4a3_fba864a91b55.slice/crio-conmon-ad709b24144d870fcafc47d97dd53fb0d9f0d84a71d7824bcefb62f47a8c89e4.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 13:07:19.162842 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:19.162801    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q" event={"ID":"75391895-2b05-426d-98ad-87ad85d2e9c3","Type":"ContainerStarted","Data":"4d2447981a522cbf7003cb6702442c0af55f422dd1521822112c515e9b06111c"}
Oct 30 13:07:19.163033 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:19.162868    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q" event={"ID":"75391895-2b05-426d-98ad-87ad85d2e9c3","Type":"ContainerStarted","Data":"34f0c24244f7b06467b0d506e09e5ba197643c0b5c3994a964b1e9481a9078aa"}
Oct 30 13:07:19.243537 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:19.243463    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q" podStartSLOduration=3.243439255 podStartE2EDuration="3.243439255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:19.237720073 +0000 UTC m=+1334.284854026" watchObservedRunningTime="2026-10-30 13:07:19.243439255 +0000 UTC m=+1334.290573209"
Oct 30 13:07:19.580149 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:07:19.580101    4544 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456ae1e2_ef5b_427c_a4a3_fba864a91b55.slice/crio-ad709b24144d870fcafc47d97dd53fb0d9f0d84a71d7824bcefb62f47a8c89e4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456ae1e2_ef5b_427c_a4a3_fba864a91b55.slice/crio-conmon-ad709b24144d870fcafc47d97dd53fb0d9f0d84a71d7824bcefb62f47a8c89e4.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 13:07:19.580668 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:07:19.580279    4544 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456ae1e2_ef5b_427c_a4a3_fba864a91b55.slice/crio-ad709b24144d870fcafc47d97dd53fb0d9f0d84a71d7824bcefb62f47a8c89e4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456ae1e2_ef5b_427c_a4a3_fba864a91b55.slice/crio-conmon-ad709b24144d870fcafc47d97dd53fb0d9f0d84a71d7824bcefb62f47a8c89e4.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 13:07:19.641325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:19.640452    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:07:19.648666 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:19.648631    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:07:19.738246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:19.735839    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:20.171285 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:20.171252    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:07:20.172458 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:20.172432    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:07:21.737293 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:07:21.730521    4544 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456ae1e2_ef5b_427c_a4a3_fba864a91b55.slice/crio-conmon-ad709b24144d870fcafc47d97dd53fb0d9f0d84a71d7824bcefb62f47a8c89e4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456ae1e2_ef5b_427c_a4a3_fba864a91b55.slice/crio-ad709b24144d870fcafc47d97dd53fb0d9f0d84a71d7824bcefb62f47a8c89e4.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 13:07:21.960492 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:21.960453    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:22.125244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:22.121665    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:22.875946 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:22.875912    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:23.086419 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:23.086389    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:23.247942 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:23.247829    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:23.838209 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:23.826278    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:24.010886 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:24.010848    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:24.044680 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:24.044644    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h"]
Oct 30 13:07:24.045496 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:24.045480    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b940dbd-f31b-4620-84e8-f32fc7feef3e" containerName="istio-proxy"
Oct 30 13:07:24.050763 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:24.050740    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h"
Oct 30 13:07:24.080594 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:24.080556    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-2afe58-dockercfg-dvdgj\""
Oct 30 13:07:24.080594 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:24.080577    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-2afe58\""
Oct 30 13:07:24.086140 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:24.086110    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e24e8752-2npsq"]
Oct 30 13:07:24.091332 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:24.091261    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e24e8752-2npsq"]
Oct 30 13:07:24.097267 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:24.096904    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h"]
Oct 30 13:07:24.246359 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:24.243851    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgp6l\" (UniqueName: \"kubernetes.io/projected/329fbf95-10bc-4b2f-afc2-14c679d9edd8-kube-api-access-sgp6l\") pod \"payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h\" (UID: \"329fbf95-10bc-4b2f-afc2-14c679d9edd8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h"
Oct 30 13:07:24.246359 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:24.243970    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/329fbf95-10bc-4b2f-afc2-14c679d9edd8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h\" (UID: \"329fbf95-10bc-4b2f-afc2-14c679d9edd8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h"
Oct 30 13:07:24.460352 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:24.450344    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m"]
Oct 30 13:07:24.537284 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:24.526496    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m"]
Oct 30 13:07:24.537565 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:24.529367    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m"
Oct 30 13:07:24.558178 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:24.555586    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/329fbf95-10bc-4b2f-afc2-14c679d9edd8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h\" (UID: \"329fbf95-10bc-4b2f-afc2-14c679d9edd8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h"
Oct 30 13:07:24.558178 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:24.555757    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/611e617c-64ab-4006-972d-fa789281cc38-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m\" (UID: \"611e617c-64ab-4006-972d-fa789281cc38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m"
Oct 30 13:07:24.558178 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:24.555922    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xt6x\" (UniqueName: \"kubernetes.io/projected/611e617c-64ab-4006-972d-fa789281cc38-kube-api-access-4xt6x\") pod \"payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m\" (UID: \"611e617c-64ab-4006-972d-fa789281cc38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m"
Oct 30 13:07:24.558178 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:24.556069    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgp6l\" (UniqueName: \"kubernetes.io/projected/329fbf95-10bc-4b2f-afc2-14c679d9edd8-kube-api-access-sgp6l\") pod \"payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h\" (UID: \"329fbf95-10bc-4b2f-afc2-14c679d9edd8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h"
Oct 30 13:07:24.558178 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:24.556550    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/329fbf95-10bc-4b2f-afc2-14c679d9edd8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h\" (UID: \"329fbf95-10bc-4b2f-afc2-14c679d9edd8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h"
Oct 30 13:07:24.584130 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:24.584097    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgp6l\" (UniqueName: \"kubernetes.io/projected/329fbf95-10bc-4b2f-afc2-14c679d9edd8-kube-api-access-sgp6l\") pod \"payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h\" (UID: \"329fbf95-10bc-4b2f-afc2-14c679d9edd8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h"
Oct 30 13:07:24.657303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:24.657212    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xt6x\" (UniqueName: \"kubernetes.io/projected/611e617c-64ab-4006-972d-fa789281cc38-kube-api-access-4xt6x\") pod \"payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m\" (UID: \"611e617c-64ab-4006-972d-fa789281cc38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m"
Oct 30 13:07:24.657553 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:24.657534    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/611e617c-64ab-4006-972d-fa789281cc38-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m\" (UID: \"611e617c-64ab-4006-972d-fa789281cc38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m"
Oct 30 13:07:24.658720 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:24.658702    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/611e617c-64ab-4006-972d-fa789281cc38-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m\" (UID: \"611e617c-64ab-4006-972d-fa789281cc38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m"
Oct 30 13:07:24.668131 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:24.668104    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h"
Oct 30 13:07:24.684239 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:24.684186    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xt6x\" (UniqueName: \"kubernetes.io/projected/611e617c-64ab-4006-972d-fa789281cc38-kube-api-access-4xt6x\") pod \"payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m\" (UID: \"611e617c-64ab-4006-972d-fa789281cc38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m"
Oct 30 13:07:24.878497 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:24.878458    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m"
Oct 30 13:07:24.983926 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:24.983895    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h"]
Oct 30 13:07:24.986199 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:07:24.986166    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod329fbf95_10bc_4b2f_afc2_14c679d9edd8.slice/crio-b6aaffbba92aa7ff5fd0671241d717770317bf76fbc92231c1a03ca631c8a730 WatchSource:0}: Error finding container b6aaffbba92aa7ff5fd0671241d717770317bf76fbc92231c1a03ca631c8a730: Status 404 returned error can't find the container with id b6aaffbba92aa7ff5fd0671241d717770317bf76fbc92231c1a03ca631c8a730
Oct 30 13:07:25.064273 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:25.058266    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:25.168338 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:25.168299    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a52feac-644f-4fe3-9599-8f6af1172191" path="/var/lib/kubelet/pods/3a52feac-644f-4fe3-9599-8f6af1172191/volumes"
Oct 30 13:07:25.213370 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:07:25.213339    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod611e617c_64ab_4006_972d_fa789281cc38.slice/crio-14bcd1d0c1c39d110888b39465badad2ce57d8c8ca40eb24d049ebf8089093bd WatchSource:0}: Error finding container 14bcd1d0c1c39d110888b39465badad2ce57d8c8ca40eb24d049ebf8089093bd: Status 404 returned error can't find the container with id 14bcd1d0c1c39d110888b39465badad2ce57d8c8ca40eb24d049ebf8089093bd
Oct 30 13:07:25.214613 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:25.214586    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m"]
Oct 30 13:07:25.577981 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:25.577940    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m" event={"ID":"611e617c-64ab-4006-972d-fa789281cc38","Type":"ContainerStarted","Data":"526813699e40d766f1321b213aaceeb866a0070b92310b5f5662b449af998e01"}
Oct 30 13:07:25.578196 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:25.578004    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m"
Oct 30 13:07:25.578196 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:25.578027    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m" event={"ID":"611e617c-64ab-4006-972d-fa789281cc38","Type":"ContainerStarted","Data":"14bcd1d0c1c39d110888b39465badad2ce57d8c8ca40eb24d049ebf8089093bd"}
Oct 30 13:07:25.579871 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:25.579841    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h" event={"ID":"329fbf95-10bc-4b2f-afc2-14c679d9edd8","Type":"ContainerStarted","Data":"552f1749d883c7e3d623a150d03d8efc63459572b4d8bffe2bf42a057fab6afd"}
Oct 30 13:07:25.580003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:25.579884    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h" event={"ID":"329fbf95-10bc-4b2f-afc2-14c679d9edd8","Type":"ContainerStarted","Data":"b6aaffbba92aa7ff5fd0671241d717770317bf76fbc92231c1a03ca631c8a730"}
Oct 30 13:07:25.580003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:25.579938    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h"
Oct 30 13:07:25.624081 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:25.624017    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m" podStartSLOduration=1.623998542 podStartE2EDuration="1.623998542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:25.61875652 +0000 UTC m=+1340.665890489" watchObservedRunningTime="2026-10-30 13:07:25.623998542 +0000 UTC m=+1340.671132495"
Oct 30 13:07:25.663720 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:25.663659    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h" podStartSLOduration=1.663642107 podStartE2EDuration="1.663642107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:25.660154896 +0000 UTC m=+1340.707288848" watchObservedRunningTime="2026-10-30 13:07:25.663642107 +0000 UTC m=+1340.710776061"
Oct 30 13:07:27.013146 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:27.013096    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:28.598876 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:07:28.598823    4544 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456ae1e2_ef5b_427c_a4a3_fba864a91b55.slice/crio-ad709b24144d870fcafc47d97dd53fb0d9f0d84a71d7824bcefb62f47a8c89e4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456ae1e2_ef5b_427c_a4a3_fba864a91b55.slice/crio-conmon-ad709b24144d870fcafc47d97dd53fb0d9f0d84a71d7824bcefb62f47a8c89e4.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 13:07:29.010528 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:07:29.010488    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:07:29.189045 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:29.188934    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:29.618174 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:29.618139    4544 generic.go:356] "Generic (PLEG): container finished" podID="f7df19fe-9b2b-4e96-a6d2-e1d5015c931f" containerID="11cf2aa8460bc44631b9a50a36c260adcfafc7702591893ffccd29d607a766d6" exitCode=0
Oct 30 13:07:29.618632 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:29.618229    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx" event={"ID":"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f","Type":"ContainerDied","Data":"11cf2aa8460bc44631b9a50a36c260adcfafc7702591893ffccd29d607a766d6"}
Oct 30 13:07:29.618734 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:29.618718    4544 scope.go:122] "RemoveContainer" containerID="11cf2aa8460bc44631b9a50a36c260adcfafc7702591893ffccd29d607a766d6"
Oct 30 13:07:29.620373 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:29.620345    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:07:29.620457 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:29.620418    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:07:29.620494 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:29.620456    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:07:29.653457 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:29.653402    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:29.792474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:29.792442    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:30.349685 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:30.349646    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:07:30.644570 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:30.644480    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx" event={"ID":"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f","Type":"ContainerStarted","Data":"4c144566ea356b79c48949ac7032ba4c2ee3c05ec6818308e79394f9504da025"}
Oct 30 13:07:30.782534 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:30.782484    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:07:30.783565 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:30.783535    4544 patch_prober.go:29] interesting pod/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 13:07:30.783688 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:30.783595    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx" podUID="f7df19fe-9b2b-4e96-a6d2-e1d5015c931f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 13:07:31.157395 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:31.157326    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:31.782862 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:31.782827    4544 patch_prober.go:29] interesting pod/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 13:07:31.783269 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:31.782883    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx" podUID="f7df19fe-9b2b-4e96-a6d2-e1d5015c931f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 13:07:32.782829 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:32.782790    4544 patch_prober.go:29] interesting pod/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 13:07:32.783030 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:32.782859    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx" podUID="f7df19fe-9b2b-4e96-a6d2-e1d5015c931f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 13:07:33.091400 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:33.091354    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:33.783127 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:33.783085    4544 patch_prober.go:29] interesting pod/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 13:07:33.783603 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:33.783147    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx" podUID="f7df19fe-9b2b-4e96-a6d2-e1d5015c931f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 13:07:34.783418 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:34.783371    4544 patch_prober.go:29] interesting pod/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 13:07:34.783833 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:34.783446    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx" podUID="f7df19fe-9b2b-4e96-a6d2-e1d5015c931f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 13:07:35.002476 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.002439    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:35.223557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.223518    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:35.504525 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.504426    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"]
Oct 30 13:07:35.510483 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.510456    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:07:35.541834 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.541782    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-1c0e6c42-openshift-default-dockercfg-zstsq\""
Oct 30 13:07:35.560775 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.560708    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"]
Oct 30 13:07:35.583925 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.583890    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48119de6-b81b-4f07-b304-a33e16ab2d68-istio-podinfo\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:07:35.584249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.584213    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-credential-socket\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:07:35.584359 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.584349    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-workload-certs\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:07:35.584501 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.584466    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-istio-data\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:07:35.584642 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.584617    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48119de6-b81b-4f07-b304-a33e16ab2d68-istiod-ca-cert\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:07:35.584729 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.584647    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-workload-socket\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:07:35.584801 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.584724    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r84s7\" (UniqueName: \"kubernetes.io/projected/48119de6-b81b-4f07-b304-a33e16ab2d68-kube-api-access-r84s7\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:07:35.584801 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.584748    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-istio-envoy\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:07:35.584801 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.584793    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48119de6-b81b-4f07-b304-a33e16ab2d68-istio-token\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:07:35.640368 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.640331    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:35.696594 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.696553    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-credential-socket\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:07:35.722361 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.722320    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-workload-certs\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:07:35.722649 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.722619    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-istio-data\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:07:35.722976 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.722954    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48119de6-b81b-4f07-b304-a33e16ab2d68-istiod-ca-cert\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:07:35.723122 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.723105    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-workload-socket\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:07:35.723378 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.723361    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r84s7\" (UniqueName: \"kubernetes.io/projected/48119de6-b81b-4f07-b304-a33e16ab2d68-kube-api-access-r84s7\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:07:35.723518 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.723502    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-istio-envoy\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:07:35.723702 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.723676    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48119de6-b81b-4f07-b304-a33e16ab2d68-istio-token\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:07:35.723943 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.723922    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48119de6-b81b-4f07-b304-a33e16ab2d68-istio-podinfo\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:07:35.740249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.734558    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48119de6-b81b-4f07-b304-a33e16ab2d68-istiod-ca-cert\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:07:35.740249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.721692    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-credential-socket\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:07:35.740249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.735025    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-workload-certs\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:07:35.740249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.735290    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-istio-data\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:07:35.744112 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.744079    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-workload-socket\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:07:35.763353 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.763267    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48119de6-b81b-4f07-b304-a33e16ab2d68-istio-podinfo\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:07:35.816628 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.786631    4544 patch_prober.go:29] interesting pod/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 13:07:35.816628 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.786717    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx" podUID="f7df19fe-9b2b-4e96-a6d2-e1d5015c931f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 13:07:35.816628 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.799531    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-istio-envoy\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:07:35.830248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.825099    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r84s7\" (UniqueName: \"kubernetes.io/projected/48119de6-b81b-4f07-b304-a33e16ab2d68-kube-api-access-r84s7\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:07:35.833002 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.832929    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48119de6-b81b-4f07-b304-a33e16ab2d68-istio-token\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:07:35.861906 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.861621    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:07:35.949992 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:35.949878    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:36.378490 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:07:36.378452    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48119de6_b81b_4f07_b304_a33e16ab2d68.slice/crio-d40973bc0792a36a5e9b66b36871ee77900c97dd52fb82192e27174eb40352bd WatchSource:0}: Error finding container d40973bc0792a36a5e9b66b36871ee77900c97dd52fb82192e27174eb40352bd: Status 404 returned error can't find the container with id d40973bc0792a36a5e9b66b36871ee77900c97dd52fb82192e27174eb40352bd
Oct 30 13:07:36.379191 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:36.379160    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"]
Oct 30 13:07:36.381160 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:36.381125    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:07:36.381301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:36.381196    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:07:36.381301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:36.381265    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:07:36.593416 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:36.593387    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h"
Oct 30 13:07:36.593607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:36.593455    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m"
Oct 30 13:07:36.789597 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:36.789517    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:07:36.789993 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:36.789974    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:07:36.791109 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:36.791086    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:07:36.823907 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:07:36.823860    4544 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456ae1e2_ef5b_427c_a4a3_fba864a91b55.slice/crio-conmon-ad709b24144d870fcafc47d97dd53fb0d9f0d84a71d7824bcefb62f47a8c89e4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456ae1e2_ef5b_427c_a4a3_fba864a91b55.slice/crio-ad709b24144d870fcafc47d97dd53fb0d9f0d84a71d7824bcefb62f47a8c89e4.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 13:07:36.855680 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:36.855642    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d" event={"ID":"48119de6-b81b-4f07-b304-a33e16ab2d68","Type":"ContainerStarted","Data":"0ce2a491e46e690ccbac4fa6554b14a117fa4a58145487fe1e086774fa4f2492"}
Oct 30 13:07:36.855895 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:36.855707    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d" event={"ID":"48119de6-b81b-4f07-b304-a33e16ab2d68","Type":"ContainerStarted","Data":"d40973bc0792a36a5e9b66b36871ee77900c97dd52fb82192e27174eb40352bd"}
Oct 30 13:07:36.864641 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:36.864608    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:07:37.039327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:37.039270    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d" podStartSLOduration=2.039249954 podStartE2EDuration="2.039249954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:37.033270151 +0000 UTC m=+1352.080404104" watchObservedRunningTime="2026-10-30 13:07:37.039249954 +0000 UTC m=+1352.086383903"
Oct 30 13:07:37.586991 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:37.586956    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:37.872310 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:37.872184    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:07:38.616613 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:07:38.616572    4544 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456ae1e2_ef5b_427c_a4a3_fba864a91b55.slice/crio-conmon-ad709b24144d870fcafc47d97dd53fb0d9f0d84a71d7824bcefb62f47a8c89e4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456ae1e2_ef5b_427c_a4a3_fba864a91b55.slice/crio-ad709b24144d870fcafc47d97dd53fb0d9f0d84a71d7824bcefb62f47a8c89e4.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 13:07:38.873851 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:38.873762    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:07:38.874952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:38.874931    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:07:39.591709 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:39.591670    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:39.803316 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:39.799282    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:41.090951 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:41.090921    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a63d1b22-trb28"]
Oct 30 13:07:41.098801 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:41.098768    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a63d1b22-trb28"]
Oct 30 13:07:41.367856 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:41.367770    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:41.486061 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:41.486027    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:42.043176 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:42.043141    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:42.193244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:42.178412    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:42.569422 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:42.569382    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:42.704420 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:42.704375    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt"]
Oct 30 13:07:42.752690 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:42.752657    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx"]
Oct 30 13:07:42.884757 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:42.884711    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:43.077144 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:43.077078    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:43.123201 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:43.123165    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d7b6fa9-cc8f-45a8-b463-2b240666966f" path="/var/lib/kubelet/pods/8d7b6fa9-cc8f-45a8-b463-2b240666966f/volumes"
Oct 30 13:07:44.468693 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:44.468655    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:45.949971 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:45.949926    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb"]
Oct 30 13:07:45.950503 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:45.950214    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb" podUID="4485c1ce-c862-400b-9bfa-79b1a745dcd9" containerName="maas-api" containerID="cri-o://ecc1e20d3a0e0747b43654feaa5d268aeb56570f8fef03e62b4db425319c8fe9" gracePeriod=30
Oct 30 13:07:46.022266 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:46.020709    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:46.155118 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:46.154930    4544 generic.go:356] "Generic (PLEG): container finished" podID="4485c1ce-c862-400b-9bfa-79b1a745dcd9" containerID="ecc1e20d3a0e0747b43654feaa5d268aeb56570f8fef03e62b4db425319c8fe9" exitCode=0
Oct 30 13:07:46.155118 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:46.155036    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb" event={"ID":"4485c1ce-c862-400b-9bfa-79b1a745dcd9","Type":"ContainerDied","Data":"ecc1e20d3a0e0747b43654feaa5d268aeb56570f8fef03e62b4db425319c8fe9"}
Oct 30 13:07:46.465716 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:46.465686    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb"
Oct 30 13:07:46.558466 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:46.558429    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4485c1ce-c862-400b-9bfa-79b1a745dcd9-kube-api-access-ssw87\" (UniqueName: \"kubernetes.io/projected/4485c1ce-c862-400b-9bfa-79b1a745dcd9-kube-api-access-ssw87\") pod \"4485c1ce-c862-400b-9bfa-79b1a745dcd9\" (UID: \"4485c1ce-c862-400b-9bfa-79b1a745dcd9\") "
Oct 30 13:07:46.558697 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:46.558517    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4485c1ce-c862-400b-9bfa-79b1a745dcd9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4485c1ce-c862-400b-9bfa-79b1a745dcd9-metrics-certs\") pod \"4485c1ce-c862-400b-9bfa-79b1a745dcd9\" (UID: \"4485c1ce-c862-400b-9bfa-79b1a745dcd9\") "
Oct 30 13:07:46.558774 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:46.558703    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4485c1ce-c862-400b-9bfa-79b1a745dcd9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4485c1ce-c862-400b-9bfa-79b1a745dcd9-maas-api-tls\") pod \"4485c1ce-c862-400b-9bfa-79b1a745dcd9\" (UID: \"4485c1ce-c862-400b-9bfa-79b1a745dcd9\") "
Oct 30 13:07:46.558774 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:46.558741    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4485c1ce-c862-400b-9bfa-79b1a745dcd9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4485c1ce-c862-400b-9bfa-79b1a745dcd9-openshift-service-ca\") pod \"4485c1ce-c862-400b-9bfa-79b1a745dcd9\" (UID: \"4485c1ce-c862-400b-9bfa-79b1a745dcd9\") "
Oct 30 13:07:46.559575 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:46.559535    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4485c1ce-c862-400b-9bfa-79b1a745dcd9-openshift-service-ca" pod "4485c1ce-c862-400b-9bfa-79b1a745dcd9" (UID: "4485c1ce-c862-400b-9bfa-79b1a745dcd9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:07:46.560965 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:46.560939    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4485c1ce-c862-400b-9bfa-79b1a745dcd9-maas-api-tls" pod "4485c1ce-c862-400b-9bfa-79b1a745dcd9" (UID: "4485c1ce-c862-400b-9bfa-79b1a745dcd9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:07:46.561094 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:46.561066    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4485c1ce-c862-400b-9bfa-79b1a745dcd9-kube-api-access-ssw87" pod "4485c1ce-c862-400b-9bfa-79b1a745dcd9" (UID: "4485c1ce-c862-400b-9bfa-79b1a745dcd9"). InnerVolumeSpecName "kube-api-access-ssw87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:46.561729 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:46.561708    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4485c1ce-c862-400b-9bfa-79b1a745dcd9-metrics-certs" pod "4485c1ce-c862-400b-9bfa-79b1a745dcd9" (UID: "4485c1ce-c862-400b-9bfa-79b1a745dcd9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:07:46.660591 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:46.660558    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssw87\" (UniqueName: \"kubernetes.io/projected/4485c1ce-c862-400b-9bfa-79b1a745dcd9-kube-api-access-ssw87\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:07:46.660591 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:46.660588    4544 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4485c1ce-c862-400b-9bfa-79b1a745dcd9-metrics-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:07:46.660591 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:46.660598    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4485c1ce-c862-400b-9bfa-79b1a745dcd9-maas-api-tls\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:07:46.660867 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:46.660613    4544 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4485c1ce-c862-400b-9bfa-79b1a745dcd9-openshift-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:07:46.846529 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:46.846495    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:47.107352 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:47.107200    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:47.181286 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:47.176298    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d6c1321-jcxvw"]
Oct 30 13:07:47.186295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:47.186207    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb"
Oct 30 13:07:47.186438 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:47.186414    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb" event={"ID":"4485c1ce-c862-400b-9bfa-79b1a745dcd9","Type":"ContainerDied","Data":"991c184649f9d57eff71690db53a6ad9c45a8de2f9439689ce93e1f46a1735ff"}
Oct 30 13:07:47.186497 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:47.186471    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d6c1321-jcxvw"]
Oct 30 13:07:47.186545 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:47.186503    4544 scope.go:122] "RemoveContainer" containerID="ecc1e20d3a0e0747b43654feaa5d268aeb56570f8fef03e62b4db425319c8fe9"
Oct 30 13:07:47.287863 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:47.287829    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb"]
Oct 30 13:07:47.312913 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:47.312878    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-85be4bb9-55d9bd69c6-tbwzb"]
Oct 30 13:07:47.369705 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:47.369619    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"]
Oct 30 13:07:47.369863 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:47.369830    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx" podUID="f7df19fe-9b2b-4e96-a6d2-e1d5015c931f" containerName="istio-proxy" containerID="cri-o://4c144566ea356b79c48949ac7032ba4c2ee3c05ec6818308e79394f9504da025" gracePeriod=30
Oct 30 13:07:47.370823 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:47.370794    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:47.562254 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:47.560783    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:47.709865 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:47.709738    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt" podUID="13c0f5bf-8735-4702-8d62-86810366b1f9" containerName="payload-processing" containerID="cri-o://321d8b8100146eca43e97a75d3834cebd0aa8fa663e26b35b880f5cd9760f966" gracePeriod=25
Oct 30 13:07:47.761877 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:47.756315    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx" podUID="cadd0f75-a378-43fa-8bdd-3182071fa2c1" containerName="payload-pre-processing" containerID="cri-o://dd0ea9f05ca907bbb41a9b2b44ae3b3c47dca66a6d4c3eae785801d1e36f0408" gracePeriod=25
Oct 30 13:07:48.090152 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.090128    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt"
Oct 30 13:07:48.147768 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.145934    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:48.165085 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.165057    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx"
Oct 30 13:07:48.196927 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.196893    4544 generic.go:356] "Generic (PLEG): container finished" podID="13c0f5bf-8735-4702-8d62-86810366b1f9" containerID="321d8b8100146eca43e97a75d3834cebd0aa8fa663e26b35b880f5cd9760f966" exitCode=0
Oct 30 13:07:48.197135 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.196969    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt"
Oct 30 13:07:48.197135 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.196988    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt" event={"ID":"13c0f5bf-8735-4702-8d62-86810366b1f9","Type":"ContainerDied","Data":"321d8b8100146eca43e97a75d3834cebd0aa8fa663e26b35b880f5cd9760f966"}
Oct 30 13:07:48.197135 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.197085    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt" event={"ID":"13c0f5bf-8735-4702-8d62-86810366b1f9","Type":"ContainerDied","Data":"0d9e36c82fb3ead8f467a340a78652aa1ef523111f71bd11b718698371ea57c8"}
Oct 30 13:07:48.197135 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.197115    4544 scope.go:122] "RemoveContainer" containerID="321d8b8100146eca43e97a75d3834cebd0aa8fa663e26b35b880f5cd9760f966"
Oct 30 13:07:48.199802 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.199774    4544 generic.go:356] "Generic (PLEG): container finished" podID="cadd0f75-a378-43fa-8bdd-3182071fa2c1" containerID="dd0ea9f05ca907bbb41a9b2b44ae3b3c47dca66a6d4c3eae785801d1e36f0408" exitCode=0
Oct 30 13:07:48.199910 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.199856    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx" event={"ID":"cadd0f75-a378-43fa-8bdd-3182071fa2c1","Type":"ContainerDied","Data":"dd0ea9f05ca907bbb41a9b2b44ae3b3c47dca66a6d4c3eae785801d1e36f0408"}
Oct 30 13:07:48.199960 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.199906    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx" event={"ID":"cadd0f75-a378-43fa-8bdd-3182071fa2c1","Type":"ContainerDied","Data":"ebae8a1eb9c428fdd0f01f3d434e0b3f8541bda70d684a59d7bb6f2c64ac3afe"}
Oct 30 13:07:48.200003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.199980    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx"
Oct 30 13:07:48.209165 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.209143    4544 scope.go:122] "RemoveContainer" containerID="321d8b8100146eca43e97a75d3834cebd0aa8fa663e26b35b880f5cd9760f966"
Oct 30 13:07:48.209712 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:07:48.209685    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"321d8b8100146eca43e97a75d3834cebd0aa8fa663e26b35b880f5cd9760f966\": container with ID starting with 321d8b8100146eca43e97a75d3834cebd0aa8fa663e26b35b880f5cd9760f966 not found: ID does not exist" containerID="321d8b8100146eca43e97a75d3834cebd0aa8fa663e26b35b880f5cd9760f966"
Oct 30 13:07:48.209826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.209725    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"321d8b8100146eca43e97a75d3834cebd0aa8fa663e26b35b880f5cd9760f966"} err="failed to get container status \"321d8b8100146eca43e97a75d3834cebd0aa8fa663e26b35b880f5cd9760f966\": rpc error: code = NotFound desc = could not find container \"321d8b8100146eca43e97a75d3834cebd0aa8fa663e26b35b880f5cd9760f966\": container with ID starting with 321d8b8100146eca43e97a75d3834cebd0aa8fa663e26b35b880f5cd9760f966 not found: ID does not exist"
Oct 30 13:07:48.209826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.209749    4544 scope.go:122] "RemoveContainer" containerID="dd0ea9f05ca907bbb41a9b2b44ae3b3c47dca66a6d4c3eae785801d1e36f0408"
Oct 30 13:07:48.223830 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.223475    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cadd0f75-a378-43fa-8bdd-3182071fa2c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cadd0f75-a378-43fa-8bdd-3182071fa2c1-plugins-config-volume\") pod \"cadd0f75-a378-43fa-8bdd-3182071fa2c1\" (UID: \"cadd0f75-a378-43fa-8bdd-3182071fa2c1\") "
Oct 30 13:07:48.223830 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.223561    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cadd0f75-a378-43fa-8bdd-3182071fa2c1-kube-api-access-wqzh7\" (UniqueName: \"kubernetes.io/projected/cadd0f75-a378-43fa-8bdd-3182071fa2c1-kube-api-access-wqzh7\") pod \"cadd0f75-a378-43fa-8bdd-3182071fa2c1\" (UID: \"cadd0f75-a378-43fa-8bdd-3182071fa2c1\") "
Oct 30 13:07:48.223830 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.223650    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13c0f5bf-8735-4702-8d62-86810366b1f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13c0f5bf-8735-4702-8d62-86810366b1f9-plugins-config-volume\") pod \"13c0f5bf-8735-4702-8d62-86810366b1f9\" (UID: \"13c0f5bf-8735-4702-8d62-86810366b1f9\") "
Oct 30 13:07:48.223830 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.223737    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13c0f5bf-8735-4702-8d62-86810366b1f9-kube-api-access-l7tnh\" (UniqueName: \"kubernetes.io/projected/13c0f5bf-8735-4702-8d62-86810366b1f9-kube-api-access-l7tnh\") pod \"13c0f5bf-8735-4702-8d62-86810366b1f9\" (UID: \"13c0f5bf-8735-4702-8d62-86810366b1f9\") "
Oct 30 13:07:48.223830 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.223780    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cadd0f75-a378-43fa-8bdd-3182071fa2c1-plugins-config-volume" pod "cadd0f75-a378-43fa-8bdd-3182071fa2c1" (UID: "cadd0f75-a378-43fa-8bdd-3182071fa2c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:07:48.224164 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.224146    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13c0f5bf-8735-4702-8d62-86810366b1f9-plugins-config-volume" pod "13c0f5bf-8735-4702-8d62-86810366b1f9" (UID: "13c0f5bf-8735-4702-8d62-86810366b1f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:07:48.224854 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.224713    4544 scope.go:122] "RemoveContainer" containerID="dd0ea9f05ca907bbb41a9b2b44ae3b3c47dca66a6d4c3eae785801d1e36f0408"
Oct 30 13:07:48.224854 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.224834    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cadd0f75-a378-43fa-8bdd-3182071fa2c1-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:07:48.224854 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.224856    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13c0f5bf-8735-4702-8d62-86810366b1f9-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:07:48.225209 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:07:48.225188    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd0ea9f05ca907bbb41a9b2b44ae3b3c47dca66a6d4c3eae785801d1e36f0408\": container with ID starting with dd0ea9f05ca907bbb41a9b2b44ae3b3c47dca66a6d4c3eae785801d1e36f0408 not found: ID does not exist" containerID="dd0ea9f05ca907bbb41a9b2b44ae3b3c47dca66a6d4c3eae785801d1e36f0408"
Oct 30 13:07:48.225363 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.225333    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd0ea9f05ca907bbb41a9b2b44ae3b3c47dca66a6d4c3eae785801d1e36f0408"} err="failed to get container status \"dd0ea9f05ca907bbb41a9b2b44ae3b3c47dca66a6d4c3eae785801d1e36f0408\": rpc error: code = NotFound desc = could not find container \"dd0ea9f05ca907bbb41a9b2b44ae3b3c47dca66a6d4c3eae785801d1e36f0408\": container with ID starting with dd0ea9f05ca907bbb41a9b2b44ae3b3c47dca66a6d4c3eae785801d1e36f0408 not found: ID does not exist"
Oct 30 13:07:48.226280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.226213    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cadd0f75-a378-43fa-8bdd-3182071fa2c1-kube-api-access-wqzh7" pod "cadd0f75-a378-43fa-8bdd-3182071fa2c1" (UID: "cadd0f75-a378-43fa-8bdd-3182071fa2c1"). InnerVolumeSpecName "kube-api-access-wqzh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:48.226587 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.226570    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13c0f5bf-8735-4702-8d62-86810366b1f9-kube-api-access-l7tnh" pod "13c0f5bf-8735-4702-8d62-86810366b1f9" (UID: "13c0f5bf-8735-4702-8d62-86810366b1f9"). InnerVolumeSpecName "kube-api-access-l7tnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:48.326092 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.326057    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7tnh\" (UniqueName: \"kubernetes.io/projected/13c0f5bf-8735-4702-8d62-86810366b1f9-kube-api-access-l7tnh\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:07:48.326092 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.326088    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqzh7\" (UniqueName: \"kubernetes.io/projected/cadd0f75-a378-43fa-8bdd-3182071fa2c1-kube-api-access-wqzh7\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:07:48.549536 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.549502    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt"]
Oct 30 13:07:48.560653 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.560624    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt"]
Oct 30 13:07:48.607975 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.607899    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx"]
Oct 30 13:07:48.623344 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.623315    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx"]
Oct 30 13:07:48.706169 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:07:48.706130    4544 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456ae1e2_ef5b_427c_a4a3_fba864a91b55.slice/crio-ad709b24144d870fcafc47d97dd53fb0d9f0d84a71d7824bcefb62f47a8c89e4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456ae1e2_ef5b_427c_a4a3_fba864a91b55.slice/crio-conmon-ad709b24144d870fcafc47d97dd53fb0d9f0d84a71d7824bcefb62f47a8c89e4.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 13:07:48.709794 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.709764    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"]
Oct 30 13:07:48.710638 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.710597    4544 cpu_manager.go:414] "containerMap: removing container" podUID="cadd0f75-a378-43fa-8bdd-3182071fa2c1" containerName="payload-pre-processing"
Oct 30 13:07:48.710638 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.710636    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cadd0f75-a378-43fa-8bdd-3182071fa2c1" containerName="payload-pre-processing"
Oct 30 13:07:48.710840 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.710659    4544 cpu_manager.go:414] "containerMap: removing container" podUID="13c0f5bf-8735-4702-8d62-86810366b1f9" containerName="payload-processing"
Oct 30 13:07:48.710840 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.710667    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13c0f5bf-8735-4702-8d62-86810366b1f9" containerName="payload-processing"
Oct 30 13:07:48.710840 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.710707    4544 cpu_manager.go:414] "containerMap: removing container" podUID="4485c1ce-c862-400b-9bfa-79b1a745dcd9" containerName="maas-api"
Oct 30 13:07:48.710840 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.710713    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4485c1ce-c862-400b-9bfa-79b1a745dcd9" containerName="maas-api"
Oct 30 13:07:48.710987 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.710868    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="4485c1ce-c862-400b-9bfa-79b1a745dcd9" containerName="maas-api"
Oct 30 13:07:48.710987 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.710885    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="cadd0f75-a378-43fa-8bdd-3182071fa2c1" containerName="payload-pre-processing"
Oct 30 13:07:48.710987 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.710895    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="13c0f5bf-8735-4702-8d62-86810366b1f9" containerName="payload-processing"
Oct 30 13:07:48.716272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.716250    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:07:48.726816 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.726763    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:48.769280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.756740    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-bcd76c-openshift-default-dockercfg-znm5v\""
Oct 30 13:07:48.787059 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.780647    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-credential-socket\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:07:48.787059 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.780707    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istio-envoy\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:07:48.787059 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.780744    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-workload-socket\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:07:48.787059 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.780885    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istio-token\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:07:48.787059 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.780992    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-workload-certs\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:07:48.787059 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.781198    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istiod-ca-cert\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:07:48.787059 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.781261    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istio-podinfo\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:07:48.787059 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.781495    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istio-data\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:07:48.787059 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.781539    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlm9d\" (UniqueName: \"kubernetes.io/projected/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-kube-api-access-vlm9d\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:07:48.905301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.896784    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-workload-certs\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:07:48.905301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.896953    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istiod-ca-cert\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:07:48.905301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.896990    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istio-podinfo\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:07:48.905301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.897064    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istio-data\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:07:48.905301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.897088    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlm9d\" (UniqueName: \"kubernetes.io/projected/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-kube-api-access-vlm9d\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:07:48.905301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.897166    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-credential-socket\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:07:48.905301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.897203    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istio-envoy\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:07:48.953243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.897257    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-workload-socket\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:07:48.953243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.918057    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istio-data\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:07:48.953243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.918313    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-credential-socket\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:07:48.953243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.918537    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-workload-certs\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:07:48.953243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.920039    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istiod-ca-cert\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:07:48.953243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.922555    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-workload-socket\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:07:48.953243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:48.933095    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istio-token\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:07:49.029484 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:49.010495    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"]
Oct 30 13:07:49.137343 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:49.130553    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13c0f5bf-8735-4702-8d62-86810366b1f9" path="/var/lib/kubelet/pods/13c0f5bf-8735-4702-8d62-86810366b1f9/volumes"
Oct 30 13:07:49.137343 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:49.131424    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4485c1ce-c862-400b-9bfa-79b1a745dcd9" path="/var/lib/kubelet/pods/4485c1ce-c862-400b-9bfa-79b1a745dcd9/volumes"
Oct 30 13:07:49.137343 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:49.132168    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cadd0f75-a378-43fa-8bdd-3182071fa2c1" path="/var/lib/kubelet/pods/cadd0f75-a378-43fa-8bdd-3182071fa2c1/volumes"
Oct 30 13:07:49.137343 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:49.133459    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3031e4f-f22e-471e-aba2-4bee2f6b4a82" path="/var/lib/kubelet/pods/f3031e4f-f22e-471e-aba2-4bee2f6b4a82/volumes"
Oct 30 13:07:49.137343 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:49.135527    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istio-podinfo\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:07:49.138366 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:49.138306    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istio-envoy\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:07:49.138686 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:49.138664    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlm9d\" (UniqueName: \"kubernetes.io/projected/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-kube-api-access-vlm9d\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:07:49.142736 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:49.142686    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istio-token\") pod \"e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") " pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:07:49.278377 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:49.278287    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:49.378657 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:49.378609    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:07:49.720257 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:07:49.720205    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode308ff03_37ae_4b7f_b27e_05b2f46ce66c.slice/crio-1f9e04a17caeb8e61d9e42180fa4636b2b7b5134ad453d9e11fad111510649bc WatchSource:0}: Error finding container 1f9e04a17caeb8e61d9e42180fa4636b2b7b5134ad453d9e11fad111510649bc: Status 404 returned error can't find the container with id 1f9e04a17caeb8e61d9e42180fa4636b2b7b5134ad453d9e11fad111510649bc
Oct 30 13:07:49.720634 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:49.720601    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"]
Oct 30 13:07:49.723422 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:49.723238    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:07:49.724048 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:49.724020    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:07:49.724241 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:49.724206    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:07:49.754619 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:49.749368    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:50.292505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:50.292471    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs" event={"ID":"e308ff03-37ae-4b7f-b27e-05b2f46ce66c","Type":"ContainerStarted","Data":"804ab07a9a2dd4eeca0ccd71e14f562405ef85199228a5aeb185b2b29ec62402"}
Oct 30 13:07:50.292881 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:50.292529    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs" event={"ID":"e308ff03-37ae-4b7f-b27e-05b2f46ce66c","Type":"ContainerStarted","Data":"1f9e04a17caeb8e61d9e42180fa4636b2b7b5134ad453d9e11fad111510649bc"}
Oct 30 13:07:50.357754 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:50.357697    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs" podStartSLOduration=2.357681283 podStartE2EDuration="2.357681283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:50.352450807 +0000 UTC m=+1365.399584762" watchObservedRunningTime="2026-10-30 13:07:50.357681283 +0000 UTC m=+1365.404815298"
Oct 30 13:07:50.379530 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:50.379485    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:07:50.384786 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:50.384757    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:07:51.301569 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:51.301535    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:07:51.302529 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:51.302510    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:07:51.683920 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:51.683880    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:51.729256 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:07:51.728397    4544 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456ae1e2_ef5b_427c_a4a3_fba864a91b55.slice/crio-conmon-ad709b24144d870fcafc47d97dd53fb0d9f0d84a71d7824bcefb62f47a8c89e4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456ae1e2_ef5b_427c_a4a3_fba864a91b55.slice/crio-ad709b24144d870fcafc47d97dd53fb0d9f0d84a71d7824bcefb62f47a8c89e4.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 13:07:51.795280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:51.795173    4544 patch_prober.go:29] interesting pod/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:07:51.795280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:51.795249    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx" podUID="f7df19fe-9b2b-4e96-a6d2-e1d5015c931f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:07:52.661064 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:52.661029    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:07:52.746828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:52.746790    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-kube-api-access-kjtrv\" (UniqueName: \"kubernetes.io/projected/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-kube-api-access-kjtrv\") pod \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") "
Oct 30 13:07:52.747082 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:52.746880    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-workload-certs\") pod \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") "
Oct 30 13:07:52.747082 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:52.746930    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istio-podinfo\") pod \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") "
Oct 30 13:07:52.747082 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:52.746979    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istio-token\" (UniqueName: \"kubernetes.io/projected/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istio-token\") pod \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") "
Oct 30 13:07:52.747082 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:52.747051    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istiod-ca-cert\") pod \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") "
Oct 30 13:07:52.747358 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:52.747108    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istio-envoy\") pod \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") "
Oct 30 13:07:52.747358 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:52.747141    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istio-data\") pod \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") "
Oct 30 13:07:52.747358 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:52.747303    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-workload-socket\") pod \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") "
Oct 30 13:07:52.748021 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:52.747991    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-workload-certs" pod "f7df19fe-9b2b-4e96-a6d2-e1d5015c931f" (UID: "f7df19fe-9b2b-4e96-a6d2-e1d5015c931f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:52.748262 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:52.748239    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:07:52.748367 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:52.748343    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istiod-ca-cert" pod "f7df19fe-9b2b-4e96-a6d2-e1d5015c931f" (UID: "f7df19fe-9b2b-4e96-a6d2-e1d5015c931f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:07:52.748518 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:52.748494    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istio-data" pod "f7df19fe-9b2b-4e96-a6d2-e1d5015c931f" (UID: "f7df19fe-9b2b-4e96-a6d2-e1d5015c931f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:52.748649 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:52.748620    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-workload-socket" pod "f7df19fe-9b2b-4e96-a6d2-e1d5015c931f" (UID: "f7df19fe-9b2b-4e96-a6d2-e1d5015c931f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:52.751671 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:52.750635    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istio-podinfo" pod "f7df19fe-9b2b-4e96-a6d2-e1d5015c931f" (UID: "f7df19fe-9b2b-4e96-a6d2-e1d5015c931f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:07:52.752618 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:52.752590    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istio-envoy" pod "f7df19fe-9b2b-4e96-a6d2-e1d5015c931f" (UID: "f7df19fe-9b2b-4e96-a6d2-e1d5015c931f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:52.752743 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:52.752721    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-kube-api-access-kjtrv" pod "f7df19fe-9b2b-4e96-a6d2-e1d5015c931f" (UID: "f7df19fe-9b2b-4e96-a6d2-e1d5015c931f"). InnerVolumeSpecName "kube-api-access-kjtrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:52.752810 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:52.752754    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istio-token" pod "f7df19fe-9b2b-4e96-a6d2-e1d5015c931f" (UID: "f7df19fe-9b2b-4e96-a6d2-e1d5015c931f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:52.849131 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:52.849098    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-credential-socket\") pod \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\" (UID: \"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f\") "
Oct 30 13:07:52.849402 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:52.849379    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-credential-socket" pod "f7df19fe-9b2b-4e96-a6d2-e1d5015c931f" (UID: "f7df19fe-9b2b-4e96-a6d2-e1d5015c931f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:52.849709 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:52.849695    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:07:52.849755 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:52.849713    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjtrv\" (UniqueName: \"kubernetes.io/projected/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-kube-api-access-kjtrv\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:07:52.849755 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:52.849725    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:07:52.849755 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:52.849734    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:07:52.849755 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:52.849741    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:07:52.849755 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:52.849749    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:07:52.849915 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:52.849761    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:07:52.849915 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:52.849769    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:07:53.032767 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:53.032663    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:53.293577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:53.293451    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:53.320403 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:53.320372    4544 generic.go:356] "Generic (PLEG): container finished" podID="f7df19fe-9b2b-4e96-a6d2-e1d5015c931f" containerID="4c144566ea356b79c48949ac7032ba4c2ee3c05ec6818308e79394f9504da025" exitCode=0
Oct 30 13:07:53.320555 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:53.320428    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"
Oct 30 13:07:53.320555 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:53.320452    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx" event={"ID":"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f","Type":"ContainerDied","Data":"4c144566ea356b79c48949ac7032ba4c2ee3c05ec6818308e79394f9504da025"}
Oct 30 13:07:53.320555 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:53.320511    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx" event={"ID":"f7df19fe-9b2b-4e96-a6d2-e1d5015c931f","Type":"ContainerDied","Data":"ad585926ec6fc89caa9701b197185717178ec91caa1149771c9b02ac3cc8d4f6"}
Oct 30 13:07:53.320555 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:53.320540    4544 scope.go:122] "RemoveContainer" containerID="4c144566ea356b79c48949ac7032ba4c2ee3c05ec6818308e79394f9504da025"
Oct 30 13:07:53.332003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:53.331983    4544 scope.go:122] "RemoveContainer" containerID="11cf2aa8460bc44631b9a50a36c260adcfafc7702591893ffccd29d607a766d6"
Oct 30 13:07:53.342734 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:53.342713    4544 scope.go:122] "RemoveContainer" containerID="4c144566ea356b79c48949ac7032ba4c2ee3c05ec6818308e79394f9504da025"
Oct 30 13:07:53.343048 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:07:53.343026    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c144566ea356b79c48949ac7032ba4c2ee3c05ec6818308e79394f9504da025\": container with ID starting with 4c144566ea356b79c48949ac7032ba4c2ee3c05ec6818308e79394f9504da025 not found: ID does not exist" containerID="4c144566ea356b79c48949ac7032ba4c2ee3c05ec6818308e79394f9504da025"
Oct 30 13:07:53.343108 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:53.343057    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c144566ea356b79c48949ac7032ba4c2ee3c05ec6818308e79394f9504da025"} err="failed to get container status \"4c144566ea356b79c48949ac7032ba4c2ee3c05ec6818308e79394f9504da025\": rpc error: code = NotFound desc = could not find container \"4c144566ea356b79c48949ac7032ba4c2ee3c05ec6818308e79394f9504da025\": container with ID starting with 4c144566ea356b79c48949ac7032ba4c2ee3c05ec6818308e79394f9504da025 not found: ID does not exist"
Oct 30 13:07:53.343108 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:53.343080    4544 scope.go:122] "RemoveContainer" containerID="11cf2aa8460bc44631b9a50a36c260adcfafc7702591893ffccd29d607a766d6"
Oct 30 13:07:53.343359 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:07:53.343338    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11cf2aa8460bc44631b9a50a36c260adcfafc7702591893ffccd29d607a766d6\": container with ID starting with 11cf2aa8460bc44631b9a50a36c260adcfafc7702591893ffccd29d607a766d6 not found: ID does not exist" containerID="11cf2aa8460bc44631b9a50a36c260adcfafc7702591893ffccd29d607a766d6"
Oct 30 13:07:53.343422 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:53.343366    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11cf2aa8460bc44631b9a50a36c260adcfafc7702591893ffccd29d607a766d6"} err="failed to get container status \"11cf2aa8460bc44631b9a50a36c260adcfafc7702591893ffccd29d607a766d6\": rpc error: code = NotFound desc = could not find container \"11cf2aa8460bc44631b9a50a36c260adcfafc7702591893ffccd29d607a766d6\": container with ID starting with 11cf2aa8460bc44631b9a50a36c260adcfafc7702591893ffccd29d607a766d6 not found: ID does not exist"
Oct 30 13:07:53.406544 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:53.406509    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"]
Oct 30 13:07:53.417066 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:53.417032    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx"]
Oct 30 13:07:53.957282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:53.957085    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:54.167894 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:54.167850    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:54.333372 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:54.333339    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:54.967063 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:54.967033    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:55.059253 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:55.059196    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88bd852c-9kk2r"]
Oct 30 13:07:55.067943 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:55.067902    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88bd852c-9kk2r"]
Oct 30 13:07:55.088194 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:55.088155    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b89906d0-bc2c-48ca-bb49-4a8cd45d2f85" path="/var/lib/kubelet/pods/b89906d0-bc2c-48ca-bb49-4a8cd45d2f85/volumes"
Oct 30 13:07:55.088860 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:55.088836    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7df19fe-9b2b-4e96-a6d2-e1d5015c931f" path="/var/lib/kubelet/pods/f7df19fe-9b2b-4e96-a6d2-e1d5015c931f/volumes"
Oct 30 13:07:55.142700 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:55.142667    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7"]
Oct 30 13:07:55.143395 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:55.143380    4544 cpu_manager.go:414] "containerMap: removing container" podUID="f7df19fe-9b2b-4e96-a6d2-e1d5015c931f" containerName="istio-proxy"
Oct 30 13:07:55.143439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:55.143397    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7df19fe-9b2b-4e96-a6d2-e1d5015c931f" containerName="istio-proxy"
Oct 30 13:07:55.143543 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:55.143533    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7df19fe-9b2b-4e96-a6d2-e1d5015c931f" containerName="istio-proxy"
Oct 30 13:07:55.143582 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:55.143547    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7df19fe-9b2b-4e96-a6d2-e1d5015c931f" containerName="istio-proxy"
Oct 30 13:07:55.147652 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:55.147623    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7"
Oct 30 13:07:55.166146 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:55.166123    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-bcd76c-dockercfg-2gtnb\""
Oct 30 13:07:55.166146 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:55.166131    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-bcd76c\""
Oct 30 13:07:55.172835 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:55.172811    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7"]
Oct 30 13:07:55.174890 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:55.174864    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:55.190911 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:55.190877    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qttj7\" (UniqueName: \"kubernetes.io/projected/2beb5b6e-8222-48ce-b46b-b90e478277ed-kube-api-access-qttj7\") pod \"payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7\" (UID: \"2beb5b6e-8222-48ce-b46b-b90e478277ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7"
Oct 30 13:07:55.191761 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:55.191262    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2beb5b6e-8222-48ce-b46b-b90e478277ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7\" (UID: \"2beb5b6e-8222-48ce-b46b-b90e478277ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7"
Oct 30 13:07:55.313365 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:55.304580    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qttj7\" (UniqueName: \"kubernetes.io/projected/2beb5b6e-8222-48ce-b46b-b90e478277ed-kube-api-access-qttj7\") pod \"payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7\" (UID: \"2beb5b6e-8222-48ce-b46b-b90e478277ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7"
Oct 30 13:07:55.325246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:55.313740    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2beb5b6e-8222-48ce-b46b-b90e478277ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7\" (UID: \"2beb5b6e-8222-48ce-b46b-b90e478277ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7"
Oct 30 13:07:55.325246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:55.314846    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2beb5b6e-8222-48ce-b46b-b90e478277ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7\" (UID: \"2beb5b6e-8222-48ce-b46b-b90e478277ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7"
Oct 30 13:07:55.425246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:55.389563    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p"]
Oct 30 13:07:55.425246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:55.391029    4544 cpu_manager.go:414] "containerMap: removing container" podUID="f7df19fe-9b2b-4e96-a6d2-e1d5015c931f" containerName="istio-proxy"
Oct 30 13:07:55.425246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:55.391055    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7df19fe-9b2b-4e96-a6d2-e1d5015c931f" containerName="istio-proxy"
Oct 30 13:07:55.425246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:55.421615    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qttj7\" (UniqueName: \"kubernetes.io/projected/2beb5b6e-8222-48ce-b46b-b90e478277ed-kube-api-access-qttj7\") pod \"payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7\" (UID: \"2beb5b6e-8222-48ce-b46b-b90e478277ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7"
Oct 30 13:07:55.462633 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:55.462573    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7"
Oct 30 13:07:55.555877 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:55.549780    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p"
Oct 30 13:07:55.647353 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:55.646011    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn9hc\" (UniqueName: \"kubernetes.io/projected/276d8bab-6921-4d6d-a482-258fbcbca9b7-kube-api-access-bn9hc\") pod \"payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p\" (UID: \"276d8bab-6921-4d6d-a482-258fbcbca9b7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p"
Oct 30 13:07:55.647353 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:55.646150    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/276d8bab-6921-4d6d-a482-258fbcbca9b7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p\" (UID: \"276d8bab-6921-4d6d-a482-258fbcbca9b7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p"
Oct 30 13:07:55.647353 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:55.646633    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p"]
Oct 30 13:07:55.786263 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:55.783327    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn9hc\" (UniqueName: \"kubernetes.io/projected/276d8bab-6921-4d6d-a482-258fbcbca9b7-kube-api-access-bn9hc\") pod \"payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p\" (UID: \"276d8bab-6921-4d6d-a482-258fbcbca9b7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p"
Oct 30 13:07:55.786263 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:55.783466    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/276d8bab-6921-4d6d-a482-258fbcbca9b7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p\" (UID: \"276d8bab-6921-4d6d-a482-258fbcbca9b7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p"
Oct 30 13:07:55.801252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:55.792947    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/276d8bab-6921-4d6d-a482-258fbcbca9b7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p\" (UID: \"276d8bab-6921-4d6d-a482-258fbcbca9b7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p"
Oct 30 13:07:55.863452 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:55.846120    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn9hc\" (UniqueName: \"kubernetes.io/projected/276d8bab-6921-4d6d-a482-258fbcbca9b7-kube-api-access-bn9hc\") pod \"payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p\" (UID: \"276d8bab-6921-4d6d-a482-258fbcbca9b7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p"
Oct 30 13:07:56.083357 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:56.082996    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p"
Oct 30 13:07:56.303130 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:07:56.303085    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2beb5b6e_8222_48ce_b46b_b90e478277ed.slice/crio-4ffb28eccab1288450a99416b7b68710b35f33b0eacaa377be9b9e8e1c2d0f7b WatchSource:0}: Error finding container 4ffb28eccab1288450a99416b7b68710b35f33b0eacaa377be9b9e8e1c2d0f7b: Status 404 returned error can't find the container with id 4ffb28eccab1288450a99416b7b68710b35f33b0eacaa377be9b9e8e1c2d0f7b
Oct 30 13:07:56.303454 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:56.303433    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7"]
Oct 30 13:07:56.404770 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:56.404735    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p"]
Oct 30 13:07:56.405374 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:07:56.405350    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod276d8bab_6921_4d6d_a482_258fbcbca9b7.slice/crio-254fdc065f12370202da7902544c5c8fbb22d08a504bec4190ed929d98d4c591 WatchSource:0}: Error finding container 254fdc065f12370202da7902544c5c8fbb22d08a504bec4190ed929d98d4c591: Status 404 returned error can't find the container with id 254fdc065f12370202da7902544c5c8fbb22d08a504bec4190ed929d98d4c591
Oct 30 13:07:56.744301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:56.738856    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:56.806618 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:56.800485    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7" event={"ID":"2beb5b6e-8222-48ce-b46b-b90e478277ed","Type":"ContainerStarted","Data":"56b51e3a0198e871c6f7c6774b900ad7c42b56a5e29064554ce173f8dbecc2ca"}
Oct 30 13:07:56.806618 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:56.800558    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7" event={"ID":"2beb5b6e-8222-48ce-b46b-b90e478277ed","Type":"ContainerStarted","Data":"4ffb28eccab1288450a99416b7b68710b35f33b0eacaa377be9b9e8e1c2d0f7b"}
Oct 30 13:07:56.806618 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:56.806416    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7"
Oct 30 13:07:56.872453 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:56.872412    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p" event={"ID":"276d8bab-6921-4d6d-a482-258fbcbca9b7","Type":"ContainerStarted","Data":"507152f2d87a669374635d0ea2c82180f38d4aff04ada125aff000ca05732f05"}
Oct 30 13:07:56.872774 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:56.872735    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p" event={"ID":"276d8bab-6921-4d6d-a482-258fbcbca9b7","Type":"ContainerStarted","Data":"254fdc065f12370202da7902544c5c8fbb22d08a504bec4190ed929d98d4c591"}
Oct 30 13:07:56.873927 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:56.873904    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p"
Oct 30 13:07:56.943654 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:56.943589    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7" podStartSLOduration=1.9435674029999999 podStartE2EDuration="1.943567403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:56.902025823 +0000 UTC m=+1371.949159777" watchObservedRunningTime="2026-10-30 13:07:56.943567403 +0000 UTC m=+1371.990701357"
Oct 30 13:07:56.955156 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:56.954687    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p" podStartSLOduration=1.954663591 podStartE2EDuration="1.954663591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:56.945411372 +0000 UTC m=+1371.992545325" watchObservedRunningTime="2026-10-30 13:07:56.954663591 +0000 UTC m=+1372.001797545"
Oct 30 13:07:57.158244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:57.155948    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:57.310086 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:57.310035    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:58.733246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:07:58.730989    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:07:58.988147 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:07:58.988057    4544 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456ae1e2_ef5b_427c_a4a3_fba864a91b55.slice/crio-ad709b24144d870fcafc47d97dd53fb0d9f0d84a71d7824bcefb62f47a8c89e4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456ae1e2_ef5b_427c_a4a3_fba864a91b55.slice/crio-conmon-ad709b24144d870fcafc47d97dd53fb0d9f0d84a71d7824bcefb62f47a8c89e4.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 13:07:59.008397 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:07:59.008362    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:08:00.040961 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:00.040926    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2bbc545-9zqtg"]
Oct 30 13:08:00.046184 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:00.046149    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2bbc545-9zqtg"]
Oct 30 13:08:00.750438 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:00.750406    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc"]
Oct 30 13:08:00.756398 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:00.756376    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc"
Oct 30 13:08:00.792806 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:00.791674    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc"]
Oct 30 13:08:00.797163 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:00.797058    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-1c0e6c42-serving-cert\""
Oct 30 13:08:00.799734 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:00.799698    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91d76077-b236-4f06-a6ab-428c729cfe22-metrics-certs\") pod \"maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc\" (UID: \"91d76077-b236-4f06-a6ab-428c729cfe22\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc"
Oct 30 13:08:00.799893 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:00.799869    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d76077-b236-4f06-a6ab-428c729cfe22-openshift-service-ca\") pod \"maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc\" (UID: \"91d76077-b236-4f06-a6ab-428c729cfe22\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc"
Oct 30 13:08:00.799958 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:00.799940    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91d76077-b236-4f06-a6ab-428c729cfe22-maas-api-tls\") pod \"maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc\" (UID: \"91d76077-b236-4f06-a6ab-428c729cfe22\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc"
Oct 30 13:08:00.800013 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:00.799965    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2m8z\" (UniqueName: \"kubernetes.io/projected/91d76077-b236-4f06-a6ab-428c729cfe22-kube-api-access-p2m8z\") pod \"maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc\" (UID: \"91d76077-b236-4f06-a6ab-428c729cfe22\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc"
Oct 30 13:08:00.876894 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:00.876827    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:00.901447 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:00.901411    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91d76077-b236-4f06-a6ab-428c729cfe22-metrics-certs\") pod \"maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc\" (UID: \"91d76077-b236-4f06-a6ab-428c729cfe22\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc"
Oct 30 13:08:00.901617 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:00.901579    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d76077-b236-4f06-a6ab-428c729cfe22-openshift-service-ca\") pod \"maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc\" (UID: \"91d76077-b236-4f06-a6ab-428c729cfe22\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc"
Oct 30 13:08:00.901672 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:00.901660    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91d76077-b236-4f06-a6ab-428c729cfe22-maas-api-tls\") pod \"maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc\" (UID: \"91d76077-b236-4f06-a6ab-428c729cfe22\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc"
Oct 30 13:08:00.901714 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:00.901694    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2m8z\" (UniqueName: \"kubernetes.io/projected/91d76077-b236-4f06-a6ab-428c729cfe22-kube-api-access-p2m8z\") pod \"maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc\" (UID: \"91d76077-b236-4f06-a6ab-428c729cfe22\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc"
Oct 30 13:08:00.902336 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:08:00.902306    4544 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-serving-cert: secret "maas-api-e2e-mt-1c0e6c42-serving-cert" not found
Oct 30 13:08:00.902336 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:00.902330    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d76077-b236-4f06-a6ab-428c729cfe22-openshift-service-ca\") pod \"maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc\" (UID: \"91d76077-b236-4f06-a6ab-428c729cfe22\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc"
Oct 30 13:08:00.902493 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:08:00.902381    4544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91d76077-b236-4f06-a6ab-428c729cfe22-maas-api-tls podName:91d76077-b236-4f06-a6ab-428c729cfe22 nodeName:}" failed. No retries permitted until 2026-10-30 13:08:01.402361312 +0000 UTC m=+1376.449495264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/91d76077-b236-4f06-a6ab-428c729cfe22-maas-api-tls") pod "maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc" (UID: "91d76077-b236-4f06-a6ab-428c729cfe22") : secret "maas-api-e2e-mt-1c0e6c42-serving-cert" not found
Oct 30 13:08:00.904040 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:00.904012    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91d76077-b236-4f06-a6ab-428c729cfe22-metrics-certs\") pod \"maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc\" (UID: \"91d76077-b236-4f06-a6ab-428c729cfe22\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc"
Oct 30 13:08:00.949268 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:00.949233    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2m8z\" (UniqueName: \"kubernetes.io/projected/91d76077-b236-4f06-a6ab-428c729cfe22-kube-api-access-p2m8z\") pod \"maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc\" (UID: \"91d76077-b236-4f06-a6ab-428c729cfe22\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc"
Oct 30 13:08:01.235247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:01.214598    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80094f69-bff5-4ccd-8849-4af37f390d86" path="/var/lib/kubelet/pods/80094f69-bff5-4ccd-8849-4af37f390d86/volumes"
Oct 30 13:08:01.473363 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:01.473321    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91d76077-b236-4f06-a6ab-428c729cfe22-maas-api-tls\") pod \"maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc\" (UID: \"91d76077-b236-4f06-a6ab-428c729cfe22\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc"
Oct 30 13:08:01.475880 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:01.475851    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91d76077-b236-4f06-a6ab-428c729cfe22-maas-api-tls\") pod \"maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc\" (UID: \"91d76077-b236-4f06-a6ab-428c729cfe22\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc"
Oct 30 13:08:01.675162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:01.672710    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc"
Oct 30 13:08:02.098323 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:02.098290    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc"]
Oct 30 13:08:02.098907 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:08:02.098880    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91d76077_b236_4f06_a6ab_428c729cfe22.slice/crio-d699124d38d7a96b15295764a76e6a7426008f811f32bf6da756e73eb2f6d6e1 WatchSource:0}: Error finding container d699124d38d7a96b15295764a76e6a7426008f811f32bf6da756e73eb2f6d6e1: Status 404 returned error can't find the container with id d699124d38d7a96b15295764a76e6a7426008f811f32bf6da756e73eb2f6d6e1
Oct 30 13:08:02.179539 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:02.179500    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:02.380301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:02.377683    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:02.656783 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:02.656693    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:02.779768 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:02.779735    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:02.938992 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:02.938898    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:03.169200 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:03.169168    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc"
Oct 30 13:08:03.169415 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:03.169273    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc" event={"ID":"91d76077-b236-4f06-a6ab-428c729cfe22","Type":"ContainerStarted","Data":"1ab85eb6440b4fba06fe87f118720df43de3d46e9db4eff2b5795352e36419e2"}
Oct 30 13:08:03.169415 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:03.169307    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc" event={"ID":"91d76077-b236-4f06-a6ab-428c729cfe22","Type":"ContainerStarted","Data":"d699124d38d7a96b15295764a76e6a7426008f811f32bf6da756e73eb2f6d6e1"}
Oct 30 13:08:04.918738 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:04.918695    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:06.213401 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:06.213365    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:08:06.213890 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:06.213501    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q" podUID="75391895-2b05-426d-98ad-87ad85d2e9c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:07.081846 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:07.081796    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:07.669538 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:07.669449    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:07.884927 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:07.884893    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7"
Oct 30 13:08:07.928028 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:07.927974    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc" podStartSLOduration=7.528587807 podStartE2EDuration="7.92795911s" totalImagesPullingTime="399.371303ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:03.238963098 +0000 UTC m=+1378.286097052" watchObservedRunningTime="2026-10-30 13:08:07.92795911 +0000 UTC m=+1382.975093063"
Oct 30 13:08:08.142594 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:08.142502    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:08.228188 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:08.228153    4544 generic.go:356] "Generic (PLEG): container finished" podID="75391895-2b05-426d-98ad-87ad85d2e9c3" containerID="4d2447981a522cbf7003cb6702442c0af55f422dd1521822112c515e9b06111c" exitCode=0
Oct 30 13:08:08.228382 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:08.228237    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q" event={"ID":"75391895-2b05-426d-98ad-87ad85d2e9c3","Type":"ContainerDied","Data":"4d2447981a522cbf7003cb6702442c0af55f422dd1521822112c515e9b06111c"}
Oct 30 13:08:08.228750 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:08.228724    4544 scope.go:122] "RemoveContainer" containerID="4d2447981a522cbf7003cb6702442c0af55f422dd1521822112c515e9b06111c"
Oct 30 13:08:08.230541 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:08.230508    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:08:08.230629 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:08.230587    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:08:08.230629 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:08.230623    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:08:08.397298 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:08.394723    4544 scope.go:122] "RemoveContainer" containerID="b17e27dc77390b0928855b70c377080f1e71655b394ee947531449fcfe2efc6c"
Oct 30 13:08:08.420377 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:08.420352    4544 scope.go:122] "RemoveContainer" containerID="355b209fed7301cb190c5b9a492114a7e5b65075a0385d3cb6cf443047e23c97"
Oct 30 13:08:08.431308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:08.431284    4544 scope.go:122] "RemoveContainer" containerID="5bcceaf30c078cd17436b2de6eac5be36c038837ba56b7384464c46a45330cef"
Oct 30 13:08:08.442834 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:08.442808    4544 scope.go:122] "RemoveContainer" containerID="4bfcf27bf6e968cd62b1f148ad88b712ce418ed82499a5ece8489faa9079eec5"
Oct 30 13:08:08.456794 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:08.456765    4544 scope.go:122] "RemoveContainer" containerID="b5fe2a3154b88fa34b8b96a317061d1ebb777983d29b4d718cf9b816013b09d0"
Oct 30 13:08:08.472529 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:08.472387    4544 scope.go:122] "RemoveContainer" containerID="3ee9945efefdf83761fc35608def7457367fccb932cde9cbf68e666bf9b9080b"
Oct 30 13:08:08.936280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:08.932978    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:09.358816 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:09.358781    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23283e03-wgfxf"]
Oct 30 13:08:09.359042 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:09.358845    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q" event={"ID":"75391895-2b05-426d-98ad-87ad85d2e9c3","Type":"ContainerStarted","Data":"afa41dd1dabc3afefd67cdaf51460fce13ca39abb7b80869946180bd527ddedb"}
Oct 30 13:08:09.359042 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:09.358912    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23283e03-wgfxf"]
Oct 30 13:08:09.359042 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:09.359000    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p"
Oct 30 13:08:09.359212 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:09.359070    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc"
Oct 30 13:08:09.493252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:09.483336    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:09.643426 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:09.642687    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:08:09.643426 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:09.642894    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 13:08:09.643426 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:09.642981    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q" podUID="75391895-2b05-426d-98ad-87ad85d2e9c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 13:08:09.787893 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:09.787856    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:10.633398 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:10.633356    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 13:08:10.633926 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:10.633495    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q" podUID="75391895-2b05-426d-98ad-87ad85d2e9c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 13:08:11.042692 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:11.042309    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32ab2b8d-9pf49"]
Oct 30 13:08:11.047702 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:11.047667    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32ab2b8d-9pf49"]
Oct 30 13:08:11.090332 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:11.090286    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6034433c-144c-49b8-a3c8-38d2511e6c18" path="/var/lib/kubelet/pods/6034433c-144c-49b8-a3c8-38d2511e6c18/volumes"
Oct 30 13:08:11.090969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:11.090954    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca24aa78-b130-48be-9da0-f37d84a0fd6d" path="/var/lib/kubelet/pods/ca24aa78-b130-48be-9da0-f37d84a0fd6d/volumes"
Oct 30 13:08:11.633732 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:11.633698    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 13:08:11.634638 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:11.634606    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q" podUID="75391895-2b05-426d-98ad-87ad85d2e9c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 13:08:11.776709 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:11.776667    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:12.635244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:12.634320    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 13:08:12.635244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:12.634401    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q" podUID="75391895-2b05-426d-98ad-87ad85d2e9c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 13:08:12.801790 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:12.801755    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:13.006997 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:13.006888    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:13.138156 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:13.138117    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:13.593570 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:13.593520    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:13.636246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:13.635268    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 13:08:13.636246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:13.635351    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q" podUID="75391895-2b05-426d-98ad-87ad85d2e9c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 13:08:13.866926 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:13.866806    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:13.870257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:13.870212    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb"]
Oct 30 13:08:13.878041 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:13.878015    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb"
Oct 30 13:08:13.904327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:13.904017    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb"]
Oct 30 13:08:13.953764 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:13.953719    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-bcd76c-serving-cert\""
Oct 30 13:08:13.968279 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:13.968239    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vptbb\" (UniqueName: \"kubernetes.io/projected/0824f178-4f12-4b6c-8609-4c953e28d039-kube-api-access-vptbb\") pod \"maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb\" (UID: \"0824f178-4f12-4b6c-8609-4c953e28d039\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb"
Oct 30 13:08:13.968462 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:13.968398    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0824f178-4f12-4b6c-8609-4c953e28d039-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb\" (UID: \"0824f178-4f12-4b6c-8609-4c953e28d039\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb"
Oct 30 13:08:13.968539 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:13.968458    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0824f178-4f12-4b6c-8609-4c953e28d039-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb\" (UID: \"0824f178-4f12-4b6c-8609-4c953e28d039\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb"
Oct 30 13:08:13.968594 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:13.968574    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0824f178-4f12-4b6c-8609-4c953e28d039-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb\" (UID: \"0824f178-4f12-4b6c-8609-4c953e28d039\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb"
Oct 30 13:08:13.995815 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:13.995779    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:14.075705 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:14.074391    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0824f178-4f12-4b6c-8609-4c953e28d039-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb\" (UID: \"0824f178-4f12-4b6c-8609-4c953e28d039\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb"
Oct 30 13:08:14.085006 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:14.084959    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0824f178-4f12-4b6c-8609-4c953e28d039-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb\" (UID: \"0824f178-4f12-4b6c-8609-4c953e28d039\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb"
Oct 30 13:08:14.093249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:14.085277    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0824f178-4f12-4b6c-8609-4c953e28d039-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb\" (UID: \"0824f178-4f12-4b6c-8609-4c953e28d039\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb"
Oct 30 13:08:14.093842 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:14.093810    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vptbb\" (UniqueName: \"kubernetes.io/projected/0824f178-4f12-4b6c-8609-4c953e28d039-kube-api-access-vptbb\") pod \"maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb\" (UID: \"0824f178-4f12-4b6c-8609-4c953e28d039\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb"
Oct 30 13:08:14.100258 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:08:14.098718    4544 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-bcd76c-serving-cert: secret "maas-api-e2e-shared-a-w5-bcd76c-serving-cert" not found
Oct 30 13:08:14.100258 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:08:14.098797    4544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0824f178-4f12-4b6c-8609-4c953e28d039-maas-api-tls podName:0824f178-4f12-4b6c-8609-4c953e28d039 nodeName:}" failed. No retries permitted until 2026-10-30 13:08:14.598774323 +0000 UTC m=+1389.645908260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0824f178-4f12-4b6c-8609-4c953e28d039-maas-api-tls") pod "maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb" (UID: "0824f178-4f12-4b6c-8609-4c953e28d039") : secret "maas-api-e2e-shared-a-w5-bcd76c-serving-cert" not found
Oct 30 13:08:14.100258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:14.086769    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0824f178-4f12-4b6c-8609-4c953e28d039-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb\" (UID: \"0824f178-4f12-4b6c-8609-4c953e28d039\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb"
Oct 30 13:08:14.118290 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:14.117894    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0824f178-4f12-4b6c-8609-4c953e28d039-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb\" (UID: \"0824f178-4f12-4b6c-8609-4c953e28d039\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb"
Oct 30 13:08:14.179245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:14.175754    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vptbb\" (UniqueName: \"kubernetes.io/projected/0824f178-4f12-4b6c-8609-4c953e28d039-kube-api-access-vptbb\") pod \"maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb\" (UID: \"0824f178-4f12-4b6c-8609-4c953e28d039\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb"
Oct 30 13:08:14.181172 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:14.181138    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:14.480211 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:14.480119    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:14.632590 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:14.632544    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0824f178-4f12-4b6c-8609-4c953e28d039-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb\" (UID: \"0824f178-4f12-4b6c-8609-4c953e28d039\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb"
Oct 30 13:08:14.634393 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:14.634341    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 13:08:14.634667 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:14.634633    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q" podUID="75391895-2b05-426d-98ad-87ad85d2e9c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 13:08:14.640043 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:14.640015    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0824f178-4f12-4b6c-8609-4c953e28d039-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb\" (UID: \"0824f178-4f12-4b6c-8609-4c953e28d039\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb"
Oct 30 13:08:14.796387 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:14.796297    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb"
Oct 30 13:08:15.142248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:15.142172    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:15.144614 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:15.144591    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb"]
Oct 30 13:08:15.146702 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:08:15.146662    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0824f178_4f12_4b6c_8609_4c953e28d039.slice/crio-011b16e1fd6bc71611b3cc4b8bcb0cf327fda5ef7dabfe73839f66c7b2377472 WatchSource:0}: Error finding container 011b16e1fd6bc71611b3cc4b8bcb0cf327fda5ef7dabfe73839f66c7b2377472: Status 404 returned error can't find the container with id 011b16e1fd6bc71611b3cc4b8bcb0cf327fda5ef7dabfe73839f66c7b2377472
Oct 30 13:08:15.327370 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:15.327328    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:15.399779 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:15.396338    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb"]
Oct 30 13:08:15.410169 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:15.407317    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb"
Oct 30 13:08:15.433246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:15.427619    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-7a71ca\""
Oct 30 13:08:15.473716 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:15.471669    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-7a71ca-dockercfg-q8x7l\""
Oct 30 13:08:15.539247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:15.514640    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb8dl\" (UniqueName: \"kubernetes.io/projected/f673c81d-7be1-4c6c-bd5c-904136739702-kube-api-access-fb8dl\") pod \"payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb\" (UID: \"f673c81d-7be1-4c6c-bd5c-904136739702\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb"
Oct 30 13:08:15.539247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:15.514831    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f673c81d-7be1-4c6c-bd5c-904136739702-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb\" (UID: \"f673c81d-7be1-4c6c-bd5c-904136739702\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb"
Oct 30 13:08:15.611875 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:15.611845    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb"]
Oct 30 13:08:15.766830 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:15.766733    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb8dl\" (UniqueName: \"kubernetes.io/projected/f673c81d-7be1-4c6c-bd5c-904136739702-kube-api-access-fb8dl\") pod \"payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb\" (UID: \"f673c81d-7be1-4c6c-bd5c-904136739702\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb"
Oct 30 13:08:15.767798 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:15.767770    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f673c81d-7be1-4c6c-bd5c-904136739702-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb\" (UID: \"f673c81d-7be1-4c6c-bd5c-904136739702\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb"
Oct 30 13:08:15.769511 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:15.769485    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f673c81d-7be1-4c6c-bd5c-904136739702-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb\" (UID: \"f673c81d-7be1-4c6c-bd5c-904136739702\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb"
Oct 30 13:08:15.892885 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:15.892837    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb" event={"ID":"0824f178-4f12-4b6c-8609-4c953e28d039","Type":"ContainerStarted","Data":"011b16e1fd6bc71611b3cc4b8bcb0cf327fda5ef7dabfe73839f66c7b2377472"}
Oct 30 13:08:15.935247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:15.917697    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:08:15.935247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:15.919386    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:08:15.935247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:15.925624    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:08:15.937749 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:15.937664    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb8dl\" (UniqueName: \"kubernetes.io/projected/f673c81d-7be1-4c6c-bd5c-904136739702-kube-api-access-fb8dl\") pod \"payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb\" (UID: \"f673c81d-7be1-4c6c-bd5c-904136739702\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb"
Oct 30 13:08:16.101920 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:16.100039    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb"
Oct 30 13:08:16.388341 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:16.388305    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:16.454504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:16.454471    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb"]
Oct 30 13:08:16.455875 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:08:16.455843    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf673c81d_7be1_4c6c_bd5c_904136739702.slice/crio-4922763cb77301c035fc53d4672378889758481c771b48d919fc632297c6da21 WatchSource:0}: Error finding container 4922763cb77301c035fc53d4672378889758481c771b48d919fc632297c6da21: Status 404 returned error can't find the container with id 4922763cb77301c035fc53d4672378889758481c771b48d919fc632297c6da21
Oct 30 13:08:16.902738 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:16.902594    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb" event={"ID":"0824f178-4f12-4b6c-8609-4c953e28d039","Type":"ContainerStarted","Data":"a29d9f05976fecb3a4dd4c22e4962a0fae470b1850b962433bd4adcdbaf52428"}
Oct 30 13:08:16.902738 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:16.902709    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb"
Oct 30 13:08:16.904540 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:16.904504    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb" event={"ID":"f673c81d-7be1-4c6c-bd5c-904136739702","Type":"ContainerStarted","Data":"c434a75ca1235f1fb4b4e289578aa4aeb4f96d3713cb50ed2a49ada207c025c6"}
Oct 30 13:08:16.904682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:16.904548    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb" event={"ID":"f673c81d-7be1-4c6c-bd5c-904136739702","Type":"ContainerStarted","Data":"4922763cb77301c035fc53d4672378889758481c771b48d919fc632297c6da21"}
Oct 30 13:08:16.904682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:16.904593    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb"
Oct 30 13:08:16.949209 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:16.949141    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb" podStartSLOduration=3.170260109 podStartE2EDuration="3.949123524s" totalImagesPullingTime="778.863415ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:16.944603722 +0000 UTC m=+1391.991737669" watchObservedRunningTime="2026-10-30 13:08:16.949123524 +0000 UTC m=+1391.996257477"
Oct 30 13:08:16.988658 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:16.988597    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb" podStartSLOduration=1.9885805140000001 podStartE2EDuration="1.988580514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:16.984214756 +0000 UTC m=+1392.031348709" watchObservedRunningTime="2026-10-30 13:08:16.988580514 +0000 UTC m=+1392.035714466"
Oct 30 13:08:18.393036 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:18.393001    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:20.192433 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:20.192387    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:20.681024 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:20.680903    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:20.826246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:20.825356    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:21.357132 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:21.357063    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:22.697256 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:22.696787    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:23.006689 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:23.006607    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb"
Oct 30 13:08:24.088759 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:24.088724    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:24.503964 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:24.503873    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"]
Oct 30 13:08:24.511162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:24.511135    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:08:24.553664 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:24.553628    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-0b2c92-openshift-default-dockercfg-4kbg9\""
Oct 30 13:08:24.583351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:24.583310    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"]
Oct 30 13:08:24.721244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:24.706936    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-workload-socket\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:08:24.731242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:24.721683    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-credential-socket\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:08:24.795904 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:24.794355    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65e7c2e0-231b-4455-b017-995e6691cf31-istiod-ca-cert\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:08:24.819314 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:24.807567    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-istio-envoy\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:08:25.039748 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:25.039714    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65e7c2e0-231b-4455-b017-995e6691cf31-istio-token\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:08:25.040100 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:25.040073    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-workload-certs\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:08:25.040320 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:25.040303    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65e7c2e0-231b-4455-b017-995e6691cf31-istio-podinfo\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:08:25.040591 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:25.040570    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9fgs\" (UniqueName: \"kubernetes.io/projected/65e7c2e0-231b-4455-b017-995e6691cf31-kube-api-access-g9fgs\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:08:25.056303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:25.040784    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-istio-data\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:08:25.056765 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:24.782864    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:25.163924 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:25.163887    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65e7c2e0-231b-4455-b017-995e6691cf31-istiod-ca-cert\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:08:25.164581 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:25.164553    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-istio-envoy\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:08:25.164714 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:25.164699    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65e7c2e0-231b-4455-b017-995e6691cf31-istio-token\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:08:25.164856 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:25.164839    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-workload-certs\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:08:25.165011 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:25.164994    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65e7c2e0-231b-4455-b017-995e6691cf31-istio-podinfo\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:08:25.165353 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:25.165332    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9fgs\" (UniqueName: \"kubernetes.io/projected/65e7c2e0-231b-4455-b017-995e6691cf31-kube-api-access-g9fgs\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:08:25.165570 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:25.165552    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-workload-certs\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:08:25.165696 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:25.165564    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-istio-data\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:08:25.165828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:25.165808    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-istio-data\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:08:25.165933 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:25.165869    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-workload-socket\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:08:25.166123 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:25.166105    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-workload-socket\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:08:25.166248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:25.166171    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-credential-socket\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:08:25.166449 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:25.166432    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-credential-socket\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:08:25.166574 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:25.164695    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65e7c2e0-231b-4455-b017-995e6691cf31-istiod-ca-cert\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:08:25.186849 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:25.177087    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-istio-envoy\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:08:25.186849 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:25.177139    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65e7c2e0-231b-4455-b017-995e6691cf31-istio-podinfo\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:08:25.199931 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:25.199900    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65e7c2e0-231b-4455-b017-995e6691cf31-istio-token\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:08:25.200196 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:25.200171    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9fgs\" (UniqueName: \"kubernetes.io/projected/65e7c2e0-231b-4455-b017-995e6691cf31-kube-api-access-g9fgs\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:08:25.264631 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:25.264588    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:25.470946 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:25.470904    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:08:25.580261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:25.573888    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:25.719278 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:25.715820    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:26.187862 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:26.187827    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"]
Oct 30 13:08:26.188606 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:08:26.188574    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65e7c2e0_231b_4455_b017_995e6691cf31.slice/crio-4565b4874c12002cf54b4d2a97d33cf51e0fa6771ad5a10826526339e9b9698e WatchSource:0}: Error finding container 4565b4874c12002cf54b4d2a97d33cf51e0fa6771ad5a10826526339e9b9698e: Status 404 returned error can't find the container with id 4565b4874c12002cf54b4d2a97d33cf51e0fa6771ad5a10826526339e9b9698e
Oct 30 13:08:26.191715 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:26.191672    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:08:26.191833 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:26.191767    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:08:26.191833 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:26.191819    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:08:26.206556 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:26.206502    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg" event={"ID":"65e7c2e0-231b-4455-b017-995e6691cf31","Type":"ContainerStarted","Data":"4565b4874c12002cf54b4d2a97d33cf51e0fa6771ad5a10826526339e9b9698e"}
Oct 30 13:08:26.943293 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:26.943260    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:27.100827 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:27.100789    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:27.701278 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:27.671286    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl"]
Oct 30 13:08:27.701278 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:27.695874    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl"]
Oct 30 13:08:27.701278 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:27.695938    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg" event={"ID":"65e7c2e0-231b-4455-b017-995e6691cf31","Type":"ContainerStarted","Data":"0ed787d6496b5ec272bc824ffd4aad4e76a5fe8227a8cf4b830090292cbdd1a2"}
Oct 30 13:08:27.701278 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:27.696086    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl"
Oct 30 13:08:27.716291 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:27.713349    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-17fc433e\""
Oct 30 13:08:27.716291 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:27.713553    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-17fc433e-dockercfg-kjcrc\""
Oct 30 13:08:27.759881 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:27.759637    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg" podStartSLOduration=3.759616699 podStartE2EDuration="3.759616699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:27.754652398 +0000 UTC m=+1402.801786363" watchObservedRunningTime="2026-10-30 13:08:27.759616699 +0000 UTC m=+1402.806750654"
Oct 30 13:08:27.913923 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:27.890396    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc29k\" (UniqueName: \"kubernetes.io/projected/eea3f02b-aece-4e51-b7c2-5eb3c54d1dca-kube-api-access-bc29k\") pod \"payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl\" (UID: \"eea3f02b-aece-4e51-b7c2-5eb3c54d1dca\") " pod="openshift-ingress/payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl"
Oct 30 13:08:27.913923 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:27.890490    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea3f02b-aece-4e51-b7c2-5eb3c54d1dca-plugins-config-volume\") pod \"payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl\" (UID: \"eea3f02b-aece-4e51-b7c2-5eb3c54d1dca\") " pod="openshift-ingress/payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl"
Oct 30 13:08:28.000325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:27.992576    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc29k\" (UniqueName: \"kubernetes.io/projected/eea3f02b-aece-4e51-b7c2-5eb3c54d1dca-kube-api-access-bc29k\") pod \"payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl\" (UID: \"eea3f02b-aece-4e51-b7c2-5eb3c54d1dca\") " pod="openshift-ingress/payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl"
Oct 30 13:08:28.000325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:27.992653    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea3f02b-aece-4e51-b7c2-5eb3c54d1dca-plugins-config-volume\") pod \"payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl\" (UID: \"eea3f02b-aece-4e51-b7c2-5eb3c54d1dca\") " pod="openshift-ingress/payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl"
Oct 30 13:08:28.000325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:27.993804    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea3f02b-aece-4e51-b7c2-5eb3c54d1dca-plugins-config-volume\") pod \"payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl\" (UID: \"eea3f02b-aece-4e51-b7c2-5eb3c54d1dca\") " pod="openshift-ingress/payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl"
Oct 30 13:08:28.000325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:27.993817    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:28.037243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:28.033477    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb"
Oct 30 13:08:28.047139 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:28.047062    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc29k\" (UniqueName: \"kubernetes.io/projected/eea3f02b-aece-4e51-b7c2-5eb3c54d1dca-kube-api-access-bc29k\") pod \"payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl\" (UID: \"eea3f02b-aece-4e51-b7c2-5eb3c54d1dca\") " pod="openshift-ingress/payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl"
Oct 30 13:08:28.326306 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:28.323245    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl"
Oct 30 13:08:28.474526 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:28.474459    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:08:28.483619 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:28.483583    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:08:28.722326 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:28.713665    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:08:28.722326 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:28.716209    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:08:28.779368 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:28.776526    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl"]
Oct 30 13:08:29.010471 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:08:29.010386    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:08:29.723039 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:29.723002    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl" event={"ID":"eea3f02b-aece-4e51-b7c2-5eb3c54d1dca","Type":"ContainerStarted","Data":"839b91602c0adb3d5135f332ad05c565516c0b5a9da817b7d1f6725e8ddd6c4b"}
Oct 30 13:08:29.723576 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:29.723060    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl" event={"ID":"eea3f02b-aece-4e51-b7c2-5eb3c54d1dca","Type":"ContainerStarted","Data":"05e2db94f36473f52ee03be650ad3b2da431392943ba90844ca3844adc2efd6e"}
Oct 30 13:08:29.723576 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:29.723267    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl"
Oct 30 13:08:29.764962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:29.764904    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl" podStartSLOduration=2.7648877990000003 podStartE2EDuration="2.764887799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:29.762034359 +0000 UTC m=+1404.809168312" watchObservedRunningTime="2026-10-30 13:08:29.764887799 +0000 UTC m=+1404.812021753"
Oct 30 13:08:30.033328 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:30.033195    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd085d5d-jpfpz"]
Oct 30 13:08:30.038214 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:30.038176    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd085d5d-jpfpz"]
Oct 30 13:08:30.285788 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:30.285703    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:30.453162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:30.453101    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:30.657244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:30.653130    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:31.090476 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:31.090440    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a254746-b5be-4d7c-b6ec-3c749effbebf" path="/var/lib/kubelet/pods/3a254746-b5be-4d7c-b6ec-3c749effbebf/volumes"
Oct 30 13:08:32.099362 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:32.099273    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:32.212240 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:32.212181    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:32.523303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:32.518200    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:32.808938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:32.808855    4544 generic.go:356] "Generic (PLEG): container finished" podID="48119de6-b81b-4f07-b304-a33e16ab2d68" containerID="0ce2a491e46e690ccbac4fa6554b14a117fa4a58145487fe1e086774fa4f2492" exitCode=0
Oct 30 13:08:32.809347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:32.809312    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d" event={"ID":"48119de6-b81b-4f07-b304-a33e16ab2d68","Type":"ContainerDied","Data":"0ce2a491e46e690ccbac4fa6554b14a117fa4a58145487fe1e086774fa4f2492"}
Oct 30 13:08:32.810026 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:32.810006    4544 scope.go:122] "RemoveContainer" containerID="0ce2a491e46e690ccbac4fa6554b14a117fa4a58145487fe1e086774fa4f2492"
Oct 30 13:08:32.814337 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:32.814281    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:08:32.815063 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:32.815035    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:08:32.815255 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:32.815236    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:08:33.045405 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:33.045376    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6e33f02-lm5p7"]
Oct 30 13:08:33.051058 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:33.051027    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6e33f02-lm5p7"]
Oct 30 13:08:33.099906 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:33.099868    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56f7ecc5-d9a7-4f5c-a572-338b114f3e0f" path="/var/lib/kubelet/pods/56f7ecc5-d9a7-4f5c-a572-338b114f3e0f/volumes"
Oct 30 13:08:33.318450 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:33.318406    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:33.891014 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:33.890973    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d" event={"ID":"48119de6-b81b-4f07-b304-a33e16ab2d68","Type":"ContainerStarted","Data":"3da43ddb550a515febd09ab873c92ec7751ad065f0f75cc82d5ddb90f011fe82"}
Oct 30 13:08:34.765612 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:34.765572    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:34.867576 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:34.867545    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:08:34.868741 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:34.868620    4544 patch_prober.go:29] interesting pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 13:08:34.869010 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:34.868976    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d" podUID="48119de6-b81b-4f07-b304-a33e16ab2d68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 13:08:35.173095 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:35.173052    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:35.864390 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:35.864352    4544 patch_prober.go:29] interesting pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 13:08:35.864814 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:35.864413    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d" podUID="48119de6-b81b-4f07-b304-a33e16ab2d68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 13:08:36.864365 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:36.864328    4544 patch_prober.go:29] interesting pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 13:08:36.864555 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:36.864384    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d" podUID="48119de6-b81b-4f07-b304-a33e16ab2d68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 13:08:37.606243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:37.605242    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:37.723902 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:37.723864    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:37.876556 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:37.876460    4544 patch_prober.go:29] interesting pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 13:08:37.877883 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:37.877202    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d" podUID="48119de6-b81b-4f07-b304-a33e16ab2d68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 13:08:37.951922 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:37.951883    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:38.875371 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:38.875330    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:08:38.877185 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:38.876029    4544 patch_prober.go:29] interesting pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 13:08:38.877794 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:38.877767    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d" podUID="48119de6-b81b-4f07-b304-a33e16ab2d68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 13:08:39.709332 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:39.709296    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:39.878247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:39.872213    4544 patch_prober.go:29] interesting pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 13:08:39.878247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:39.872310    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d" podUID="48119de6-b81b-4f07-b304-a33e16ab2d68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 13:08:40.768934 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:40.768903    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl"
Oct 30 13:08:40.869363 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:40.869328    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:08:40.870724 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:40.870697    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:08:41.047867 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:41.047780    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cdd98ad-nb2n7"]
Oct 30 13:08:41.053628 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:41.053597    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cdd98ad-nb2n7"]
Oct 30 13:08:41.107579 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:41.107547    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aa572f0-ed04-4786-8c41-3ddfa27d85b3" path="/var/lib/kubelet/pods/5aa572f0-ed04-4786-8c41-3ddfa27d85b3/volumes"
Oct 30 13:08:41.494077 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:41.494031    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:41.977014 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:41.976973    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:43.032727 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:43.032688    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-269db3d3-bgv75"]
Oct 30 13:08:43.045323 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:43.045293    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-269db3d3-bgv75"]
Oct 30 13:08:43.089231 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:43.089195    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01f0b3fb-8556-4fd5-a819-5b1f35a8c640" path="/var/lib/kubelet/pods/01f0b3fb-8556-4fd5-a819-5b1f35a8c640/volumes"
Oct 30 13:08:43.710260 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:43.710202    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:45.710016 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:45.709982    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:47.732247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:47.727983    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:48.044083 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:48.043992    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ab7218e-vfc2f"]
Oct 30 13:08:48.054577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:48.054530    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ab7218e-vfc2f"]
Oct 30 13:08:49.088454 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:49.088423    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bee74c09-88b6-490d-8f7c-576adec205ba" path="/var/lib/kubelet/pods/bee74c09-88b6-490d-8f7c-576adec205ba/volumes"
Oct 30 13:08:49.224246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:49.212612    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:49.620661 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:49.620580    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:49.758682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:49.758645    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:50.076198 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:50.076159    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"]
Oct 30 13:08:50.082435 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:50.082411    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:08:50.117785 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:50.117740    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"]
Oct 30 13:08:50.127504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:50.127461    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-0e874e-openshift-default-dockercfg-4czbc\""
Oct 30 13:08:50.128390 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:50.128357    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/565c6710-ac33-437a-b2cc-5572a325523d-istio-token\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:08:50.128522 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:50.128468    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-workload-socket\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:08:50.128619 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:50.128601    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-istio-data\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:08:50.128685 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:50.128665    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-workload-certs\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:08:50.128815 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:50.128800    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqvhj\" (UniqueName: \"kubernetes.io/projected/565c6710-ac33-437a-b2cc-5572a325523d-kube-api-access-qqvhj\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:08:50.128915 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:50.128902    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-credential-socket\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:08:50.128980 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:50.128930    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-istio-envoy\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:08:50.129035 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:50.128956    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/565c6710-ac33-437a-b2cc-5572a325523d-istio-podinfo\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:08:50.129090 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:50.129033    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/565c6710-ac33-437a-b2cc-5572a325523d-istiod-ca-cert\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:08:50.184577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:50.177722    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:50.345295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:50.344853    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-credential-socket\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:08:50.345295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:50.344896    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-istio-envoy\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:08:50.345295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:50.344922    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/565c6710-ac33-437a-b2cc-5572a325523d-istio-podinfo\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:08:50.345295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:50.344972    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/565c6710-ac33-437a-b2cc-5572a325523d-istiod-ca-cert\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:08:50.345295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:50.345039    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/565c6710-ac33-437a-b2cc-5572a325523d-istio-token\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:08:50.345295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:50.345106    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-workload-socket\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:08:50.345295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:50.345196    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-istio-data\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:08:50.345295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:50.345274    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-workload-certs\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:08:50.345810 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:50.345446    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqvhj\" (UniqueName: \"kubernetes.io/projected/565c6710-ac33-437a-b2cc-5572a325523d-kube-api-access-qqvhj\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:08:50.357320 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:50.346327    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-credential-socket\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:08:50.357320 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:50.354768    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:50.357320 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:50.357140    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/565c6710-ac33-437a-b2cc-5572a325523d-istiod-ca-cert\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:08:50.365254 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:50.361512    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-istio-data\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:08:50.365254 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:50.361796    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-workload-socket\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:08:50.365254 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:50.362032    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-workload-certs\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:08:50.393243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:50.386195    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-istio-envoy\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:08:50.405244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:50.398180    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/565c6710-ac33-437a-b2cc-5572a325523d-istio-podinfo\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:08:50.448288 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:50.445276    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/565c6710-ac33-437a-b2cc-5572a325523d-istio-token\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:08:50.501301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:50.496187    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqvhj\" (UniqueName: \"kubernetes.io/projected/565c6710-ac33-437a-b2cc-5572a325523d-kube-api-access-qqvhj\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:08:50.707009 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:50.705792    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:08:51.075119 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:51.075088    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"]
Oct 30 13:08:51.077808 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:08:51.077771    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod565c6710_ac33_437a_b2cc_5572a325523d.slice/crio-4f0ae5283c3f27a5faa911332c32f300bb414d59418e2daae21007fbb2ec632c WatchSource:0}: Error finding container 4f0ae5283c3f27a5faa911332c32f300bb414d59418e2daae21007fbb2ec632c: Status 404 returned error can't find the container with id 4f0ae5283c3f27a5faa911332c32f300bb414d59418e2daae21007fbb2ec632c
Oct 30 13:08:51.080525 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:51.080491    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:08:51.080635 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:51.080568    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:08:51.080635 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:51.080616    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:08:51.535290 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:51.535154    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h" event={"ID":"565c6710-ac33-437a-b2cc-5572a325523d","Type":"ContainerStarted","Data":"4c146428fb3ecc1e27887d68f26980067a3c9352a0ecf97b6871ee162561feaa"}
Oct 30 13:08:51.535290 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:51.535259    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h" event={"ID":"565c6710-ac33-437a-b2cc-5572a325523d","Type":"ContainerStarted","Data":"4f0ae5283c3f27a5faa911332c32f300bb414d59418e2daae21007fbb2ec632c"}
Oct 30 13:08:51.583208 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:51.583141    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h" podStartSLOduration=1.583121478 podStartE2EDuration="1.583121478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:51.581928602 +0000 UTC m=+1426.629062555" watchObservedRunningTime="2026-10-30 13:08:51.583121478 +0000 UTC m=+1426.630255433"
Oct 30 13:08:51.719252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:51.715411    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:08:52.138901 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:52.138860    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:52.317527 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:52.316621    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:52.349059 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:52.349010    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:08:52.349397 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:52.349350    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs" podUID="e308ff03-37ae-4b7f-b27e-05b2f46ce66c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:52.714054 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:52.714019    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:08:53.686246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:53.679474    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:08:53.686246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:53.680880    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:08:54.176293 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:54.176261    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:54.291733 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:54.291701    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:54.480667 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:54.480576    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:54.622658 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:54.622624    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:54.804355 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:54.804175    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:55.730071 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:55.730036    4544 generic.go:356] "Generic (PLEG): container finished" podID="e308ff03-37ae-4b7f-b27e-05b2f46ce66c" containerID="804ab07a9a2dd4eeca0ccd71e14f562405ef85199228a5aeb185b2b29ec62402" exitCode=0
Oct 30 13:08:55.730660 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:55.730117    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs" event={"ID":"e308ff03-37ae-4b7f-b27e-05b2f46ce66c","Type":"ContainerDied","Data":"804ab07a9a2dd4eeca0ccd71e14f562405ef85199228a5aeb185b2b29ec62402"}
Oct 30 13:08:55.730745 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:55.730721    4544 scope.go:122] "RemoveContainer" containerID="804ab07a9a2dd4eeca0ccd71e14f562405ef85199228a5aeb185b2b29ec62402"
Oct 30 13:08:55.732695 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:55.732664    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:08:55.732765 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:55.732736    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:08:55.732810 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:55.732770    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:08:56.349764 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:56.349719    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:56.509082 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:56.509046    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:56.967245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:56.948422    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs" event={"ID":"e308ff03-37ae-4b7f-b27e-05b2f46ce66c","Type":"ContainerStarted","Data":"6aa0b9aac2f00d985e160d270a25eedf47c770451d610e36f60c1829e38b0023"}
Oct 30 13:08:57.379304 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:57.379266    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:08:57.380377 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:57.380351    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 13:08:57.380499 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:57.380415    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs" podUID="e308ff03-37ae-4b7f-b27e-05b2f46ce66c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 13:08:57.636301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:57.634018    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:58.379795 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:58.379758    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 13:08:58.380201 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:58.379816    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs" podUID="e308ff03-37ae-4b7f-b27e-05b2f46ce66c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 13:08:59.008876 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:08:59.008839    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:08:59.379332 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:59.379295    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 13:08:59.379536 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:59.379358    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs" podUID="e308ff03-37ae-4b7f-b27e-05b2f46ce66c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 13:08:59.549013 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:59.548973    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:08:59.663046 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:08:59.662924    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:00.047162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:00.047070    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d1c7a0c-slws2"]
Oct 30 13:09:00.058277 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:00.058241    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d1c7a0c-slws2"]
Oct 30 13:09:00.379514 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:00.379477    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 13:09:00.379706 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:00.379536    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs" podUID="e308ff03-37ae-4b7f-b27e-05b2f46ce66c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 13:09:01.216244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:01.208623    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30a39639-10fa-4c33-a923-f3ddfaf3d9aa" path="/var/lib/kubelet/pods/30a39639-10fa-4c33-a923-f3ddfaf3d9aa/volumes"
Oct 30 13:09:01.382385 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:01.381750    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 13:09:01.382385 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:01.381818    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs" podUID="e308ff03-37ae-4b7f-b27e-05b2f46ce66c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 13:09:01.878438 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:01.878402    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:02.379815 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:02.379774    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 13:09:02.380277 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:02.379838    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs" podUID="e308ff03-37ae-4b7f-b27e-05b2f46ce66c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 13:09:03.392797 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:03.392756    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:09:03.393654 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:03.393629    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:09:03.395352 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:03.395327    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:09:04.053244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:04.048105    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:06.409776 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:06.409735    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:08.258791 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:08.258753    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:08.578730 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:08.578696    4544 scope.go:122] "RemoveContainer" containerID="168d03a3ebd722e60fecfac7c7daa1e4f15ba656c5b69f7fcf5a78972f11f26b"
Oct 30 13:09:08.591676 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:08.591646    4544 scope.go:122] "RemoveContainer" containerID="e4e479758007a580acb0d94e67e670901af2fab5ed7c83ab2c59ecb8985ee70d"
Oct 30 13:09:08.602881 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:08.602845    4544 scope.go:122] "RemoveContainer" containerID="c9b7a6f2d925ea2360a7142996ee4d14fac7b14d5881f4c5587e2fed5aeec1dc"
Oct 30 13:09:08.614329 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:08.614294    4544 scope.go:122] "RemoveContainer" containerID="343aca8b21c018e28baaff4af6c65f56b341d45c54c481893bfe28187cc6330f"
Oct 30 13:09:08.625434 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:08.625415    4544 scope.go:122] "RemoveContainer" containerID="1c46156a40d4e705e107e50846912fd51df24f5db6d5d537683c14674ca086b5"
Oct 30 13:09:08.639966 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:08.639934    4544 scope.go:122] "RemoveContainer" containerID="55bec12d02494bfbf68cb0b269d577e82e9b0ebdbba2846a2261ea0fafae3b47"
Oct 30 13:09:08.651290 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:08.651258    4544 scope.go:122] "RemoveContainer" containerID="f2b0c8724717e5f839c36c07a23fc988f3278c7cfe6553a7defc6280de70c228"
Oct 30 13:09:10.454094 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:10.454022    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:12.037685 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:12.037650    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c974f1-22fj9"]
Oct 30 13:09:12.048593 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:12.048559    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c974f1-22fj9"]
Oct 30 13:09:12.476030 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:12.475993    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:13.095785 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:13.095740    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d04db7d3-e497-4c07-97b4-3d2732f4f1cb" path="/var/lib/kubelet/pods/d04db7d3-e497-4c07-97b4-3d2732f4f1cb/volumes"
Oct 30 13:09:14.120045 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:14.120004    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:14.246821 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:14.246784    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:15.158352 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:15.158312    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22cb3314-jt7gp"]
Oct 30 13:09:15.165382 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:15.165349    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb3314-jt7gp"
Oct 30 13:09:15.189009 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:15.188971    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22cb3314-jt7gp"]
Oct 30 13:09:15.194704 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:15.194667    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:09:15.194907 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:15.194783    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:09:15.298826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:15.298788    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcdcq\" (UniqueName: \"kubernetes.io/projected/ad9b8e49-10ba-43ae-a17e-20ee84fef5bc-kube-api-access-bcdcq\") pod \"maas-api-revoke-sub-22cb3314-jt7gp\" (UID: \"ad9b8e49-10ba-43ae-a17e-20ee84fef5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb3314-jt7gp"
Oct 30 13:09:15.299053 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:15.299032    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad9b8e49-10ba-43ae-a17e-20ee84fef5bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22cb3314-jt7gp\" (UID: \"ad9b8e49-10ba-43ae-a17e-20ee84fef5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb3314-jt7gp"
Oct 30 13:09:15.299105 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:15.299069    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad9b8e49-10ba-43ae-a17e-20ee84fef5bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22cb3314-jt7gp\" (UID: \"ad9b8e49-10ba-43ae-a17e-20ee84fef5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb3314-jt7gp"
Oct 30 13:09:15.400453 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:15.400406    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad9b8e49-10ba-43ae-a17e-20ee84fef5bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22cb3314-jt7gp\" (UID: \"ad9b8e49-10ba-43ae-a17e-20ee84fef5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb3314-jt7gp"
Oct 30 13:09:15.400453 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:15.400454    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad9b8e49-10ba-43ae-a17e-20ee84fef5bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22cb3314-jt7gp\" (UID: \"ad9b8e49-10ba-43ae-a17e-20ee84fef5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb3314-jt7gp"
Oct 30 13:09:15.400727 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:15.400643    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcdcq\" (UniqueName: \"kubernetes.io/projected/ad9b8e49-10ba-43ae-a17e-20ee84fef5bc-kube-api-access-bcdcq\") pod \"maas-api-revoke-sub-22cb3314-jt7gp\" (UID: \"ad9b8e49-10ba-43ae-a17e-20ee84fef5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb3314-jt7gp"
Oct 30 13:09:15.401287 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:15.401199    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad9b8e49-10ba-43ae-a17e-20ee84fef5bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22cb3314-jt7gp\" (UID: \"ad9b8e49-10ba-43ae-a17e-20ee84fef5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb3314-jt7gp"
Oct 30 13:09:15.403205 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:15.403177    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad9b8e49-10ba-43ae-a17e-20ee84fef5bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22cb3314-jt7gp\" (UID: \"ad9b8e49-10ba-43ae-a17e-20ee84fef5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb3314-jt7gp"
Oct 30 13:09:15.426021 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:15.425948    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcdcq\" (UniqueName: \"kubernetes.io/projected/ad9b8e49-10ba-43ae-a17e-20ee84fef5bc-kube-api-access-bcdcq\") pod \"maas-api-revoke-sub-22cb3314-jt7gp\" (UID: \"ad9b8e49-10ba-43ae-a17e-20ee84fef5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb3314-jt7gp"
Oct 30 13:09:15.480191 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:15.480160    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb3314-jt7gp"
Oct 30 13:09:15.792929 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:15.792899    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22cb3314-jt7gp"]
Oct 30 13:09:15.795029 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:09:15.794998    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad9b8e49_10ba_43ae_a17e_20ee84fef5bc.slice/crio-332216512a69215b9a08f5933f567bd685c5387bf49c94ca08ba6c16c383d0ef WatchSource:0}: Error finding container 332216512a69215b9a08f5933f567bd685c5387bf49c94ca08ba6c16c383d0ef: Status 404 returned error can't find the container with id 332216512a69215b9a08f5933f567bd685c5387bf49c94ca08ba6c16c383d0ef
Oct 30 13:09:15.954475 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:15.954420    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:16.260515 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.260471    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:16.410748 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.410437    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"]
Oct 30 13:09:16.426517 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.426131    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:09:16.446853 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.446815    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-75abce-openshift-default-dockercfg-7cql7\""
Oct 30 13:09:16.459431 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.459269    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"]
Oct 30 13:09:16.483523 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.483481    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:16.498908 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.498858    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb3314-jt7gp" event={"ID":"ad9b8e49-10ba-43ae-a17e-20ee84fef5bc","Type":"ContainerStarted","Data":"d1517540eed9dbb7bccd957e276759e40137460f0660895410b3e5d34ea97de1"}
Oct 30 13:09:16.499125 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.498929    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb3314-jt7gp" event={"ID":"ad9b8e49-10ba-43ae-a17e-20ee84fef5bc","Type":"ContainerStarted","Data":"332216512a69215b9a08f5933f567bd685c5387bf49c94ca08ba6c16c383d0ef"}
Oct 30 13:09:16.599244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.597263    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istio-podinfo\") pod \"e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") " pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:09:16.599244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.597419    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istiod-ca-cert\") pod \"e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") " pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:09:16.599244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.597505    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpmwk\" (UniqueName: \"kubernetes.io/projected/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-kube-api-access-vpmwk\") pod \"e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") " pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:09:16.624244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.621081    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istio-envoy\") pod \"e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") " pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:09:16.624244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.621284    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istio-token\") pod \"e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") " pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:09:16.624244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.621330    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-workload-socket\") pod \"e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") " pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:09:16.624244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.621359    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istio-data\") pod \"e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") " pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:09:16.624244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.621430    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-credential-socket\") pod \"e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") " pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:09:16.624244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.621625    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-workload-certs\") pod \"e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") " pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:09:16.624244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.624041    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb3314-jt7gp" podStartSLOduration=1.623918895 podStartE2EDuration="1.623918895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:16.569071382 +0000 UTC m=+1451.616205335" watchObservedRunningTime="2026-10-30 13:09:16.623918895 +0000 UTC m=+1451.671052849"
Oct 30 13:09:16.644247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.635976    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:16.727242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.724782    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istio-token\") pod \"e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") " pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:09:16.727242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.724863    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-workload-socket\") pod \"e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") " pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:09:16.727242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.724895    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istio-data\") pod \"e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") " pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:09:16.727242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.724991    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-credential-socket\") pod \"e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") " pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:09:16.727242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.725193    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-workload-certs\") pod \"e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") " pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:09:16.727242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.725322    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istio-podinfo\") pod \"e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") " pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:09:16.727242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.725446    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istiod-ca-cert\") pod \"e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") " pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:09:16.727242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.725524    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpmwk\" (UniqueName: \"kubernetes.io/projected/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-kube-api-access-vpmwk\") pod \"e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") " pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:09:16.727242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.725664    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istio-envoy\") pod \"e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") " pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:09:16.738254 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.737035    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-workload-socket\") pod \"e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") " pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:09:16.738254 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.737441    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-credential-socket\") pod \"e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") " pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:09:16.738254 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.737767    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-workload-certs\") pod \"e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") " pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:09:16.738567 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.738522    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istiod-ca-cert\") pod \"e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") " pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:09:16.742250 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.739870    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istio-data\") pod \"e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") " pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:09:16.751307 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.751240    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istio-podinfo\") pod \"e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") " pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:09:16.793241 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.781481    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpmwk\" (UniqueName: \"kubernetes.io/projected/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-kube-api-access-vpmwk\") pod \"e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") " pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:09:16.793241 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.782622    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istio-token\") pod \"e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") " pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:09:16.793241 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:16.791626    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istio-envoy\") pod \"e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") " pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:09:17.073812 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:17.066199    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:09:17.451559 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:17.451527    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"]
Oct 30 13:09:17.453815 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:09:17.453785    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb56cc33e_52a6_4f72_b0b4_b438e5ea24d6.slice/crio-b10d9d0462f62a060b5d188f28fa10ef6a7e9f86c6b8b5c39a137cc376f0a16d WatchSource:0}: Error finding container b10d9d0462f62a060b5d188f28fa10ef6a7e9f86c6b8b5c39a137cc376f0a16d: Status 404 returned error can't find the container with id b10d9d0462f62a060b5d188f28fa10ef6a7e9f86c6b8b5c39a137cc376f0a16d
Oct 30 13:09:17.456818 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:17.456766    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:09:17.456937 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:17.456866    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:09:17.456937 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:17.456913    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:09:17.507959 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:17.507922    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r" event={"ID":"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6","Type":"ContainerStarted","Data":"b10d9d0462f62a060b5d188f28fa10ef6a7e9f86c6b8b5c39a137cc376f0a16d"}
Oct 30 13:09:17.509869 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:17.509838    4544 generic.go:356] "Generic (PLEG): container finished" podID="ad9b8e49-10ba-43ae-a17e-20ee84fef5bc" containerID="d1517540eed9dbb7bccd957e276759e40137460f0660895410b3e5d34ea97de1" exitCode=0
Oct 30 13:09:17.510017 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:17.509874    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb3314-jt7gp" event={"ID":"ad9b8e49-10ba-43ae-a17e-20ee84fef5bc","Type":"ContainerDied","Data":"d1517540eed9dbb7bccd957e276759e40137460f0660895410b3e5d34ea97de1"}
Oct 30 13:09:18.103521 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:18.103486    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:18.543031 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:18.542936    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r" event={"ID":"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6","Type":"ContainerStarted","Data":"2b81707a031b5c82769d5aee00a697487f3780d697793daf3cee864d856c1008"}
Oct 30 13:09:18.721001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:18.720973    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb3314-jt7gp"
Oct 30 13:09:18.763445 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:18.763392    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r" podStartSLOduration=2.76337754 podStartE2EDuration="2.76337754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:18.6358024 +0000 UTC m=+1453.682936355" watchObservedRunningTime="2026-10-30 13:09:18.76337754 +0000 UTC m=+1453.810511492"
Oct 30 13:09:18.892774 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:18.892722    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad9b8e49-10ba-43ae-a17e-20ee84fef5bc-kube-api-access-bcdcq\" (UniqueName: \"kubernetes.io/projected/ad9b8e49-10ba-43ae-a17e-20ee84fef5bc-kube-api-access-bcdcq\") pod \"ad9b8e49-10ba-43ae-a17e-20ee84fef5bc\" (UID: \"ad9b8e49-10ba-43ae-a17e-20ee84fef5bc\") "
Oct 30 13:09:18.903716 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:18.903683    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad9b8e49-10ba-43ae-a17e-20ee84fef5bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad9b8e49-10ba-43ae-a17e-20ee84fef5bc-maas-api-openshift-ca\") pod \"ad9b8e49-10ba-43ae-a17e-20ee84fef5bc\" (UID: \"ad9b8e49-10ba-43ae-a17e-20ee84fef5bc\") "
Oct 30 13:09:18.906169 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:18.906139    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad9b8e49-10ba-43ae-a17e-20ee84fef5bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad9b8e49-10ba-43ae-a17e-20ee84fef5bc-maas-api-xks-ca\") pod \"ad9b8e49-10ba-43ae-a17e-20ee84fef5bc\" (UID: \"ad9b8e49-10ba-43ae-a17e-20ee84fef5bc\") "
Oct 30 13:09:18.907708 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:18.905944    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad9b8e49-10ba-43ae-a17e-20ee84fef5bc-maas-api-openshift-ca" pod "ad9b8e49-10ba-43ae-a17e-20ee84fef5bc" (UID: "ad9b8e49-10ba-43ae-a17e-20ee84fef5bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:18.921042 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:18.921006    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad9b8e49-10ba-43ae-a17e-20ee84fef5bc-kube-api-access-bcdcq" pod "ad9b8e49-10ba-43ae-a17e-20ee84fef5bc" (UID: "ad9b8e49-10ba-43ae-a17e-20ee84fef5bc"). InnerVolumeSpecName "kube-api-access-bcdcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:18.923015 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:18.922977    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad9b8e49-10ba-43ae-a17e-20ee84fef5bc-maas-api-xks-ca" pod "ad9b8e49-10ba-43ae-a17e-20ee84fef5bc" (UID: "ad9b8e49-10ba-43ae-a17e-20ee84fef5bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:09:19.009194 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:19.009155    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcdcq\" (UniqueName: \"kubernetes.io/projected/ad9b8e49-10ba-43ae-a17e-20ee84fef5bc-kube-api-access-bcdcq\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:19.009194 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:19.009189    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad9b8e49-10ba-43ae-a17e-20ee84fef5bc-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:19.009194 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:19.009205    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad9b8e49-10ba-43ae-a17e-20ee84fef5bc-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:19.067826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:19.067792    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:09:19.074084 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:19.074057    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:09:19.552946 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:19.552908    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb3314-jt7gp"
Oct 30 13:09:19.553478 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:19.552938    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb3314-jt7gp" event={"ID":"ad9b8e49-10ba-43ae-a17e-20ee84fef5bc","Type":"ContainerDied","Data":"332216512a69215b9a08f5933f567bd685c5387bf49c94ca08ba6c16c383d0ef"}
Oct 30 13:09:19.553478 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:19.552995    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="332216512a69215b9a08f5933f567bd685c5387bf49c94ca08ba6c16c383d0ef"
Oct 30 13:09:19.553478 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:19.553321    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:09:19.554563 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:19.554544    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:09:20.300246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:20.300180    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:21.033685 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:21.033648    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9185e827-6j8kb"]
Oct 30 13:09:21.044969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:21.044926    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9185e827-6j8kb"]
Oct 30 13:09:21.090973 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:21.090936    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88815187-6e50-4b15-984a-ac4efe339e0e" path="/var/lib/kubelet/pods/88815187-6e50-4b15-984a-ac4efe339e0e/volumes"
Oct 30 13:09:21.119973 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:21.119926    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:21.273681 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:21.273645    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:22.374964 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:22.374926    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:22.988762 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:22.988726    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:23.108107 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:23.108069    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:23.476337 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:23.476288    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-75abce-bd9687d56-scc2b"]
Oct 30 13:09:23.527730 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:23.527427    4544 cpu_manager.go:414] "containerMap: removing container" podUID="ad9b8e49-10ba-43ae-a17e-20ee84fef5bc" containerName="revoke-keys"
Oct 30 13:09:23.566814 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:23.566783    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad9b8e49-10ba-43ae-a17e-20ee84fef5bc" containerName="revoke-keys"
Oct 30 13:09:23.597678 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:23.597642    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad9b8e49-10ba-43ae-a17e-20ee84fef5bc" containerName="revoke-keys"
Oct 30 13:09:23.623601 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:23.623552    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-75abce-bd9687d56-scc2b"]
Oct 30 13:09:23.623974 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:23.623901    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-75abce-bd9687d56-scc2b"
Oct 30 13:09:23.668244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:23.652103    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-75abce\""
Oct 30 13:09:23.712762 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:23.712729    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-75abce-dockercfg-srqhl\""
Oct 30 13:09:23.753618 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:23.753509    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfbf4\" (UniqueName: \"kubernetes.io/projected/e3314158-027a-4c7d-8592-288a8ac98655-kube-api-access-mfbf4\") pod \"payload-processing-e2e-api-a-75abce-bd9687d56-scc2b\" (UID: \"e3314158-027a-4c7d-8592-288a8ac98655\") " pod="openshift-ingress/payload-processing-e2e-api-a-75abce-bd9687d56-scc2b"
Oct 30 13:09:23.753618 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:23.753593    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3314158-027a-4c7d-8592-288a8ac98655-plugins-config-volume\") pod \"payload-processing-e2e-api-a-75abce-bd9687d56-scc2b\" (UID: \"e3314158-027a-4c7d-8592-288a8ac98655\") " pod="openshift-ingress/payload-processing-e2e-api-a-75abce-bd9687d56-scc2b"
Oct 30 13:09:23.855002 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:23.854963    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfbf4\" (UniqueName: \"kubernetes.io/projected/e3314158-027a-4c7d-8592-288a8ac98655-kube-api-access-mfbf4\") pod \"payload-processing-e2e-api-a-75abce-bd9687d56-scc2b\" (UID: \"e3314158-027a-4c7d-8592-288a8ac98655\") " pod="openshift-ingress/payload-processing-e2e-api-a-75abce-bd9687d56-scc2b"
Oct 30 13:09:23.855168 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:23.855021    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3314158-027a-4c7d-8592-288a8ac98655-plugins-config-volume\") pod \"payload-processing-e2e-api-a-75abce-bd9687d56-scc2b\" (UID: \"e3314158-027a-4c7d-8592-288a8ac98655\") " pod="openshift-ingress/payload-processing-e2e-api-a-75abce-bd9687d56-scc2b"
Oct 30 13:09:23.855792 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:23.855769    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3314158-027a-4c7d-8592-288a8ac98655-plugins-config-volume\") pod \"payload-processing-e2e-api-a-75abce-bd9687d56-scc2b\" (UID: \"e3314158-027a-4c7d-8592-288a8ac98655\") " pod="openshift-ingress/payload-processing-e2e-api-a-75abce-bd9687d56-scc2b"
Oct 30 13:09:23.878190 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:23.878150    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfbf4\" (UniqueName: \"kubernetes.io/projected/e3314158-027a-4c7d-8592-288a8ac98655-kube-api-access-mfbf4\") pod \"payload-processing-e2e-api-a-75abce-bd9687d56-scc2b\" (UID: \"e3314158-027a-4c7d-8592-288a8ac98655\") " pod="openshift-ingress/payload-processing-e2e-api-a-75abce-bd9687d56-scc2b"
Oct 30 13:09:24.020703 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:24.020623    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-75abce-bd9687d56-scc2b"
Oct 30 13:09:24.342079 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:24.342021    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-75abce-bd9687d56-scc2b"]
Oct 30 13:09:24.671258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:24.671069    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-75abce-bd9687d56-scc2b" event={"ID":"e3314158-027a-4c7d-8592-288a8ac98655","Type":"ContainerStarted","Data":"550ff3ac4ce062234f76c3dd9294508eeedb0c2acdb6482a64a0e953aae8b067"}
Oct 30 13:09:24.971342 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:24.971214    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:25.032856 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:25.032819    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c7d0d09-5wsrb"]
Oct 30 13:09:25.037483 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:25.037455    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c7d0d09-5wsrb"]
Oct 30 13:09:25.085801 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:25.085771    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:25.088963 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:25.088932    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b74c7aa0-c2d0-4da4-a64f-259f43155a74" path="/var/lib/kubelet/pods/b74c7aa0-c2d0-4da4-a64f-259f43155a74/volumes"
Oct 30 13:09:25.216541 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:25.216496    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:25.681824 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:25.681789    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-75abce-bd9687d56-scc2b" event={"ID":"e3314158-027a-4c7d-8592-288a8ac98655","Type":"ContainerStarted","Data":"85976aaaf6dcf15bcab73a5c6604ab77301b731bdfd9f29935b50da2f614c8db"}
Oct 30 13:09:25.682289 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:25.681850    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-75abce-bd9687d56-scc2b"
Oct 30 13:09:25.746371 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:25.746311    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-75abce-bd9687d56-scc2b" podStartSLOduration=2.746295274 podStartE2EDuration="2.746295274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:25.741629904 +0000 UTC m=+1460.788763856" watchObservedRunningTime="2026-10-30 13:09:25.746295274 +0000 UTC m=+1460.793429227"
Oct 30 13:09:26.343280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:26.343214    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:26.821100 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:26.821002    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:27.094877 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:27.077549    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a3c14f0-tt7vj"]
Oct 30 13:09:27.145498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:27.145462    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a3c14f0-tt7vj"]
Oct 30 13:09:27.373515 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:27.373418    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:27.676301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:27.667581    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:27.897563 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:27.836314    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:28.127840 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:28.127806    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:28.265370 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:28.265329    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:29.009304 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:09:29.009260    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:09:29.089303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:29.089257    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6143936-5028-4fab-bbf5-f61a445ed99b" path="/var/lib/kubelet/pods/a6143936-5028-4fab-bbf5-f61a445ed99b/volumes"
Oct 30 13:09:29.984121 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:29.984084    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:31.811522 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:31.811488    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:31.944859 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:31.944825    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:33.028649 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:33.028611    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dad92b6-bpxw8"]
Oct 30 13:09:33.039786 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:33.039755    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dad92b6-bpxw8"]
Oct 30 13:09:33.088018 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:33.087984    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2928d5f2-e80e-4c43-9f56-6ff5d1b86759" path="/var/lib/kubelet/pods/2928d5f2-e80e-4c43-9f56-6ff5d1b86759/volumes"
Oct 30 13:09:33.577744 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:33.577710    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:33.737673 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:33.737637    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:33.772476 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:33.772431    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq"]
Oct 30 13:09:33.778546 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:33.778523    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq"
Oct 30 13:09:33.799303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:33.799266    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq"]
Oct 30 13:09:33.804532 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:33.804500    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-acd125\""
Oct 30 13:09:33.804819 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:33.804792    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-acd125-dockercfg-r2svl\""
Oct 30 13:09:33.820037 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:33.819996    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx"]
Oct 30 13:09:33.826904 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:33.826868    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx"
Oct 30 13:09:33.934381 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:33.914183    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcbvs\" (UniqueName: \"kubernetes.io/projected/fb981ab3-c15c-48d3-868d-5d5d93616212-kube-api-access-fcbvs\") pod \"payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx\" (UID: \"fb981ab3-c15c-48d3-868d-5d5d93616212\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx"
Oct 30 13:09:33.962287 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:33.935781    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8013c75e-2665-43ee-879b-ec14f338cca5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq\" (UID: \"8013c75e-2665-43ee-879b-ec14f338cca5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq"
Oct 30 13:09:33.962287 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:33.935947    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb981ab3-c15c-48d3-868d-5d5d93616212-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx\" (UID: \"fb981ab3-c15c-48d3-868d-5d5d93616212\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx"
Oct 30 13:09:33.962287 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:33.936236    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4k24\" (UniqueName: \"kubernetes.io/projected/8013c75e-2665-43ee-879b-ec14f338cca5-kube-api-access-d4k24\") pod \"payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq\" (UID: \"8013c75e-2665-43ee-879b-ec14f338cca5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq"
Oct 30 13:09:34.058724 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:34.051949    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcbvs\" (UniqueName: \"kubernetes.io/projected/fb981ab3-c15c-48d3-868d-5d5d93616212-kube-api-access-fcbvs\") pod \"payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx\" (UID: \"fb981ab3-c15c-48d3-868d-5d5d93616212\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx"
Oct 30 13:09:34.058724 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:34.052067    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8013c75e-2665-43ee-879b-ec14f338cca5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq\" (UID: \"8013c75e-2665-43ee-879b-ec14f338cca5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq"
Oct 30 13:09:34.058724 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:34.052163    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb981ab3-c15c-48d3-868d-5d5d93616212-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx\" (UID: \"fb981ab3-c15c-48d3-868d-5d5d93616212\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx"
Oct 30 13:09:34.058724 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:34.052311    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4k24\" (UniqueName: \"kubernetes.io/projected/8013c75e-2665-43ee-879b-ec14f338cca5-kube-api-access-d4k24\") pod \"payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq\" (UID: \"8013c75e-2665-43ee-879b-ec14f338cca5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq"
Oct 30 13:09:34.072271 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:34.068664    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb981ab3-c15c-48d3-868d-5d5d93616212-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx\" (UID: \"fb981ab3-c15c-48d3-868d-5d5d93616212\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx"
Oct 30 13:09:34.298139 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:34.279705    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8013c75e-2665-43ee-879b-ec14f338cca5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq\" (UID: \"8013c75e-2665-43ee-879b-ec14f338cca5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq"
Oct 30 13:09:34.298139 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:34.295919    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcbvs\" (UniqueName: \"kubernetes.io/projected/fb981ab3-c15c-48d3-868d-5d5d93616212-kube-api-access-fcbvs\") pod \"payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx\" (UID: \"fb981ab3-c15c-48d3-868d-5d5d93616212\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx"
Oct 30 13:09:34.336729 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:34.336686    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4k24\" (UniqueName: \"kubernetes.io/projected/8013c75e-2665-43ee-879b-ec14f338cca5-kube-api-access-d4k24\") pod \"payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq\" (UID: \"8013c75e-2665-43ee-879b-ec14f338cca5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq"
Oct 30 13:09:34.351520 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:34.351462    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx"]
Oct 30 13:09:34.363846 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:34.363795    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:34.375979 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:34.375936    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx"
Oct 30 13:09:34.447312 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:34.393860    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq"
Oct 30 13:09:34.680740 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:34.680702    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:34.816989 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:34.816961    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq"]
Oct 30 13:09:34.820024 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:09:34.819994    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8013c75e_2665_43ee_879b_ec14f338cca5.slice/crio-c16bc3a94bf86ed694c957d536f92719c07cfb44c85d963d3c0870abb90cad70 WatchSource:0}: Error finding container c16bc3a94bf86ed694c957d536f92719c07cfb44c85d963d3c0870abb90cad70: Status 404 returned error can't find the container with id c16bc3a94bf86ed694c957d536f92719c07cfb44c85d963d3c0870abb90cad70
Oct 30 13:09:34.830884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:34.830855    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx"]
Oct 30 13:09:34.835326 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:09:34.835294    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb981ab3_c15c_48d3_868d_5d5d93616212.slice/crio-ca17c04994dee5c5072f826613f8b19ab88afb6ee002290287db0c31535be803 WatchSource:0}: Error finding container ca17c04994dee5c5072f826613f8b19ab88afb6ee002290287db0c31535be803: Status 404 returned error can't find the container with id ca17c04994dee5c5072f826613f8b19ab88afb6ee002290287db0c31535be803
Oct 30 13:09:35.035776 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:35.035743    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3f4ca9d-48vtn"]
Oct 30 13:09:35.046301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:35.046268    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3f4ca9d-48vtn"]
Oct 30 13:09:35.089683 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:35.089644    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b82845b6-026b-4509-ac36-de4bee8e710b" path="/var/lib/kubelet/pods/b82845b6-026b-4509-ac36-de4bee8e710b/volumes"
Oct 30 13:09:35.305156 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:35.305116    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx" event={"ID":"fb981ab3-c15c-48d3-868d-5d5d93616212","Type":"ContainerStarted","Data":"56127a11104d74fee2e2b51d20306dc9198826a0132bcc9c92fa180264c09248"}
Oct 30 13:09:35.305389 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:35.305167    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx" event={"ID":"fb981ab3-c15c-48d3-868d-5d5d93616212","Type":"ContainerStarted","Data":"ca17c04994dee5c5072f826613f8b19ab88afb6ee002290287db0c31535be803"}
Oct 30 13:09:35.305389 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:35.305206    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx"
Oct 30 13:09:35.306921 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:35.306893    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq" event={"ID":"8013c75e-2665-43ee-879b-ec14f338cca5","Type":"ContainerStarted","Data":"94ed8da1a2b8104aeb2efa3559ca628109c55dcf45e01b0ee5dee61aba172e25"}
Oct 30 13:09:35.307037 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:35.306943    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq" event={"ID":"8013c75e-2665-43ee-879b-ec14f338cca5","Type":"ContainerStarted","Data":"c16bc3a94bf86ed694c957d536f92719c07cfb44c85d963d3c0870abb90cad70"}
Oct 30 13:09:35.307037 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:35.306984    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq"
Oct 30 13:09:35.348627 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:35.348522    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx" podStartSLOduration=2.348507057 podStartE2EDuration="2.348507057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:35.344125166 +0000 UTC m=+1470.391259115" watchObservedRunningTime="2026-10-30 13:09:35.348507057 +0000 UTC m=+1470.395641010"
Oct 30 13:09:35.386845 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:35.386785    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq" podStartSLOduration=2.386768943 podStartE2EDuration="2.386768943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:35.382996371 +0000 UTC m=+1470.430130338" watchObservedRunningTime="2026-10-30 13:09:35.386768943 +0000 UTC m=+1470.433902896"
Oct 30 13:09:36.398784 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:36.398745    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:36.699010 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:36.698913    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-75abce-bd9687d56-scc2b"
Oct 30 13:09:36.783443 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:36.783406    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:37.246899 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:37.246864    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:37.746394 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:37.746354    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:38.435988 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:38.435952    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:38.667246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:38.661614    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:40.602185 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:40.602147    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:41.110964 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:41.110928    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"]
Oct 30 13:09:41.111168 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:41.111123    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl" podUID="89b2c1ca-75ce-45bb-93bc-d1da88552cf5" containerName="istio-proxy" containerID="cri-o://aabf38611462548a1a9eccbca9a5352f764aac7b6afc6de687d730ea0818bfd8" gracePeriod=30
Oct 30 13:09:41.125153 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:41.125126    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"]
Oct 30 13:09:41.125385 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:41.125335    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455" podUID="41b0db3d-0f0b-4791-aca0-0cd31a7c78f6" containerName="istio-proxy" containerID="cri-o://77286f3d171b3dd8acf2c5fb348c27c083f88db45475d778e6fc06b2681d66b6" gracePeriod=30
Oct 30 13:09:41.129970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:41.129946    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"]
Oct 30 13:09:41.137302 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:41.130602    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg" podUID="fa2c83b0-34e5-4b64-ba1c-b38f86978e24" containerName="istio-proxy" containerID="cri-o://5581df816922fa36c10d2cc3d7d0eb62a002e34ac63bf23f81e187283e043ced" gracePeriod=30
Oct 30 13:09:41.140189 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:41.140140    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"]
Oct 30 13:09:41.140540 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:41.140490    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7" podUID="317201b4-13b3-4341-a5a2-e4be2531f5f3" containerName="istio-proxy" containerID="cri-o://93bd1fd2fea5b74865f234e18c25a3615303d6b9db471c2efc2900b3436fe170" gracePeriod=30
Oct 30 13:09:42.142813 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:42.142772    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:42.280249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:42.270416    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:43.064945 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:43.064891    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d33cb1-9c6zf"]
Oct 30 13:09:43.069506 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:43.069472    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d33cb1-9c6zf"]
Oct 30 13:09:43.091294 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:43.091256    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1bb3236-c672-4abd-85f4-56f8ae6f6c65" path="/var/lib/kubelet/pods/e1bb3236-c672-4abd-85f4-56f8ae6f6c65/volumes"
Oct 30 13:09:43.593513 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:43.593479    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:44.641245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:44.640339    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:44.752859 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:44.752818    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:46.075885 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.075836    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:46.289365 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.282373    4544 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": read tcp 10.133.0.2:48504->10.133.0.47:15021: read: connection reset by peer" start-of-body=
Oct 30 13:09:46.289365 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.282460    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455" podUID="41b0db3d-0f0b-4791-aca0-0cd31a7c78f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": read tcp 10.133.0.2:48504->10.133.0.47:15021: read: connection reset by peer"
Oct 30 13:09:46.348551 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.328854    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx"
Oct 30 13:09:46.362854 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.360197    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq"
Oct 30 13:09:46.497281 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.494819    4544 generic.go:356] "Generic (PLEG): container finished" podID="41b0db3d-0f0b-4791-aca0-0cd31a7c78f6" containerID="77286f3d171b3dd8acf2c5fb348c27c083f88db45475d778e6fc06b2681d66b6" exitCode=0
Oct 30 13:09:46.497281 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.494959    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455" event={"ID":"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6","Type":"ContainerDied","Data":"77286f3d171b3dd8acf2c5fb348c27c083f88db45475d778e6fc06b2681d66b6"}
Oct 30 13:09:46.497281 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.495003    4544 scope.go:122] "RemoveContainer" containerID="ae17880baa1f57fc9fb2a30b8f7dc4d53366af63e088ac3e0303490b44cd49e8"
Oct 30 13:09:46.498537 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.498507    4544 generic.go:356] "Generic (PLEG): container finished" podID="89b2c1ca-75ce-45bb-93bc-d1da88552cf5" containerID="aabf38611462548a1a9eccbca9a5352f764aac7b6afc6de687d730ea0818bfd8" exitCode=0
Oct 30 13:09:46.498676 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.498598    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl" event={"ID":"89b2c1ca-75ce-45bb-93bc-d1da88552cf5","Type":"ContainerDied","Data":"aabf38611462548a1a9eccbca9a5352f764aac7b6afc6de687d730ea0818bfd8"}
Oct 30 13:09:46.501428 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.501394    4544 generic.go:356] "Generic (PLEG): container finished" podID="317201b4-13b3-4341-a5a2-e4be2531f5f3" containerID="93bd1fd2fea5b74865f234e18c25a3615303d6b9db471c2efc2900b3436fe170" exitCode=0
Oct 30 13:09:46.501721 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.501699    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7" event={"ID":"317201b4-13b3-4341-a5a2-e4be2531f5f3","Type":"ContainerDied","Data":"93bd1fd2fea5b74865f234e18c25a3615303d6b9db471c2efc2900b3436fe170"}
Oct 30 13:09:46.503570 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.503376    4544 generic.go:356] "Generic (PLEG): container finished" podID="fa2c83b0-34e5-4b64-ba1c-b38f86978e24" containerID="5581df816922fa36c10d2cc3d7d0eb62a002e34ac63bf23f81e187283e043ced" exitCode=0
Oct 30 13:09:46.503570 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.503405    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg" event={"ID":"fa2c83b0-34e5-4b64-ba1c-b38f86978e24","Type":"ContainerDied","Data":"5581df816922fa36c10d2cc3d7d0eb62a002e34ac63bf23f81e187283e043ced"}
Oct 30 13:09:46.795678 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.795207    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:09:46.862699 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.862666    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 13:09:46.866572 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.866547    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:09:46.912014 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.911980    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istio-podinfo\") pod \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") "
Oct 30 13:09:46.912236 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.912027    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istiod-ca-cert\") pod \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") "
Oct 30 13:09:46.912236 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.912055    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istio-data\") pod \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") "
Oct 30 13:09:46.912236 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.912086    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istio-envoy\") pod \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") "
Oct 30 13:09:46.912236 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.912153    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-workload-socket\") pod \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") "
Oct 30 13:09:46.912464 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.912257    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-credential-socket\") pod \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") "
Oct 30 13:09:46.912464 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.912378    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istio-token\" (UniqueName: \"kubernetes.io/projected/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istio-token\") pod \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") "
Oct 30 13:09:46.912464 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.912427    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-kube-api-access-gr79h\" (UniqueName: \"kubernetes.io/projected/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-kube-api-access-gr79h\") pod \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") "
Oct 30 13:09:46.912612 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.912558    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-workload-certs\") pod \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\" (UID: \"89b2c1ca-75ce-45bb-93bc-d1da88552cf5\") "
Oct 30 13:09:46.940966 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.940673    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istiod-ca-cert" pod "89b2c1ca-75ce-45bb-93bc-d1da88552cf5" (UID: "89b2c1ca-75ce-45bb-93bc-d1da88552cf5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:46.941548 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.941213    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istio-data" pod "89b2c1ca-75ce-45bb-93bc-d1da88552cf5" (UID: "89b2c1ca-75ce-45bb-93bc-d1da88552cf5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:46.951378 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.951327    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-workload-certs" pod "89b2c1ca-75ce-45bb-93bc-d1da88552cf5" (UID: "89b2c1ca-75ce-45bb-93bc-d1da88552cf5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:46.951569 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.951427    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istio-podinfo" pod "89b2c1ca-75ce-45bb-93bc-d1da88552cf5" (UID: "89b2c1ca-75ce-45bb-93bc-d1da88552cf5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:09:46.951711 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.951689    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-workload-socket" pod "89b2c1ca-75ce-45bb-93bc-d1da88552cf5" (UID: "89b2c1ca-75ce-45bb-93bc-d1da88552cf5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:46.952604 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.952574    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-credential-socket" pod "89b2c1ca-75ce-45bb-93bc-d1da88552cf5" (UID: "89b2c1ca-75ce-45bb-93bc-d1da88552cf5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:46.955093 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.955055    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istio-token" pod "89b2c1ca-75ce-45bb-93bc-d1da88552cf5" (UID: "89b2c1ca-75ce-45bb-93bc-d1da88552cf5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:46.959905 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.955654    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-kube-api-access-gr79h" pod "89b2c1ca-75ce-45bb-93bc-d1da88552cf5" (UID: "89b2c1ca-75ce-45bb-93bc-d1da88552cf5"). InnerVolumeSpecName "kube-api-access-gr79h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:46.966014 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:46.965971    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istio-envoy" pod "89b2c1ca-75ce-45bb-93bc-d1da88552cf5" (UID: "89b2c1ca-75ce-45bb-93bc-d1da88552cf5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:47.041299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.015271    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-workload-certs\") pod \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") "
Oct 30 13:09:47.041299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.015329    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-istio-data\") pod \"317201b4-13b3-4341-a5a2-e4be2531f5f3\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") "
Oct 30 13:09:47.041299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.015375    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/317201b4-13b3-4341-a5a2-e4be2531f5f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/317201b4-13b3-4341-a5a2-e4be2531f5f3-istiod-ca-cert\") pod \"317201b4-13b3-4341-a5a2-e4be2531f5f3\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") "
Oct 30 13:09:47.041299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.015412    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istio-podinfo\") pod \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") "
Oct 30 13:09:47.041299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.015448    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/317201b4-13b3-4341-a5a2-e4be2531f5f3-kube-api-access-tkxk4\" (UniqueName: \"kubernetes.io/projected/317201b4-13b3-4341-a5a2-e4be2531f5f3-kube-api-access-tkxk4\") pod \"317201b4-13b3-4341-a5a2-e4be2531f5f3\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") "
Oct 30 13:09:47.041299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.015503    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istio-envoy\") pod \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") "
Oct 30 13:09:47.041299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.015581    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-credential-socket\") pod \"317201b4-13b3-4341-a5a2-e4be2531f5f3\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") "
Oct 30 13:09:47.041299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.015624    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istio-token\" (UniqueName: \"kubernetes.io/projected/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istio-token\") pod \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") "
Oct 30 13:09:47.041299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.015675    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istiod-ca-cert\") pod \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") "
Oct 30 13:09:47.041299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.015717    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-workload-certs\") pod \"317201b4-13b3-4341-a5a2-e4be2531f5f3\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") "
Oct 30 13:09:47.041299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.015768    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/317201b4-13b3-4341-a5a2-e4be2531f5f3-istio-token\" (UniqueName: \"kubernetes.io/projected/317201b4-13b3-4341-a5a2-e4be2531f5f3-istio-token\") pod \"317201b4-13b3-4341-a5a2-e4be2531f5f3\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") "
Oct 30 13:09:47.041299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.015867    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istio-data\") pod \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") "
Oct 30 13:09:47.041299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.015904    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-workload-socket\") pod \"317201b4-13b3-4341-a5a2-e4be2531f5f3\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") "
Oct 30 13:09:47.041299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.015984    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-workload-socket\") pod \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") "
Oct 30 13:09:47.041299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.016099    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-credential-socket\") pod \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") "
Oct 30 13:09:47.041299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.016167    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-kube-api-access-bcjbm\" (UniqueName: \"kubernetes.io/projected/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-kube-api-access-bcjbm\") pod \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\" (UID: \"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6\") "
Oct 30 13:09:47.041299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.016235    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/317201b4-13b3-4341-a5a2-e4be2531f5f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/317201b4-13b3-4341-a5a2-e4be2531f5f3-istio-podinfo\") pod \"317201b4-13b3-4341-a5a2-e4be2531f5f3\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") "
Oct 30 13:09:47.041299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.016388    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-istio-envoy\") pod \"317201b4-13b3-4341-a5a2-e4be2531f5f3\" (UID: \"317201b4-13b3-4341-a5a2-e4be2531f5f3\") "
Oct 30 13:09:47.041299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.017267    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.041299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.017291    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.041299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.017306    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.061666 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.061159    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istiod-ca-cert" pod "41b0db3d-0f0b-4791-aca0-0cd31a7c78f6" (UID: "41b0db3d-0f0b-4791-aca0-0cd31a7c78f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:47.061666 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.061406    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.061666 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.061439    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.061666 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.061455    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.061666 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.061473    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.061666 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.061489    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr79h\" (UniqueName: \"kubernetes.io/projected/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-kube-api-access-gr79h\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.061666 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.061505    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b2c1ca-75ce-45bb-93bc-d1da88552cf5-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.061666 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.061560    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-workload-certs" pod "41b0db3d-0f0b-4791-aca0-0cd31a7c78f6" (UID: "41b0db3d-0f0b-4791-aca0-0cd31a7c78f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:47.062161 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.061769    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-istio-data" pod "317201b4-13b3-4341-a5a2-e4be2531f5f3" (UID: "317201b4-13b3-4341-a5a2-e4be2531f5f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:47.062551 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.062384    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/317201b4-13b3-4341-a5a2-e4be2531f5f3-istiod-ca-cert" pod "317201b4-13b3-4341-a5a2-e4be2531f5f3" (UID: "317201b4-13b3-4341-a5a2-e4be2531f5f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:47.062676 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.062648    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-credential-socket" pod "317201b4-13b3-4341-a5a2-e4be2531f5f3" (UID: "317201b4-13b3-4341-a5a2-e4be2531f5f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:47.063420 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.063244    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-workload-socket" pod "317201b4-13b3-4341-a5a2-e4be2531f5f3" (UID: "317201b4-13b3-4341-a5a2-e4be2531f5f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:47.063570 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.063433    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-workload-certs" pod "317201b4-13b3-4341-a5a2-e4be2531f5f3" (UID: "317201b4-13b3-4341-a5a2-e4be2531f5f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:47.064357 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.064064    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-credential-socket" pod "41b0db3d-0f0b-4791-aca0-0cd31a7c78f6" (UID: "41b0db3d-0f0b-4791-aca0-0cd31a7c78f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:47.064357 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.064149    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-workload-socket" pod "41b0db3d-0f0b-4791-aca0-0cd31a7c78f6" (UID: "41b0db3d-0f0b-4791-aca0-0cd31a7c78f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:47.064607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.064530    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istio-data" pod "41b0db3d-0f0b-4791-aca0-0cd31a7c78f6" (UID: "41b0db3d-0f0b-4791-aca0-0cd31a7c78f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:47.066916 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.066882    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-kube-api-access-bcjbm" pod "41b0db3d-0f0b-4791-aca0-0cd31a7c78f6" (UID: "41b0db3d-0f0b-4791-aca0-0cd31a7c78f6"). InnerVolumeSpecName "kube-api-access-bcjbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:47.067890 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.067458    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/317201b4-13b3-4341-a5a2-e4be2531f5f3-istio-podinfo" pod "317201b4-13b3-4341-a5a2-e4be2531f5f3" (UID: "317201b4-13b3-4341-a5a2-e4be2531f5f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:09:47.071521 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.071485    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/317201b4-13b3-4341-a5a2-e4be2531f5f3-kube-api-access-tkxk4" pod "317201b4-13b3-4341-a5a2-e4be2531f5f3" (UID: "317201b4-13b3-4341-a5a2-e4be2531f5f3"). InnerVolumeSpecName "kube-api-access-tkxk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:47.072920 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.072887    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istio-podinfo" pod "41b0db3d-0f0b-4791-aca0-0cd31a7c78f6" (UID: "41b0db3d-0f0b-4791-aca0-0cd31a7c78f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:09:47.073107 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.073083    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istio-token" pod "41b0db3d-0f0b-4791-aca0-0cd31a7c78f6" (UID: "41b0db3d-0f0b-4791-aca0-0cd31a7c78f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:47.073341 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.073302    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-istio-envoy" pod "317201b4-13b3-4341-a5a2-e4be2531f5f3" (UID: "317201b4-13b3-4341-a5a2-e4be2531f5f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:47.074555 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.074526    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istio-envoy" pod "41b0db3d-0f0b-4791-aca0-0cd31a7c78f6" (UID: "41b0db3d-0f0b-4791-aca0-0cd31a7c78f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:47.076188 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.076159    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/317201b4-13b3-4341-a5a2-e4be2531f5f3-istio-token" pod "317201b4-13b3-4341-a5a2-e4be2531f5f3" (UID: "317201b4-13b3-4341-a5a2-e4be2531f5f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:47.081197 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.081174    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:09:47.179522 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.163338    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.179522 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.163370    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.179522 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.163388    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcjbm\" (UniqueName: \"kubernetes.io/projected/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-kube-api-access-bcjbm\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.179522 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.167034    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/317201b4-13b3-4341-a5a2-e4be2531f5f3-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.179522 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.167070    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.179522 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.167085    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.179522 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.167100    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.179522 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.167115    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/317201b4-13b3-4341-a5a2-e4be2531f5f3-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.179522 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.167128    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.179522 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.167143    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkxk4\" (UniqueName: \"kubernetes.io/projected/317201b4-13b3-4341-a5a2-e4be2531f5f3-kube-api-access-tkxk4\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.179522 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.167157    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.179522 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.167172    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.179522 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.167186    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.179522 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.167201    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.179522 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.167250    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.179522 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.167267    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/317201b4-13b3-4341-a5a2-e4be2531f5f3-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.179522 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.167281    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.179522 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.167295    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/317201b4-13b3-4341-a5a2-e4be2531f5f3-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.270628 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.270592    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istiod-ca-cert\") pod \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") "
Oct 30 13:09:47.270838 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.270659    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istio-token\" (UniqueName: \"kubernetes.io/projected/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istio-token\") pod \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") "
Oct 30 13:09:47.270838 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.270689    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istio-data\") pod \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") "
Oct 30 13:09:47.270838 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.270713    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-workload-certs\") pod \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") "
Oct 30 13:09:47.270838 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.270741    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-credential-socket\") pod \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") "
Oct 30 13:09:47.271078 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.270877    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-kube-api-access-gm87t\" (UniqueName: \"kubernetes.io/projected/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-kube-api-access-gm87t\") pod \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") "
Oct 30 13:09:47.271078 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.270941    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-workload-socket\") pod \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") "
Oct 30 13:09:47.271078 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.270961    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istio-envoy\") pod \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") "
Oct 30 13:09:47.271078 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.270997    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istiod-ca-cert" pod "fa2c83b0-34e5-4b64-ba1c-b38f86978e24" (UID: "fa2c83b0-34e5-4b64-ba1c-b38f86978e24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:47.271078 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.271049    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istio-podinfo\") pod \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\" (UID: \"fa2c83b0-34e5-4b64-ba1c-b38f86978e24\") "
Oct 30 13:09:47.271378 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.271203    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-workload-certs" pod "fa2c83b0-34e5-4b64-ba1c-b38f86978e24" (UID: "fa2c83b0-34e5-4b64-ba1c-b38f86978e24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:47.271576 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.271560    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.271655 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.271578    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.272710 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.272170    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-workload-socket" pod "fa2c83b0-34e5-4b64-ba1c-b38f86978e24" (UID: "fa2c83b0-34e5-4b64-ba1c-b38f86978e24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:47.272710 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.272214    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istio-data" pod "fa2c83b0-34e5-4b64-ba1c-b38f86978e24" (UID: "fa2c83b0-34e5-4b64-ba1c-b38f86978e24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:47.273236 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.273200    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-credential-socket" pod "fa2c83b0-34e5-4b64-ba1c-b38f86978e24" (UID: "fa2c83b0-34e5-4b64-ba1c-b38f86978e24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:47.275500 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.273909    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istio-token" pod "fa2c83b0-34e5-4b64-ba1c-b38f86978e24" (UID: "fa2c83b0-34e5-4b64-ba1c-b38f86978e24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:47.275500 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.274407    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-kube-api-access-gm87t" pod "fa2c83b0-34e5-4b64-ba1c-b38f86978e24" (UID: "fa2c83b0-34e5-4b64-ba1c-b38f86978e24"). InnerVolumeSpecName "kube-api-access-gm87t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:47.275500 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.275267    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istio-podinfo" pod "fa2c83b0-34e5-4b64-ba1c-b38f86978e24" (UID: "fa2c83b0-34e5-4b64-ba1c-b38f86978e24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:09:47.276537 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.276498    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istio-envoy" pod "fa2c83b0-34e5-4b64-ba1c-b38f86978e24" (UID: "fa2c83b0-34e5-4b64-ba1c-b38f86978e24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:47.372785 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.372743    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm87t\" (UniqueName: \"kubernetes.io/projected/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-kube-api-access-gm87t\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.372785 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.372780    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.372785 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.372791    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.373060 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.372800    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.373060 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.372808    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.373060 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.372816    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.373060 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.372823    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2c83b0-34e5-4b64-ba1c-b38f86978e24-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:47.520605 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.512457    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"
Oct 30 13:09:47.520605 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.513259    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg" event={"ID":"fa2c83b0-34e5-4b64-ba1c-b38f86978e24","Type":"ContainerDied","Data":"9daf87946886a2c1fd498ba76060a844e4186aa1efc42de27cf0a0aff74d2ce9"}
Oct 30 13:09:47.520605 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.513331    4544 scope.go:122] "RemoveContainer" containerID="5581df816922fa36c10d2cc3d7d0eb62a002e34ac63bf23f81e187283e043ced"
Oct 30 13:09:47.520605 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.518486    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455" event={"ID":"41b0db3d-0f0b-4791-aca0-0cd31a7c78f6","Type":"ContainerDied","Data":"5f9bebd29999c575aabd41af53b9cfb2bbed8a3608ebcf80f275a122ad7462b1"}
Oct 30 13:09:47.520605 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.518605    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"
Oct 30 13:09:47.528189 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.527982    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl" event={"ID":"89b2c1ca-75ce-45bb-93bc-d1da88552cf5","Type":"ContainerDied","Data":"8b55b923aa981535ce9961283c94667a73833e123af62404bdc1fcc10358c842"}
Oct 30 13:09:47.528189 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.528106    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"
Oct 30 13:09:47.545670 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.545591    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7" event={"ID":"317201b4-13b3-4341-a5a2-e4be2531f5f3","Type":"ContainerDied","Data":"33ead284a0a4a18514f432c2ad2d06ffd11bcf7563a31598d4b988e0481ed783"}
Oct 30 13:09:47.545801 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.545705    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"
Oct 30 13:09:47.549835 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.549813    4544 scope.go:122] "RemoveContainer" containerID="77286f3d171b3dd8acf2c5fb348c27c083f88db45475d778e6fc06b2681d66b6"
Oct 30 13:09:47.563116 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.563092    4544 scope.go:122] "RemoveContainer" containerID="aabf38611462548a1a9eccbca9a5352f764aac7b6afc6de687d730ea0818bfd8"
Oct 30 13:09:47.576327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.576298    4544 scope.go:122] "RemoveContainer" containerID="93bd1fd2fea5b74865f234e18c25a3615303d6b9db471c2efc2900b3436fe170"
Oct 30 13:09:47.601792 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.601749    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"]
Oct 30 13:09:47.607187 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.607160    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455"]
Oct 30 13:09:47.648668 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.648584    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"]
Oct 30 13:09:47.653246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.653199    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl"]
Oct 30 13:09:47.692604 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.692572    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"]
Oct 30 13:09:47.700818 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.700786    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg"]
Oct 30 13:09:47.757311 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.757278    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"]
Oct 30 13:09:47.763236 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:47.763190    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7"]
Oct 30 13:09:48.448839 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:48.448802    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:49.090322 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:49.090282    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="317201b4-13b3-4341-a5a2-e4be2531f5f3" path="/var/lib/kubelet/pods/317201b4-13b3-4341-a5a2-e4be2531f5f3/volumes"
Oct 30 13:09:49.091046 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:49.091029    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41b0db3d-0f0b-4791-aca0-0cd31a7c78f6" path="/var/lib/kubelet/pods/41b0db3d-0f0b-4791-aca0-0cd31a7c78f6/volumes"
Oct 30 13:09:49.091696 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:49.091679    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89b2c1ca-75ce-45bb-93bc-d1da88552cf5" path="/var/lib/kubelet/pods/89b2c1ca-75ce-45bb-93bc-d1da88552cf5/volumes"
Oct 30 13:09:49.092759 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:49.092739    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa2c83b0-34e5-4b64-ba1c-b38f86978e24" path="/var/lib/kubelet/pods/fa2c83b0-34e5-4b64-ba1c-b38f86978e24/volumes"
Oct 30 13:09:49.134243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:49.134193    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9074-7845db"]
Oct 30 13:09:49.135034 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:49.135017    4544 cpu_manager.go:414] "containerMap: removing container" podUID="41b0db3d-0f0b-4791-aca0-0cd31a7c78f6" containerName="istio-proxy"
Oct 30 13:09:49.135082 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:49.135036    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41b0db3d-0f0b-4791-aca0-0cd31a7c78f6" containerName="istio-proxy"
Oct 30 13:09:49.135082 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:49.135050    4544 cpu_manager.go:414] "containerMap: removing container" podUID="317201b4-13b3-4341-a5a2-e4be2531f5f3" containerName="istio-proxy"
Oct 30 13:09:49.135082 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:49.135055    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="317201b4-13b3-4341-a5a2-e4be2531f5f3" containerName="istio-proxy"
Oct 30 13:09:49.135082 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:49.135070    4544 cpu_manager.go:414] "containerMap: removing container" podUID="89b2c1ca-75ce-45bb-93bc-d1da88552cf5" containerName="istio-proxy"
Oct 30 13:09:49.135082 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:49.135076    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89b2c1ca-75ce-45bb-93bc-d1da88552cf5" containerName="istio-proxy"
Oct 30 13:09:49.135265 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:49.135101    4544 cpu_manager.go:414] "containerMap: removing container" podUID="fa2c83b0-34e5-4b64-ba1c-b38f86978e24" containerName="istio-proxy"
Oct 30 13:09:49.135265 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:49.135106    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa2c83b0-34e5-4b64-ba1c-b38f86978e24" containerName="istio-proxy"
Oct 30 13:09:49.135265 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:49.135215    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa2c83b0-34e5-4b64-ba1c-b38f86978e24" containerName="istio-proxy"
Oct 30 13:09:49.135265 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:49.135241    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="41b0db3d-0f0b-4791-aca0-0cd31a7c78f6" containerName="istio-proxy"
Oct 30 13:09:49.135265 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:49.135248    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="317201b4-13b3-4341-a5a2-e4be2531f5f3" containerName="istio-proxy"
Oct 30 13:09:49.135265 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:49.135256    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="89b2c1ca-75ce-45bb-93bc-d1da88552cf5" containerName="istio-proxy"
Oct 30 13:09:49.135265 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:49.135261    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="41b0db3d-0f0b-4791-aca0-0cd31a7c78f6" containerName="istio-proxy"
Oct 30 13:09:49.141316 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:49.141293    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9074-7845db"
Oct 30 13:09:49.158308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:49.158274    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9074-7845db"]
Oct 30 13:09:49.166743 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:49.166711    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8xhdg\""
Oct 30 13:09:49.215672 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:49.215560    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kvfb\" (UniqueName: \"kubernetes.io/projected/7f5afa36-1940-47f3-b1bd-72f3db1ece3c-kube-api-access-5kvfb\") pod \"test-curl-9074-7845db\" (UID: \"7f5afa36-1940-47f3-b1bd-72f3db1ece3c\") " pod="opendatahub/test-curl-9074-7845db"
Oct 30 13:09:49.317180 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:49.317128    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kvfb\" (UniqueName: \"kubernetes.io/projected/7f5afa36-1940-47f3-b1bd-72f3db1ece3c-kube-api-access-5kvfb\") pod \"test-curl-9074-7845db\" (UID: \"7f5afa36-1940-47f3-b1bd-72f3db1ece3c\") " pod="opendatahub/test-curl-9074-7845db"
Oct 30 13:09:49.340129 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:49.340093    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kvfb\" (UniqueName: \"kubernetes.io/projected/7f5afa36-1940-47f3-b1bd-72f3db1ece3c-kube-api-access-5kvfb\") pod \"test-curl-9074-7845db\" (UID: \"7f5afa36-1940-47f3-b1bd-72f3db1ece3c\") " pod="opendatahub/test-curl-9074-7845db"
Oct 30 13:09:49.457191 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:49.457146    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9074-7845db"
Oct 30 13:09:49.762617 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:49.762585    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9074-7845db"]
Oct 30 13:09:49.764690 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:09:49.764664    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f5afa36_1940_47f3_b1bd_72f3db1ece3c.slice/crio-111cd09718df3ef5d53de16158ff4584955644ee80c6e1f6b86d4965a010ab5c WatchSource:0}: Error finding container 111cd09718df3ef5d53de16158ff4584955644ee80c6e1f6b86d4965a010ab5c: Status 404 returned error can't find the container with id 111cd09718df3ef5d53de16158ff4584955644ee80c6e1f6b86d4965a010ab5c
Oct 30 13:09:50.186563 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:50.186527    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:50.668044 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:50.668007    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9074-7845db" event={"ID":"7f5afa36-1940-47f3-b1bd-72f3db1ece3c","Type":"ContainerStarted","Data":"f25b8dd8a51c3106dd3852ff525f7a9ba06467f9c83e4e11613f4917d76a4a5f"}
Oct 30 13:09:50.668460 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:50.668056    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9074-7845db" event={"ID":"7f5afa36-1940-47f3-b1bd-72f3db1ece3c","Type":"ContainerStarted","Data":"111cd09718df3ef5d53de16158ff4584955644ee80c6e1f6b86d4965a010ab5c"}
Oct 30 13:09:50.717196 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:50.717136    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9074-7845db" podStartSLOduration=1.7171210399999999 podStartE2EDuration="1.71712104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:50.711891985 +0000 UTC m=+1485.759025938" watchObservedRunningTime="2026-10-30 13:09:50.71712104 +0000 UTC m=+1485.764254992"
Oct 30 13:09:51.694447 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:51.694391    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9074-7845db"]
Oct 30 13:09:51.703928 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:51.703887    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9074-7845db"]
Oct 30 13:09:52.274821 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:52.274785    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:52.314247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:52.302791    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-2rblx"]
Oct 30 13:09:52.326032 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:52.322373    4544 cpu_manager.go:414] "containerMap: removing container" podUID="41b0db3d-0f0b-4791-aca0-0cd31a7c78f6" containerName="istio-proxy"
Oct 30 13:09:52.326032 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:52.322409    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41b0db3d-0f0b-4791-aca0-0cd31a7c78f6" containerName="istio-proxy"
Oct 30 13:09:52.326032 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:52.322456    4544 cpu_manager.go:414] "containerMap: removing container" podUID="7f5afa36-1940-47f3-b1bd-72f3db1ece3c" containerName="test-curl-9074-7845db"
Oct 30 13:09:52.326032 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:52.322464    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f5afa36-1940-47f3-b1bd-72f3db1ece3c" containerName="test-curl-9074-7845db"
Oct 30 13:09:52.326032 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:52.322702    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f5afa36-1940-47f3-b1bd-72f3db1ece3c" containerName="test-curl-9074-7845db"
Oct 30 13:09:52.365336 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:52.351898    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-2rblx"
Oct 30 13:09:52.366568 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:52.366475    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:09:52.386062 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:52.382704    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:09:52.402276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:52.395943    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-2rblx"]
Oct 30 13:09:52.497807 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:52.497769    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrnqw\" (UniqueName: \"kubernetes.io/projected/00130766-10c7-4dab-ab14-99afeb886aa6-kube-api-access-lrnqw\") pod \"maas-api-revoke-keys-e2e-mt-1c0e6c42-2rblx\" (UID: \"00130766-10c7-4dab-ab14-99afeb886aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-2rblx"
Oct 30 13:09:52.498157 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:52.498128    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00130766-10c7-4dab-ab14-99afeb886aa6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1c0e6c42-2rblx\" (UID: \"00130766-10c7-4dab-ab14-99afeb886aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-2rblx"
Oct 30 13:09:52.498352 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:52.498329    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00130766-10c7-4dab-ab14-99afeb886aa6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1c0e6c42-2rblx\" (UID: \"00130766-10c7-4dab-ab14-99afeb886aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-2rblx"
Oct 30 13:09:52.599662 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:52.599615    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00130766-10c7-4dab-ab14-99afeb886aa6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1c0e6c42-2rblx\" (UID: \"00130766-10c7-4dab-ab14-99afeb886aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-2rblx"
Oct 30 13:09:52.599853 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:52.599757    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrnqw\" (UniqueName: \"kubernetes.io/projected/00130766-10c7-4dab-ab14-99afeb886aa6-kube-api-access-lrnqw\") pod \"maas-api-revoke-keys-e2e-mt-1c0e6c42-2rblx\" (UID: \"00130766-10c7-4dab-ab14-99afeb886aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-2rblx"
Oct 30 13:09:52.600025 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:52.600008    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00130766-10c7-4dab-ab14-99afeb886aa6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1c0e6c42-2rblx\" (UID: \"00130766-10c7-4dab-ab14-99afeb886aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-2rblx"
Oct 30 13:09:52.600429 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:52.600402    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00130766-10c7-4dab-ab14-99afeb886aa6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1c0e6c42-2rblx\" (UID: \"00130766-10c7-4dab-ab14-99afeb886aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-2rblx"
Oct 30 13:09:52.602464 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:52.602446    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00130766-10c7-4dab-ab14-99afeb886aa6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1c0e6c42-2rblx\" (UID: \"00130766-10c7-4dab-ab14-99afeb886aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-2rblx"
Oct 30 13:09:52.623065 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:52.623034    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrnqw\" (UniqueName: \"kubernetes.io/projected/00130766-10c7-4dab-ab14-99afeb886aa6-kube-api-access-lrnqw\") pod \"maas-api-revoke-keys-e2e-mt-1c0e6c42-2rblx\" (UID: \"00130766-10c7-4dab-ab14-99afeb886aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-2rblx"
Oct 30 13:09:52.686526 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:52.686484    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9074-7845db" podUID="7f5afa36-1940-47f3-b1bd-72f3db1ece3c" containerName="test-curl-9074-7845db" containerID="cri-o://f25b8dd8a51c3106dd3852ff525f7a9ba06467f9c83e4e11613f4917d76a4a5f" gracePeriod=2
Oct 30 13:09:52.699727 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:09:52.699690    4544 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9074-7845db\" is forbidden: User \"system:node:ip-10-0-143-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-38.ec2.internal' and this object" podUID="7f5afa36-1940-47f3-b1bd-72f3db1ece3c" pod="opendatahub/test-curl-9074-7845db"
Oct 30 13:09:52.741584 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:52.741551    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-2rblx"
Oct 30 13:09:53.064258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:53.064230    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-2rblx"]
Oct 30 13:09:53.066595 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:09:53.066566    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00130766_10c7_4dab_ab14_99afeb886aa6.slice/crio-ec4ae213bedbf98e8e5761f0d387d845e7cb56e8bc8a5b72001178547422c59d WatchSource:0}: Error finding container ec4ae213bedbf98e8e5761f0d387d845e7cb56e8bc8a5b72001178547422c59d: Status 404 returned error can't find the container with id ec4ae213bedbf98e8e5761f0d387d845e7cb56e8bc8a5b72001178547422c59d
Oct 30 13:09:53.696604 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:53.696572    4544 generic.go:356] "Generic (PLEG): container finished" podID="65e7c2e0-231b-4455-b017-995e6691cf31" containerID="0ed787d6496b5ec272bc824ffd4aad4e76a5fe8227a8cf4b830090292cbdd1a2" exitCode=0
Oct 30 13:09:53.696793 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:53.696643    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg" event={"ID":"65e7c2e0-231b-4455-b017-995e6691cf31","Type":"ContainerDied","Data":"0ed787d6496b5ec272bc824ffd4aad4e76a5fe8227a8cf4b830090292cbdd1a2"}
Oct 30 13:09:53.697202 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:53.697182    4544 scope.go:122] "RemoveContainer" containerID="0ed787d6496b5ec272bc824ffd4aad4e76a5fe8227a8cf4b830090292cbdd1a2"
Oct 30 13:09:53.698526 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:53.698502    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-2rblx" event={"ID":"00130766-10c7-4dab-ab14-99afeb886aa6","Type":"ContainerStarted","Data":"9f90330de2f8b30896bb0655a6865d51c6a3f94e5dd491ca35a7111112eddf6b"}
Oct 30 13:09:53.698607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:53.698554    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-2rblx" event={"ID":"00130766-10c7-4dab-ab14-99afeb886aa6","Type":"ContainerStarted","Data":"ec4ae213bedbf98e8e5761f0d387d845e7cb56e8bc8a5b72001178547422c59d"}
Oct 30 13:09:53.699189 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:53.699163    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:09:53.699280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:53.699264    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:09:53.699327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:53.699301    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:09:53.794997 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:53.794928    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-2rblx" podStartSLOduration=1.7949086969999999 podStartE2EDuration="1.794908697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:53.794391919 +0000 UTC m=+1488.841525872" watchObservedRunningTime="2026-10-30 13:09:53.794908697 +0000 UTC m=+1488.842042659"
Oct 30 13:09:54.143878 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:54.143841    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv"]
Oct 30 13:09:54.148619 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:54.148591    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv"
Oct 30 13:09:54.164906 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:54.164878    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-75abce-serving-cert\""
Oct 30 13:09:54.202635 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:54.202595    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv"]
Oct 30 13:09:54.227575 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:54.227535    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2df23af-713d-4c5c-9d63-5999de49d67e-metrics-certs\") pod \"maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv\" (UID: \"e2df23af-713d-4c5c-9d63-5999de49d67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv"
Oct 30 13:09:54.227753 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:54.227657    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2df23af-713d-4c5c-9d63-5999de49d67e-openshift-service-ca\") pod \"maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv\" (UID: \"e2df23af-713d-4c5c-9d63-5999de49d67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv"
Oct 30 13:09:54.227753 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:54.227743    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2df23af-713d-4c5c-9d63-5999de49d67e-maas-api-tls\") pod \"maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv\" (UID: \"e2df23af-713d-4c5c-9d63-5999de49d67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv"
Oct 30 13:09:54.227927 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:54.227877    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f778\" (UniqueName: \"kubernetes.io/projected/e2df23af-713d-4c5c-9d63-5999de49d67e-kube-api-access-6f778\") pod \"maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv\" (UID: \"e2df23af-713d-4c5c-9d63-5999de49d67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv"
Oct 30 13:09:54.241504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:54.241470    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:54.339357 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:54.339299    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2df23af-713d-4c5c-9d63-5999de49d67e-metrics-certs\") pod \"maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv\" (UID: \"e2df23af-713d-4c5c-9d63-5999de49d67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv"
Oct 30 13:09:54.339583 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:54.339408    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2df23af-713d-4c5c-9d63-5999de49d67e-openshift-service-ca\") pod \"maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv\" (UID: \"e2df23af-713d-4c5c-9d63-5999de49d67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv"
Oct 30 13:09:54.339583 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:54.339512    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2df23af-713d-4c5c-9d63-5999de49d67e-maas-api-tls\") pod \"maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv\" (UID: \"e2df23af-713d-4c5c-9d63-5999de49d67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv"
Oct 30 13:09:54.339695 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:54.339625    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f778\" (UniqueName: \"kubernetes.io/projected/e2df23af-713d-4c5c-9d63-5999de49d67e-kube-api-access-6f778\") pod \"maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv\" (UID: \"e2df23af-713d-4c5c-9d63-5999de49d67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv"
Oct 30 13:09:54.342275 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:54.342241    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2df23af-713d-4c5c-9d63-5999de49d67e-openshift-service-ca\") pod \"maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv\" (UID: \"e2df23af-713d-4c5c-9d63-5999de49d67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv"
Oct 30 13:09:54.344024 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:54.343998    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2df23af-713d-4c5c-9d63-5999de49d67e-metrics-certs\") pod \"maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv\" (UID: \"e2df23af-713d-4c5c-9d63-5999de49d67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv"
Oct 30 13:09:54.344627 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:54.344609    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2df23af-713d-4c5c-9d63-5999de49d67e-maas-api-tls\") pod \"maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv\" (UID: \"e2df23af-713d-4c5c-9d63-5999de49d67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv"
Oct 30 13:09:54.424247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:54.395101    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f778\" (UniqueName: \"kubernetes.io/projected/e2df23af-713d-4c5c-9d63-5999de49d67e-kube-api-access-6f778\") pod \"maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv\" (UID: \"e2df23af-713d-4c5c-9d63-5999de49d67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv"
Oct 30 13:09:54.424591 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:54.414465    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:54.463928 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:54.463890    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv"
Oct 30 13:09:54.710536 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:54.710507    4544 generic.go:356] "Generic (PLEG): container finished" podID="7f5afa36-1940-47f3-b1bd-72f3db1ece3c" containerID="f25b8dd8a51c3106dd3852ff525f7a9ba06467f9c83e4e11613f4917d76a4a5f" exitCode=137
Oct 30 13:09:54.744319 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:54.738934    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg" event={"ID":"65e7c2e0-231b-4455-b017-995e6691cf31","Type":"ContainerStarted","Data":"ceb3727fa65a96d13a78c790ea362893510032fc0a7dc9625cba29c0564297be"}
Oct 30 13:09:54.747967 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:54.747644    4544 generic.go:356] "Generic (PLEG): container finished" podID="00130766-10c7-4dab-ab14-99afeb886aa6" containerID="9f90330de2f8b30896bb0655a6865d51c6a3f94e5dd491ca35a7111112eddf6b" exitCode=0
Oct 30 13:09:54.747967 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:54.747739    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-2rblx" event={"ID":"00130766-10c7-4dab-ab14-99afeb886aa6","Type":"ContainerDied","Data":"9f90330de2f8b30896bb0655a6865d51c6a3f94e5dd491ca35a7111112eddf6b"}
Oct 30 13:09:54.906326 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:54.906242    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv"]
Oct 30 13:09:54.940261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:54.938371    4544 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:09:54.995366 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:54.995337    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9074-7845db"
Oct 30 13:09:55.069169 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:55.069128    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f5afa36-1940-47f3-b1bd-72f3db1ece3c-kube-api-access-5kvfb\" (UniqueName: \"kubernetes.io/projected/7f5afa36-1940-47f3-b1bd-72f3db1ece3c-kube-api-access-5kvfb\") pod \"7f5afa36-1940-47f3-b1bd-72f3db1ece3c\" (UID: \"7f5afa36-1940-47f3-b1bd-72f3db1ece3c\") "
Oct 30 13:09:55.071409 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:55.071374    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f5afa36-1940-47f3-b1bd-72f3db1ece3c-kube-api-access-5kvfb" pod "7f5afa36-1940-47f3-b1bd-72f3db1ece3c" (UID: "7f5afa36-1940-47f3-b1bd-72f3db1ece3c"). InnerVolumeSpecName "kube-api-access-5kvfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:55.098690 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:55.098650    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f5afa36-1940-47f3-b1bd-72f3db1ece3c" path="/var/lib/kubelet/pods/7f5afa36-1940-47f3-b1bd-72f3db1ece3c/volumes"
Oct 30 13:09:55.171146 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:55.171053    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kvfb\" (UniqueName: \"kubernetes.io/projected/7f5afa36-1940-47f3-b1bd-72f3db1ece3c-kube-api-access-5kvfb\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:55.471259 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:55.471163    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:09:55.472328 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:55.472297    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 13:09:55.472440 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:55.472353    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg" podUID="65e7c2e0-231b-4455-b017-995e6691cf31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 13:09:55.809303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:55.808916    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv" event={"ID":"e2df23af-713d-4c5c-9d63-5999de49d67e","Type":"ContainerStarted","Data":"76669f4398827ed3a4f0d4db40ed05a178f7689b6da92930ab64c335aa2b9590"}
Oct 30 13:09:55.809303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:55.808988    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv" event={"ID":"e2df23af-713d-4c5c-9d63-5999de49d67e","Type":"ContainerStarted","Data":"f7080cd3e8128a3248033b923a3f12f2659b0f5c471ffed64324aab5c8f00c53"}
Oct 30 13:09:55.817279 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:55.810277    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv"
Oct 30 13:09:55.843951 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:55.841010    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9074-7845db"
Oct 30 13:09:55.843951 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:55.841793    4544 scope.go:122] "RemoveContainer" containerID="f25b8dd8a51c3106dd3852ff525f7a9ba06467f9c83e4e11613f4917d76a4a5f"
Oct 30 13:09:55.933319 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:55.932891    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv" podStartSLOduration=1.529342659 podStartE2EDuration="1.932858244s" totalImagesPullingTime="403.515585ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:55.918705501 +0000 UTC m=+1490.965839455" watchObservedRunningTime="2026-10-30 13:09:55.932858244 +0000 UTC m=+1490.979992200"
Oct 30 13:09:56.396807 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:56.396773    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-2rblx"
Oct 30 13:09:56.472506 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:56.472471    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 13:09:56.472723 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:56.472536    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg" podUID="65e7c2e0-231b-4455-b017-995e6691cf31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 13:09:56.561164 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:56.561127    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00130766-10c7-4dab-ab14-99afeb886aa6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00130766-10c7-4dab-ab14-99afeb886aa6-maas-api-openshift-ca\") pod \"00130766-10c7-4dab-ab14-99afeb886aa6\" (UID: \"00130766-10c7-4dab-ab14-99afeb886aa6\") "
Oct 30 13:09:56.561416 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:56.561177    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00130766-10c7-4dab-ab14-99afeb886aa6-kube-api-access-lrnqw\" (UniqueName: \"kubernetes.io/projected/00130766-10c7-4dab-ab14-99afeb886aa6-kube-api-access-lrnqw\") pod \"00130766-10c7-4dab-ab14-99afeb886aa6\" (UID: \"00130766-10c7-4dab-ab14-99afeb886aa6\") "
Oct 30 13:09:56.561416 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:56.561323    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00130766-10c7-4dab-ab14-99afeb886aa6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00130766-10c7-4dab-ab14-99afeb886aa6-maas-api-xks-ca\") pod \"00130766-10c7-4dab-ab14-99afeb886aa6\" (UID: \"00130766-10c7-4dab-ab14-99afeb886aa6\") "
Oct 30 13:09:56.561649 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:56.561614    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00130766-10c7-4dab-ab14-99afeb886aa6-maas-api-openshift-ca" pod "00130766-10c7-4dab-ab14-99afeb886aa6" (UID: "00130766-10c7-4dab-ab14-99afeb886aa6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:56.562185 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:56.562161    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00130766-10c7-4dab-ab14-99afeb886aa6-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:56.563487 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:56.563461    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00130766-10c7-4dab-ab14-99afeb886aa6-maas-api-xks-ca" pod "00130766-10c7-4dab-ab14-99afeb886aa6" (UID: "00130766-10c7-4dab-ab14-99afeb886aa6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:09:56.563628 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:56.563508    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00130766-10c7-4dab-ab14-99afeb886aa6-kube-api-access-lrnqw" pod "00130766-10c7-4dab-ab14-99afeb886aa6" (UID: "00130766-10c7-4dab-ab14-99afeb886aa6"). InnerVolumeSpecName "kube-api-access-lrnqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:56.611758 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:56.611717    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:56.667308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:56.664518    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrnqw\" (UniqueName: \"kubernetes.io/projected/00130766-10c7-4dab-ab14-99afeb886aa6-kube-api-access-lrnqw\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:56.667308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:56.664563    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00130766-10c7-4dab-ab14-99afeb886aa6-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:09:56.854652 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:56.854616    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-2rblx" event={"ID":"00130766-10c7-4dab-ab14-99afeb886aa6","Type":"ContainerDied","Data":"ec4ae213bedbf98e8e5761f0d387d845e7cb56e8bc8a5b72001178547422c59d"}
Oct 30 13:09:56.854841 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:56.854668    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec4ae213bedbf98e8e5761f0d387d845e7cb56e8bc8a5b72001178547422c59d"
Oct 30 13:09:56.854841 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:56.854724    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-2rblx"
Oct 30 13:09:57.053713 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:57.053534    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e65256d4-89rzm"]
Oct 30 13:09:57.053713 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:57.053603    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e65256d4-89rzm"]
Oct 30 13:09:57.093103 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:57.093069    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f1334f7-9c63-429d-b0e0-b16dc4e8f690" path="/var/lib/kubelet/pods/9f1334f7-9c63-429d-b0e0-b16dc4e8f690/volumes"
Oct 30 13:09:57.472447 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:57.472405    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 13:09:57.472650 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:57.472471    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg" podUID="65e7c2e0-231b-4455-b017-995e6691cf31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 13:09:58.472100 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:58.472006    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 13:09:58.472100 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:58.472071    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg" podUID="65e7c2e0-231b-4455-b017-995e6691cf31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 13:09:58.691322 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:58.691282    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:09:58.730308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:58.725367    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:09:59.008696 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:09:59.008613    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:09:59.472200 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:59.472158    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 13:09:59.472664 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:09:59.472270    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg" podUID="65e7c2e0-231b-4455-b017-995e6691cf31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 13:10:00.477172 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:00.477142    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:10:00.478397 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:00.478373    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:10:00.556787 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:00.556752    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:02.113464 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:02.103303    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0a2cb4b-pt66h"]
Oct 30 13:10:02.138271 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:02.126318    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0a2cb4b-pt66h"]
Oct 30 13:10:02.352326 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:02.351249    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:02.875873 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:02.875841    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv"
Oct 30 13:10:02.998569 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:02.998529    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:03.169697 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:03.169607    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="012f6e7f-8187-4fe2-a2d1-1ae080efed14" path="/var/lib/kubelet/pods/012f6e7f-8187-4fe2-a2d1-1ae080efed14/volumes"
Oct 30 13:10:04.833413 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:04.833375    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:05.649509 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:05.649430    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:05.756233 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:05.756185    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:06.279246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.265269    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"]
Oct 30 13:10:06.279246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.270611    4544 cpu_manager.go:414] "containerMap: removing container" podUID="00130766-10c7-4dab-ab14-99afeb886aa6" containerName="revoke-keys"
Oct 30 13:10:06.279246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.270642    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00130766-10c7-4dab-ab14-99afeb886aa6" containerName="revoke-keys"
Oct 30 13:10:06.279246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.270912    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="00130766-10c7-4dab-ab14-99afeb886aa6" containerName="revoke-keys"
Oct 30 13:10:06.279246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.276004    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:10:06.334463 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.317599    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"]
Oct 30 13:10:06.334838 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.317650    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f3160a-openshift-default-dockercfg-xf52v\""
Oct 30 13:10:06.386248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.368510    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:06.416510 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.416464    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-workload-certs\") pod \"e2e-api-a-f3160a-openshift-default-67c76678c-77hqr\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") " pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:10:06.416782 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.416758    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bbc446d-8807-4613-9d86-b87b0733d2cf-istio-token\") pod \"e2e-api-a-f3160a-openshift-default-67c76678c-77hqr\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") " pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:10:06.416915 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.416897    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pltd\" (UniqueName: \"kubernetes.io/projected/4bbc446d-8807-4613-9d86-b87b0733d2cf-kube-api-access-2pltd\") pod \"e2e-api-a-f3160a-openshift-default-67c76678c-77hqr\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") " pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:10:06.421741 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.417119    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-workload-socket\") pod \"e2e-api-a-f3160a-openshift-default-67c76678c-77hqr\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") " pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:10:06.446246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.425503    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-credential-socket\") pod \"e2e-api-a-f3160a-openshift-default-67c76678c-77hqr\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") " pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:10:06.452255 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.450640    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-istio-envoy\") pod \"e2e-api-a-f3160a-openshift-default-67c76678c-77hqr\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") " pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:10:06.452255 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.450704    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-istio-data\") pod \"e2e-api-a-f3160a-openshift-default-67c76678c-77hqr\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") " pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:10:06.452255 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.450901    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bbc446d-8807-4613-9d86-b87b0733d2cf-istiod-ca-cert\") pod \"e2e-api-a-f3160a-openshift-default-67c76678c-77hqr\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") " pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:10:06.452255 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.450971    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bbc446d-8807-4613-9d86-b87b0733d2cf-istio-podinfo\") pod \"e2e-api-a-f3160a-openshift-default-67c76678c-77hqr\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") " pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:10:06.670938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.668566    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bbc446d-8807-4613-9d86-b87b0733d2cf-istiod-ca-cert\") pod \"e2e-api-a-f3160a-openshift-default-67c76678c-77hqr\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") " pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:10:06.670938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.668641    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bbc446d-8807-4613-9d86-b87b0733d2cf-istio-podinfo\") pod \"e2e-api-a-f3160a-openshift-default-67c76678c-77hqr\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") " pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:10:06.670938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.668730    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-workload-certs\") pod \"e2e-api-a-f3160a-openshift-default-67c76678c-77hqr\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") " pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:10:06.670938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.668762    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bbc446d-8807-4613-9d86-b87b0733d2cf-istio-token\") pod \"e2e-api-a-f3160a-openshift-default-67c76678c-77hqr\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") " pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:10:06.670938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.668792    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pltd\" (UniqueName: \"kubernetes.io/projected/4bbc446d-8807-4613-9d86-b87b0733d2cf-kube-api-access-2pltd\") pod \"e2e-api-a-f3160a-openshift-default-67c76678c-77hqr\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") " pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:10:06.670938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.668919    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-workload-socket\") pod \"e2e-api-a-f3160a-openshift-default-67c76678c-77hqr\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") " pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:10:06.670938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.669092    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-credential-socket\") pod \"e2e-api-a-f3160a-openshift-default-67c76678c-77hqr\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") " pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:10:06.670938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.669135    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-istio-envoy\") pod \"e2e-api-a-f3160a-openshift-default-67c76678c-77hqr\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") " pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:10:06.670938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.669165    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-istio-data\") pod \"e2e-api-a-f3160a-openshift-default-67c76678c-77hqr\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") " pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:10:06.671730 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.671685    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-workload-socket\") pod \"e2e-api-a-f3160a-openshift-default-67c76678c-77hqr\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") " pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:10:06.692314 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.674258    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-workload-certs\") pod \"e2e-api-a-f3160a-openshift-default-67c76678c-77hqr\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") " pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:10:06.692314 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.679456    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-credential-socket\") pod \"e2e-api-a-f3160a-openshift-default-67c76678c-77hqr\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") " pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:10:06.692314 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.684736    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bbc446d-8807-4613-9d86-b87b0733d2cf-istiod-ca-cert\") pod \"e2e-api-a-f3160a-openshift-default-67c76678c-77hqr\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") " pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:10:06.693053 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.693025    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-istio-data\") pod \"e2e-api-a-f3160a-openshift-default-67c76678c-77hqr\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") " pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:10:06.759851 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.758292    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bbc446d-8807-4613-9d86-b87b0733d2cf-istio-token\") pod \"e2e-api-a-f3160a-openshift-default-67c76678c-77hqr\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") " pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:10:06.759851 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.758414    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bbc446d-8807-4613-9d86-b87b0733d2cf-istio-podinfo\") pod \"e2e-api-a-f3160a-openshift-default-67c76678c-77hqr\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") " pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:10:06.828386 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.828332    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9077-095df6"]
Oct 30 13:10:06.880243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.871163    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-istio-envoy\") pod \"e2e-api-a-f3160a-openshift-default-67c76678c-77hqr\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") " pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:10:06.880243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.877454    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9077-095df6"]
Oct 30 13:10:06.880243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.877587    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9077-095df6"
Oct 30 13:10:06.885957 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.885919    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pltd\" (UniqueName: \"kubernetes.io/projected/4bbc446d-8807-4613-9d86-b87b0733d2cf-kube-api-access-2pltd\") pod \"e2e-api-a-f3160a-openshift-default-67c76678c-77hqr\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") " pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:10:06.893118 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.893044    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:06.896390 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:06.896358    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8xhdg\""
Oct 30 13:10:07.094900 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:07.094869    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5dmw\" (UniqueName: \"kubernetes.io/projected/a75fed60-ae62-40be-967d-1961b7d41b68-kube-api-access-m5dmw\") pod \"test-curl-9077-095df6\" (UID: \"a75fed60-ae62-40be-967d-1961b7d41b68\") " pod="opendatahub/test-curl-9077-095df6"
Oct 30 13:10:07.107961 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:07.107929    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:07.108978 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:07.108955    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:10:07.180939 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:07.180905    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls"]
Oct 30 13:10:07.210963 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:07.210921    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5dmw\" (UniqueName: \"kubernetes.io/projected/a75fed60-ae62-40be-967d-1961b7d41b68-kube-api-access-m5dmw\") pod \"test-curl-9077-095df6\" (UID: \"a75fed60-ae62-40be-967d-1961b7d41b68\") " pod="opendatahub/test-curl-9077-095df6"
Oct 30 13:10:07.242280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:07.232151    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls"]
Oct 30 13:10:07.242280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:07.233482    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls"
Oct 30 13:10:07.245246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:07.245136    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:07.259256 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:07.250812    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-acd125-serving-cert\""
Oct 30 13:10:07.360066 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:07.357165    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5dmw\" (UniqueName: \"kubernetes.io/projected/a75fed60-ae62-40be-967d-1961b7d41b68-kube-api-access-m5dmw\") pod \"test-curl-9077-095df6\" (UID: \"a75fed60-ae62-40be-967d-1961b7d41b68\") " pod="opendatahub/test-curl-9077-095df6"
Oct 30 13:10:07.379347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:07.368287    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01fb99d1-307e-4917-bb84-b0276b6f78a7-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls\" (UID: \"01fb99d1-307e-4917-bb84-b0276b6f78a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls"
Oct 30 13:10:07.379347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:07.368477    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01fb99d1-307e-4917-bb84-b0276b6f78a7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls\" (UID: \"01fb99d1-307e-4917-bb84-b0276b6f78a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls"
Oct 30 13:10:07.379347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:07.368700    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01fb99d1-307e-4917-bb84-b0276b6f78a7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls\" (UID: \"01fb99d1-307e-4917-bb84-b0276b6f78a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls"
Oct 30 13:10:07.379347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:07.368777    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxh6t\" (UniqueName: \"kubernetes.io/projected/01fb99d1-307e-4917-bb84-b0276b6f78a7-kube-api-access-qxh6t\") pod \"maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls\" (UID: \"01fb99d1-307e-4917-bb84-b0276b6f78a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls"
Oct 30 13:10:07.411359 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:07.411322    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9077-095df6"
Oct 30 13:10:07.509261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:07.506333    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01fb99d1-307e-4917-bb84-b0276b6f78a7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls\" (UID: \"01fb99d1-307e-4917-bb84-b0276b6f78a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls"
Oct 30 13:10:07.509261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:07.506561    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01fb99d1-307e-4917-bb84-b0276b6f78a7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls\" (UID: \"01fb99d1-307e-4917-bb84-b0276b6f78a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls"
Oct 30 13:10:07.509261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:07.506640    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxh6t\" (UniqueName: \"kubernetes.io/projected/01fb99d1-307e-4917-bb84-b0276b6f78a7-kube-api-access-qxh6t\") pod \"maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls\" (UID: \"01fb99d1-307e-4917-bb84-b0276b6f78a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls"
Oct 30 13:10:07.509261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:07.506840    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01fb99d1-307e-4917-bb84-b0276b6f78a7-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls\" (UID: \"01fb99d1-307e-4917-bb84-b0276b6f78a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls"
Oct 30 13:10:07.509261 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:10:07.507730    4544 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-acd125-serving-cert: secret "maas-api-e2e-shared-a-w2-acd125-serving-cert" not found
Oct 30 13:10:07.509261 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:10:07.507816    4544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01fb99d1-307e-4917-bb84-b0276b6f78a7-maas-api-tls podName:01fb99d1-307e-4917-bb84-b0276b6f78a7 nodeName:}" failed. No retries permitted until 2026-10-30 13:10:08.007790223 +0000 UTC m=+1503.054924167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/01fb99d1-307e-4917-bb84-b0276b6f78a7-maas-api-tls") pod "maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls" (UID: "01fb99d1-307e-4917-bb84-b0276b6f78a7") : secret "maas-api-e2e-shared-a-w2-acd125-serving-cert" not found
Oct 30 13:10:07.509261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:07.507990    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01fb99d1-307e-4917-bb84-b0276b6f78a7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls\" (UID: \"01fb99d1-307e-4917-bb84-b0276b6f78a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls"
Oct 30 13:10:07.534445 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:07.527939    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01fb99d1-307e-4917-bb84-b0276b6f78a7-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls\" (UID: \"01fb99d1-307e-4917-bb84-b0276b6f78a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls"
Oct 30 13:10:07.555106 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:07.553513    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxh6t\" (UniqueName: \"kubernetes.io/projected/01fb99d1-307e-4917-bb84-b0276b6f78a7-kube-api-access-qxh6t\") pod \"maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls\" (UID: \"01fb99d1-307e-4917-bb84-b0276b6f78a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls"
Oct 30 13:10:07.690557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:07.689940    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"]
Oct 30 13:10:07.695047 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:07.695008    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:10:07.695153 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:07.695090    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:10:07.695153 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:07.695121    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:10:07.766184 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:07.766146    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:07.911506 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:07.911479    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9077-095df6"]
Oct 30 13:10:07.945730 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:10:07.945693    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda75fed60_ae62_40be_967d_1961b7d41b68.slice/crio-dd097ae02a8fd5b961a48661c9b2ff0c284c299f2e019f6a0fe13a930c953670 WatchSource:0}: Error finding container dd097ae02a8fd5b961a48661c9b2ff0c284c299f2e019f6a0fe13a930c953670: Status 404 returned error can't find the container with id dd097ae02a8fd5b961a48661c9b2ff0c284c299f2e019f6a0fe13a930c953670
Oct 30 13:10:08.069731 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:08.059721    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01fb99d1-307e-4917-bb84-b0276b6f78a7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls\" (UID: \"01fb99d1-307e-4917-bb84-b0276b6f78a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls"
Oct 30 13:10:08.082245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:08.077927    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01fb99d1-307e-4917-bb84-b0276b6f78a7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls\" (UID: \"01fb99d1-307e-4917-bb84-b0276b6f78a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls"
Oct 30 13:10:08.149561 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:08.149525    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9077-095df6" event={"ID":"a75fed60-ae62-40be-967d-1961b7d41b68","Type":"ContainerStarted","Data":"dd097ae02a8fd5b961a48661c9b2ff0c284c299f2e019f6a0fe13a930c953670"}
Oct 30 13:10:08.151776 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:08.151747    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr" event={"ID":"4bbc446d-8807-4613-9d86-b87b0733d2cf","Type":"ContainerStarted","Data":"13fd4d84b6bf12c6d6f835bd82bf43d5eb7acd38a82180c57f66e497c0fea67d"}
Oct 30 13:10:08.151918 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:08.151799    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr" event={"ID":"4bbc446d-8807-4613-9d86-b87b0733d2cf","Type":"ContainerStarted","Data":"f4e098b4abd7e9df6b3be4ff723f2262afe279966bc2d3e2276e8ebf858bd34f"}
Oct 30 13:10:08.166051 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:08.165955    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls"
Oct 30 13:10:08.198776 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:08.198714    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr" podStartSLOduration=2.198692326 podStartE2EDuration="2.198692326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:08.195728679 +0000 UTC m=+1503.242862644" watchObservedRunningTime="2026-10-30 13:10:08.198692326 +0000 UTC m=+1503.245826282"
Oct 30 13:10:08.609409 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:08.609368    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls"]
Oct 30 13:10:08.809246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:08.802850    4544 scope.go:122] "RemoveContainer" containerID="2c0acec0b4ac372ae25a356166fdfb5343fb080113f128d2d604cc8eb8dd96d7"
Oct 30 13:10:08.909049 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:08.909024    4544 scope.go:122] "RemoveContainer" containerID="36486cf473b9ddf32ff2ff86ec58b5f02cae05c455be212317d8d21339f6e9a8"
Oct 30 13:10:08.943962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:08.943933    4544 scope.go:122] "RemoveContainer" containerID="8ce24da7c0298914401e0c5b39b2851eb7035207d9a98e44858b8521d35b481a"
Oct 30 13:10:08.955120 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:08.955093    4544 scope.go:122] "RemoveContainer" containerID="a434da3d88b69d9bd9317048814abe0fc87ecfd54859de15eac194ec74eaab51"
Oct 30 13:10:08.986780 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:08.983554    4544 scope.go:122] "RemoveContainer" containerID="b77d15e8a6c4f73db60579cc43f469153211bb44e822890e413285d47a5030da"
Oct 30 13:10:09.011585 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:09.011561    4544 scope.go:122] "RemoveContainer" containerID="0c96b93ee14f69dbba7c4559fb3e944e0d12943c98d9a9a0b6c218dc5753dfc5"
Oct 30 13:10:09.050797 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:09.050772    4544 scope.go:122] "RemoveContainer" containerID="5fd2b3428780d878eae86ff221999d774b6b24fff9e96af0026a7ac20d9be269"
Oct 30 13:10:09.112155 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:09.112125    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:10:09.138252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:09.134030    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:10:09.165779 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:09.165735    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls" event={"ID":"01fb99d1-307e-4917-bb84-b0276b6f78a7","Type":"ContainerStarted","Data":"9a19f54320c63b4f52d24d1a41fe2ba8f76cfdfc6c04c7f3e050148b929f0f2a"}
Oct 30 13:10:09.167588 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:09.167554    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9077-095df6" event={"ID":"a75fed60-ae62-40be-967d-1961b7d41b68","Type":"ContainerStarted","Data":"281f53f9842d27d98596c57019ece6b32a2b6fe20a311bf86e4ad1b4be410235"}
Oct 30 13:10:09.172299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:09.172276    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:10:09.173924 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:09.173903    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:10:09.286035 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:09.285978    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9077-095df6" podStartSLOduration=3.285954345 podStartE2EDuration="3.285954345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:09.277449033 +0000 UTC m=+1504.324582985" watchObservedRunningTime="2026-10-30 13:10:09.285954345 +0000 UTC m=+1504.333088299"
Oct 30 13:10:09.496921 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:09.496838    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:10.183817 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:10.183776    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls" event={"ID":"01fb99d1-307e-4917-bb84-b0276b6f78a7","Type":"ContainerStarted","Data":"919edfeac1f9d1a694fafe8d9f7fe07cfaefee798b23156e86b479542d58ae74"}
Oct 30 13:10:10.184364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:10.183863    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls"
Oct 30 13:10:10.226391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:10.226333    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls" podStartSLOduration=3.752543083 podStartE2EDuration="4.226316097s" totalImagesPullingTime="473.773014ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:10.225047006 +0000 UTC m=+1505.272180959" watchObservedRunningTime="2026-10-30 13:10:10.226316097 +0000 UTC m=+1505.273450049"
Oct 30 13:10:10.287146 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:10.286959    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9077-095df6"]
Oct 30 13:10:10.292199 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:10.292153    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9077-095df6"]
Oct 30 13:10:10.738561 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:10.738275    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:10.892118 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:10.892072    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:11.192605 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:11.192558    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9077-095df6" podUID="a75fed60-ae62-40be-967d-1961b7d41b68" containerName="test-curl-9077-095df6" containerID="cri-o://281f53f9842d27d98596c57019ece6b32a2b6fe20a311bf86e4ad1b4be410235" gracePeriod=2
Oct 30 13:10:11.827738 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:11.827688    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:12.069746 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:12.069708    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"]
Oct 30 13:10:12.069961 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:12.069932    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g" podUID="75a99162-5892-4899-8487-071967579551" containerName="istio-proxy" containerID="cri-o://4bed83bb47acdb31c9bb102d0c03712b21a56e3c53d8d02e8ada1cd9b229dffe" gracePeriod=30
Oct 30 13:10:12.082516 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:12.082400    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"]
Oct 30 13:10:12.082753 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:12.082677    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq" podUID="c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a" containerName="istio-proxy" containerID="cri-o://280156e3fa2812ed77f449e8367238112d9336d818b926a3b960626144c3198f" gracePeriod=30
Oct 30 13:10:12.088537 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:12.088504    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"]
Oct 30 13:10:12.088794 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:12.088761    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8" podUID="06d5354a-dc17-49c3-af9c-9b756ad02a00" containerName="istio-proxy" containerID="cri-o://9c4b8f133ff65b8b1f93ca72c3b8c55cb0cd97c69941ae3b7dd25da39f6ceaf8" gracePeriod=30
Oct 30 13:10:12.094838 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:12.094806    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"]
Oct 30 13:10:12.095086 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:12.095055    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4" podUID="b72c176a-dc4d-4b4e-82e9-b1e15fceaed3" containerName="istio-proxy" containerID="cri-o://4608c08ab20b056ea6221d80488d26a79c048c7c9841ca6297096f123ffd2d5a" gracePeriod=30
Oct 30 13:10:12.582312 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:12.573715    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:12.829376 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:12.828097    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc"]
Oct 30 13:10:12.830579 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:12.829697    4544 cpu_manager.go:414] "containerMap: removing container" podUID="a75fed60-ae62-40be-967d-1961b7d41b68" containerName="test-curl-9077-095df6"
Oct 30 13:10:12.830579 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:12.829723    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a75fed60-ae62-40be-967d-1961b7d41b68" containerName="test-curl-9077-095df6"
Oct 30 13:10:12.830579 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:12.829938    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="a75fed60-ae62-40be-967d-1961b7d41b68" containerName="test-curl-9077-095df6"
Oct 30 13:10:12.946734 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:12.885605    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:13.180923 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:13.180341    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc"
Oct 30 13:10:13.242956 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:13.228716    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f3160a\""
Oct 30 13:10:13.242956 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:13.233672    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f3160a-dockercfg-nf7k6\""
Oct 30 13:10:13.382944 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:13.382172    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a040f35-d461-49fb-8e95-d10adcb01a0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc\" (UID: \"5a040f35-d461-49fb-8e95-d10adcb01a0a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc"
Oct 30 13:10:13.382944 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:13.382576    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbvgc\" (UniqueName: \"kubernetes.io/projected/5a040f35-d461-49fb-8e95-d10adcb01a0a-kube-api-access-qbvgc\") pod \"payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc\" (UID: \"5a040f35-d461-49fb-8e95-d10adcb01a0a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc"
Oct 30 13:10:13.533302 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:13.466201    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc"]
Oct 30 13:10:13.533302 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:13.466280    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc"]
Oct 30 13:10:13.533302 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:13.487159    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a040f35-d461-49fb-8e95-d10adcb01a0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc\" (UID: \"5a040f35-d461-49fb-8e95-d10adcb01a0a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc"
Oct 30 13:10:13.533302 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:13.487501    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbvgc\" (UniqueName: \"kubernetes.io/projected/5a040f35-d461-49fb-8e95-d10adcb01a0a-kube-api-access-qbvgc\") pod \"payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc\" (UID: \"5a040f35-d461-49fb-8e95-d10adcb01a0a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc"
Oct 30 13:10:13.533302 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:13.489890    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a040f35-d461-49fb-8e95-d10adcb01a0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc\" (UID: \"5a040f35-d461-49fb-8e95-d10adcb01a0a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc"
Oct 30 13:10:13.575483 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:13.575440    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbvgc\" (UniqueName: \"kubernetes.io/projected/5a040f35-d461-49fb-8e95-d10adcb01a0a-kube-api-access-qbvgc\") pod \"payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc\" (UID: \"5a040f35-d461-49fb-8e95-d10adcb01a0a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc"
Oct 30 13:10:13.577270 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:13.576858    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc"]
Oct 30 13:10:13.577270 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:13.576912    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-096c8434-8xk57"]
Oct 30 13:10:13.577270 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:13.576933    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-096c8434-8xk57"]
Oct 30 13:10:13.577270 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:13.577036    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc"
Oct 30 13:10:13.590290 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:13.589993    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz8gw\" (UniqueName: \"kubernetes.io/projected/29a6d4c7-d13a-41ed-a022-57ab28b1ff00-kube-api-access-pz8gw\") pod \"payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc\" (UID: \"29a6d4c7-d13a-41ed-a022-57ab28b1ff00\") " pod="openshift-ingress/payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc"
Oct 30 13:10:13.590290 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:13.590172    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29a6d4c7-d13a-41ed-a022-57ab28b1ff00-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc\" (UID: \"29a6d4c7-d13a-41ed-a022-57ab28b1ff00\") " pod="openshift-ingress/payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc"
Oct 30 13:10:13.692138 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:13.692097    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz8gw\" (UniqueName: \"kubernetes.io/projected/29a6d4c7-d13a-41ed-a022-57ab28b1ff00-kube-api-access-pz8gw\") pod \"payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc\" (UID: \"29a6d4c7-d13a-41ed-a022-57ab28b1ff00\") " pod="openshift-ingress/payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc"
Oct 30 13:10:13.692354 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:13.692263    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29a6d4c7-d13a-41ed-a022-57ab28b1ff00-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc\" (UID: \"29a6d4c7-d13a-41ed-a022-57ab28b1ff00\") " pod="openshift-ingress/payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc"
Oct 30 13:10:13.693267 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:13.693215    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29a6d4c7-d13a-41ed-a022-57ab28b1ff00-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc\" (UID: \"29a6d4c7-d13a-41ed-a022-57ab28b1ff00\") " pod="openshift-ingress/payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc"
Oct 30 13:10:13.693490 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:13.693473    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc"
Oct 30 13:10:13.721748 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:13.720703    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz8gw\" (UniqueName: \"kubernetes.io/projected/29a6d4c7-d13a-41ed-a022-57ab28b1ff00-kube-api-access-pz8gw\") pod \"payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc\" (UID: \"29a6d4c7-d13a-41ed-a022-57ab28b1ff00\") " pod="openshift-ingress/payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc"
Oct 30 13:10:13.832929 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:13.832897    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9077-095df6"
Oct 30 13:10:13.895527 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:13.895486    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a75fed60-ae62-40be-967d-1961b7d41b68-kube-api-access-m5dmw\" (UniqueName: \"kubernetes.io/projected/a75fed60-ae62-40be-967d-1961b7d41b68-kube-api-access-m5dmw\") pod \"a75fed60-ae62-40be-967d-1961b7d41b68\" (UID: \"a75fed60-ae62-40be-967d-1961b7d41b68\") "
Oct 30 13:10:13.897720 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:13.897681    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a75fed60-ae62-40be-967d-1961b7d41b68-kube-api-access-m5dmw" pod "a75fed60-ae62-40be-967d-1961b7d41b68" (UID: "a75fed60-ae62-40be-967d-1961b7d41b68"). InnerVolumeSpecName "kube-api-access-m5dmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:13.934899 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:13.934861    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc"
Oct 30 13:10:13.997726 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:13.997688    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5dmw\" (UniqueName: \"kubernetes.io/projected/a75fed60-ae62-40be-967d-1961b7d41b68-kube-api-access-m5dmw\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:14.024176 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:14.023364    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc"]
Oct 30 13:10:14.026816 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:10:14.026651    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a040f35_d461_49fb_8e95_d10adcb01a0a.slice/crio-c9a4ce30bbf07f2e354c85234a9ce4cd885f9501970ed6c6972ac1979f1a1159 WatchSource:0}: Error finding container c9a4ce30bbf07f2e354c85234a9ce4cd885f9501970ed6c6972ac1979f1a1159: Status 404 returned error can't find the container with id c9a4ce30bbf07f2e354c85234a9ce4cd885f9501970ed6c6972ac1979f1a1159
Oct 30 13:10:14.164406 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:14.164369    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:14.294248 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:10:14.292675    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a6d4c7_d13a_41ed_a022_57ab28b1ff00.slice/crio-b5d7da5eda68523c1f837601bad6c5a59daa6703eeb5f538435ac3185ccc9782 WatchSource:0}: Error finding container b5d7da5eda68523c1f837601bad6c5a59daa6703eeb5f538435ac3185ccc9782: Status 404 returned error can't find the container with id b5d7da5eda68523c1f837601bad6c5a59daa6703eeb5f538435ac3185ccc9782
Oct 30 13:10:14.305246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:14.305159    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc"]
Oct 30 13:10:14.447151 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:14.445662    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc" event={"ID":"5a040f35-d461-49fb-8e95-d10adcb01a0a","Type":"ContainerStarted","Data":"06bb03f970e991635db89c0aed9aad7f5d40c5628e5e38ca779fa46551298f8d"}
Oct 30 13:10:14.447151 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:14.445739    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc" event={"ID":"5a040f35-d461-49fb-8e95-d10adcb01a0a","Type":"ContainerStarted","Data":"c9a4ce30bbf07f2e354c85234a9ce4cd885f9501970ed6c6972ac1979f1a1159"}
Oct 30 13:10:14.447151 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:14.447082    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc"
Oct 30 13:10:14.460776 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:14.459077    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc" event={"ID":"29a6d4c7-d13a-41ed-a022-57ab28b1ff00","Type":"ContainerStarted","Data":"b5d7da5eda68523c1f837601bad6c5a59daa6703eeb5f538435ac3185ccc9782"}
Oct 30 13:10:14.464580 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:14.464548    4544 scope.go:122] "RemoveContainer" containerID="281f53f9842d27d98596c57019ece6b32a2b6fe20a311bf86e4ad1b4be410235"
Oct 30 13:10:14.469502 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:14.464312    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9077-095df6"
Oct 30 13:10:14.470617 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:14.464233    4544 generic.go:356] "Generic (PLEG): container finished" podID="a75fed60-ae62-40be-967d-1961b7d41b68" containerID="281f53f9842d27d98596c57019ece6b32a2b6fe20a311bf86e4ad1b4be410235" exitCode=137
Oct 30 13:10:14.533554 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:10:14.533326    4544 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9077-095df6\" is forbidden: User \"system:node:ip-10-0-143-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-38.ec2.internal' and this object" podUID="a75fed60-ae62-40be-967d-1961b7d41b68" pod="opendatahub/test-curl-9077-095df6"
Oct 30 13:10:14.574043 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:14.571422    4544 scope.go:122] "RemoveContainer" containerID="281f53f9842d27d98596c57019ece6b32a2b6fe20a311bf86e4ad1b4be410235"
Oct 30 13:10:14.587622 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:10:14.587288    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"281f53f9842d27d98596c57019ece6b32a2b6fe20a311bf86e4ad1b4be410235\": container with ID starting with 281f53f9842d27d98596c57019ece6b32a2b6fe20a311bf86e4ad1b4be410235 not found: ID does not exist" containerID="281f53f9842d27d98596c57019ece6b32a2b6fe20a311bf86e4ad1b4be410235"
Oct 30 13:10:14.587622 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:14.587340    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"281f53f9842d27d98596c57019ece6b32a2b6fe20a311bf86e4ad1b4be410235"} err="failed to get container status \"281f53f9842d27d98596c57019ece6b32a2b6fe20a311bf86e4ad1b4be410235\": rpc error: code = NotFound desc = could not find container \"281f53f9842d27d98596c57019ece6b32a2b6fe20a311bf86e4ad1b4be410235\": container with ID starting with 281f53f9842d27d98596c57019ece6b32a2b6fe20a311bf86e4ad1b4be410235 not found: ID does not exist"
Oct 30 13:10:15.093306 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:15.093271    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47a963c9-3243-4afd-a05d-e35e024ee5b6" path="/var/lib/kubelet/pods/47a963c9-3243-4afd-a05d-e35e024ee5b6/volumes"
Oct 30 13:10:15.093996 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:15.093974    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a75fed60-ae62-40be-967d-1961b7d41b68" path="/var/lib/kubelet/pods/a75fed60-ae62-40be-967d-1961b7d41b68/volumes"
Oct 30 13:10:15.098180 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:10:15.098145    4544 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9077-095df6\" is forbidden: User \"system:node:ip-10-0-143-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-38.ec2.internal' and this object" podUID="a75fed60-ae62-40be-967d-1961b7d41b68" pod="opendatahub/test-curl-9077-095df6"
Oct 30 13:10:15.481786 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:15.481694    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc" event={"ID":"29a6d4c7-d13a-41ed-a022-57ab28b1ff00","Type":"ContainerStarted","Data":"f73fce2f09d11365a4a8eff5399781854cb2e359b90c9048a4f3172d296cccf4"}
Oct 30 13:10:15.482044 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:15.482015    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc"
Oct 30 13:10:15.525891 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:15.525828    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc" podStartSLOduration=3.525805777 podStartE2EDuration="3.525805777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:15.522298842 +0000 UTC m=+1510.569432796" watchObservedRunningTime="2026-10-30 13:10:15.525805777 +0000 UTC m=+1510.572939730"
Oct 30 13:10:15.531081 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:15.531006    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc" podStartSLOduration=3.5309708459999998 podStartE2EDuration="3.530970846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:14.516875695 +0000 UTC m=+1509.564009651" watchObservedRunningTime="2026-10-30 13:10:15.530970846 +0000 UTC m=+1510.578104800"
Oct 30 13:10:16.199325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:16.199293    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls"
Oct 30 13:10:16.392505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:16.392473    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:16.579001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:16.576354    4544 generic.go:356] "Generic (PLEG): container finished" podID="565c6710-ac33-437a-b2cc-5572a325523d" containerID="4c146428fb3ecc1e27887d68f26980067a3c9352a0ecf97b6871ee162561feaa" exitCode=0
Oct 30 13:10:16.593247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:16.589422    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h" event={"ID":"565c6710-ac33-437a-b2cc-5572a325523d","Type":"ContainerDied","Data":"4c146428fb3ecc1e27887d68f26980067a3c9352a0ecf97b6871ee162561feaa"}
Oct 30 13:10:16.594509 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:16.594473    4544 scope.go:122] "RemoveContainer" containerID="4c146428fb3ecc1e27887d68f26980067a3c9352a0ecf97b6871ee162561feaa"
Oct 30 13:10:16.596997 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:16.596953    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:10:16.597134 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:16.597059    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:10:16.597134 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:16.597113    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:10:17.603611 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.603281    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:10:17.641306 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.633880    4544 generic.go:356] "Generic (PLEG): container finished" podID="75a99162-5892-4899-8487-071967579551" containerID="4bed83bb47acdb31c9bb102d0c03712b21a56e3c53d8d02e8ada1cd9b229dffe" exitCode=0
Oct 30 13:10:17.641306 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.634023    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g" event={"ID":"75a99162-5892-4899-8487-071967579551","Type":"ContainerDied","Data":"4bed83bb47acdb31c9bb102d0c03712b21a56e3c53d8d02e8ada1cd9b229dffe"}
Oct 30 13:10:17.705351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.704337    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istio-podinfo\") pod \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") "
Oct 30 13:10:17.705351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.704418    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-credential-socket\") pod \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") "
Oct 30 13:10:17.705351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.704507    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-kube-api-access-sskqq\" (UniqueName: \"kubernetes.io/projected/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-kube-api-access-sskqq\") pod \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") "
Oct 30 13:10:17.705351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.704638    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-workload-socket\") pod \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") "
Oct 30 13:10:17.705351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.704686    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istiod-ca-cert\") pod \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") "
Oct 30 13:10:17.705351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.704756    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istio-envoy\") pod \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") "
Oct 30 13:10:17.705351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.704800    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istio-token\" (UniqueName: \"kubernetes.io/projected/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istio-token\") pod \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") "
Oct 30 13:10:17.705351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.704933    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-workload-certs\") pod \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") "
Oct 30 13:10:17.705351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.704983    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istio-data\") pod \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\" (UID: \"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a\") "
Oct 30 13:10:17.706066 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.706010    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istio-data" pod "c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a" (UID: "c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:17.706406 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.706384    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-workload-socket" pod "c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a" (UID: "c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:17.707005 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.706976    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istiod-ca-cert" pod "c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a" (UID: "c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:17.707360 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.707335    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-workload-certs" pod "c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a" (UID: "c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:17.707520 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.707478    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-credential-socket" pod "c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a" (UID: "c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:17.709550 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.709521    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istio-podinfo" pod "c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a" (UID: "c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:10:17.709976 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.709936    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istio-envoy" pod "c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a" (UID: "c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:17.710590 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.710554    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-kube-api-access-sskqq" pod "c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a" (UID: "c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a"). InnerVolumeSpecName "kube-api-access-sskqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:17.773878 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.736208    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h" event={"ID":"565c6710-ac33-437a-b2cc-5572a325523d","Type":"ContainerStarted","Data":"3cce96af41f9471fb3b52d776887166145d78a3b078c98cc1ca160c0c9ce9498"}
Oct 30 13:10:17.790499 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.790418    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istio-token" pod "c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a" (UID: "c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:17.793257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.790901    4544 generic.go:356] "Generic (PLEG): container finished" podID="b72c176a-dc4d-4b4e-82e9-b1e15fceaed3" containerID="4608c08ab20b056ea6221d80488d26a79c048c7c9841ca6297096f123ffd2d5a" exitCode=0
Oct 30 13:10:17.793257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.791008    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4" event={"ID":"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3","Type":"ContainerDied","Data":"4608c08ab20b056ea6221d80488d26a79c048c7c9841ca6297096f123ffd2d5a"}
Oct 30 13:10:17.793257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.791080    4544 scope.go:122] "RemoveContainer" containerID="4bbfd2bb3e5cde47f000197b8d0e4614382d950ec815493d55462c8b59f9206d"
Oct 30 13:10:17.793905 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.793812    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"
Oct 30 13:10:17.793905 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.793879    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq" event={"ID":"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a","Type":"ContainerDied","Data":"280156e3fa2812ed77f449e8367238112d9336d818b926a3b960626144c3198f"}
Oct 30 13:10:17.796429 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.796281    4544 generic.go:356] "Generic (PLEG): container finished" podID="c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a" containerID="280156e3fa2812ed77f449e8367238112d9336d818b926a3b960626144c3198f" exitCode=0
Oct 30 13:10:17.796429 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.796396    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq" event={"ID":"c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a","Type":"ContainerDied","Data":"b3fbb5e091e3cc537460ad9a52a8efbbcd0c6952d4b829a71ecd715609a4dbf2"}
Oct 30 13:10:17.802991 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.802957    4544 generic.go:356] "Generic (PLEG): container finished" podID="06d5354a-dc17-49c3-af9c-9b756ad02a00" containerID="9c4b8f133ff65b8b1f93ca72c3b8c55cb0cd97c69941ae3b7dd25da39f6ceaf8" exitCode=0
Oct 30 13:10:17.803195 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.803058    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8" event={"ID":"06d5354a-dc17-49c3-af9c-9b756ad02a00","Type":"ContainerDied","Data":"9c4b8f133ff65b8b1f93ca72c3b8c55cb0cd97c69941ae3b7dd25da39f6ceaf8"}
Oct 30 13:10:17.895439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.891802    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:10:17.895439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.893432    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:17.895439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.893460    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:17.895439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.893475    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sskqq\" (UniqueName: \"kubernetes.io/projected/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-kube-api-access-sskqq\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:17.895439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.893491    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:17.895439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.893506    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:17.895439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.893519    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:17.895439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.893533    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:17.895439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.893547    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:17.895439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.893559    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:17.984047 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.982787    4544 scope.go:122] "RemoveContainer" containerID="280156e3fa2812ed77f449e8367238112d9336d818b926a3b960626144c3198f"
Oct 30 13:10:17.994494 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.994461    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06d5354a-dc17-49c3-af9c-9b756ad02a00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06d5354a-dc17-49c3-af9c-9b756ad02a00-istio-podinfo\") pod \"06d5354a-dc17-49c3-af9c-9b756ad02a00\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") "
Oct 30 13:10:17.994640 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.994558    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06d5354a-dc17-49c3-af9c-9b756ad02a00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06d5354a-dc17-49c3-af9c-9b756ad02a00-istiod-ca-cert\") pod \"06d5354a-dc17-49c3-af9c-9b756ad02a00\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") "
Oct 30 13:10:17.994705 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.994675    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-workload-socket\") pod \"06d5354a-dc17-49c3-af9c-9b756ad02a00\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") "
Oct 30 13:10:17.994763 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.994731    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-credential-socket\") pod \"06d5354a-dc17-49c3-af9c-9b756ad02a00\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") "
Oct 30 13:10:17.994823 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.994769    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06d5354a-dc17-49c3-af9c-9b756ad02a00-istio-token\" (UniqueName: \"kubernetes.io/projected/06d5354a-dc17-49c3-af9c-9b756ad02a00-istio-token\") pod \"06d5354a-dc17-49c3-af9c-9b756ad02a00\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") "
Oct 30 13:10:17.994934 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.994901    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-istio-data\") pod \"06d5354a-dc17-49c3-af9c-9b756ad02a00\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") "
Oct 30 13:10:17.995693 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.994973    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06d5354a-dc17-49c3-af9c-9b756ad02a00-kube-api-access-l72tg\" (UniqueName: \"kubernetes.io/projected/06d5354a-dc17-49c3-af9c-9b756ad02a00-kube-api-access-l72tg\") pod \"06d5354a-dc17-49c3-af9c-9b756ad02a00\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") "
Oct 30 13:10:17.995693 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.995085    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-workload-certs\") pod \"06d5354a-dc17-49c3-af9c-9b756ad02a00\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") "
Oct 30 13:10:17.995693 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.995164    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-istio-envoy\") pod \"06d5354a-dc17-49c3-af9c-9b756ad02a00\" (UID: \"06d5354a-dc17-49c3-af9c-9b756ad02a00\") "
Oct 30 13:10:17.996612 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.996585    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-istio-data" pod "06d5354a-dc17-49c3-af9c-9b756ad02a00" (UID: "06d5354a-dc17-49c3-af9c-9b756ad02a00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:17.998040 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.997991    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06d5354a-dc17-49c3-af9c-9b756ad02a00-istiod-ca-cert" pod "06d5354a-dc17-49c3-af9c-9b756ad02a00" (UID: "06d5354a-dc17-49c3-af9c-9b756ad02a00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:17.998419 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.998252    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-credential-socket" pod "06d5354a-dc17-49c3-af9c-9b756ad02a00" (UID: "06d5354a-dc17-49c3-af9c-9b756ad02a00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:17.998419 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.998384    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-workload-socket" pod "06d5354a-dc17-49c3-af9c-9b756ad02a00" (UID: "06d5354a-dc17-49c3-af9c-9b756ad02a00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:17.998633 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:17.998611    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-workload-certs" pod "06d5354a-dc17-49c3-af9c-9b756ad02a00" (UID: "06d5354a-dc17-49c3-af9c-9b756ad02a00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:18.002407 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.002371    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-istio-envoy" pod "06d5354a-dc17-49c3-af9c-9b756ad02a00" (UID: "06d5354a-dc17-49c3-af9c-9b756ad02a00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:18.002513 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.002405    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06d5354a-dc17-49c3-af9c-9b756ad02a00-istio-podinfo" pod "06d5354a-dc17-49c3-af9c-9b756ad02a00" (UID: "06d5354a-dc17-49c3-af9c-9b756ad02a00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:10:18.002725 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.002703    4544 scope.go:122] "RemoveContainer" containerID="280156e3fa2812ed77f449e8367238112d9336d818b926a3b960626144c3198f"
Oct 30 13:10:18.004150 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.002922    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d5354a-dc17-49c3-af9c-9b756ad02a00-kube-api-access-l72tg" pod "06d5354a-dc17-49c3-af9c-9b756ad02a00" (UID: "06d5354a-dc17-49c3-af9c-9b756ad02a00"). InnerVolumeSpecName "kube-api-access-l72tg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:18.004471 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:10:18.004439    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"280156e3fa2812ed77f449e8367238112d9336d818b926a3b960626144c3198f\": container with ID starting with 280156e3fa2812ed77f449e8367238112d9336d818b926a3b960626144c3198f not found: ID does not exist" containerID="280156e3fa2812ed77f449e8367238112d9336d818b926a3b960626144c3198f"
Oct 30 13:10:18.004573 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.004485    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"280156e3fa2812ed77f449e8367238112d9336d818b926a3b960626144c3198f"} err="failed to get container status \"280156e3fa2812ed77f449e8367238112d9336d818b926a3b960626144c3198f\": rpc error: code = NotFound desc = could not find container \"280156e3fa2812ed77f449e8367238112d9336d818b926a3b960626144c3198f\": container with ID starting with 280156e3fa2812ed77f449e8367238112d9336d818b926a3b960626144c3198f not found: ID does not exist"
Oct 30 13:10:18.005467 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.005442    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d5354a-dc17-49c3-af9c-9b756ad02a00-istio-token" pod "06d5354a-dc17-49c3-af9c-9b756ad02a00" (UID: "06d5354a-dc17-49c3-af9c-9b756ad02a00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:18.043246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.037465    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 13:10:18.043246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.039099    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"]
Oct 30 13:10:18.043246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.040168    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:10:18.047116 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.047063    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq"]
Oct 30 13:10:18.106250 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.102301    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06d5354a-dc17-49c3-af9c-9b756ad02a00-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:18.106250 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.102349    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06d5354a-dc17-49c3-af9c-9b756ad02a00-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:18.106250 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.102363    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:18.106250 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.102377    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:18.106250 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.102397    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06d5354a-dc17-49c3-af9c-9b756ad02a00-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:18.106250 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.102410    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:18.106250 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.102422    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l72tg\" (UniqueName: \"kubernetes.io/projected/06d5354a-dc17-49c3-af9c-9b756ad02a00-kube-api-access-l72tg\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:18.106250 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.102437    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:18.106250 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.102450    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06d5354a-dc17-49c3-af9c-9b756ad02a00-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:18.207298 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.206981    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-workload-certs\") pod \"75a99162-5892-4899-8487-071967579551\" (UID: \"75a99162-5892-4899-8487-071967579551\") "
Oct 30 13:10:18.207298 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.207066    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istiod-ca-cert\") pod \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") "
Oct 30 13:10:18.207298 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.207139    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75a99162-5892-4899-8487-071967579551-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a99162-5892-4899-8487-071967579551-istiod-ca-cert\") pod \"75a99162-5892-4899-8487-071967579551\" (UID: \"75a99162-5892-4899-8487-071967579551\") "
Oct 30 13:10:18.207298 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.207181    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-workload-socket\") pod \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") "
Oct 30 13:10:18.207298 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.207251    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-credential-socket\") pod \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") "
Oct 30 13:10:18.207298 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.207304    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-istio-data\") pod \"75a99162-5892-4899-8487-071967579551\" (UID: \"75a99162-5892-4899-8487-071967579551\") "
Oct 30 13:10:18.207826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.207373    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-istio-envoy\") pod \"75a99162-5892-4899-8487-071967579551\" (UID: \"75a99162-5892-4899-8487-071967579551\") "
Oct 30 13:10:18.207826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.207438    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istio-podinfo\") pod \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") "
Oct 30 13:10:18.207826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.207479    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75a99162-5892-4899-8487-071967579551-istio-token\" (UniqueName: \"kubernetes.io/projected/75a99162-5892-4899-8487-071967579551-istio-token\") pod \"75a99162-5892-4899-8487-071967579551\" (UID: \"75a99162-5892-4899-8487-071967579551\") "
Oct 30 13:10:18.207826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.207509    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istio-data\") pod \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") "
Oct 30 13:10:18.207826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.207545    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75a99162-5892-4899-8487-071967579551-kube-api-access-cztw2\" (UniqueName: \"kubernetes.io/projected/75a99162-5892-4899-8487-071967579551-kube-api-access-cztw2\") pod \"75a99162-5892-4899-8487-071967579551\" (UID: \"75a99162-5892-4899-8487-071967579551\") "
Oct 30 13:10:18.207826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.207701    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-workload-certs\") pod \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") "
Oct 30 13:10:18.207826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.207745    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-kube-api-access-44tv2\" (UniqueName: \"kubernetes.io/projected/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-kube-api-access-44tv2\") pod \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") "
Oct 30 13:10:18.207826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.207786    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-workload-socket\") pod \"75a99162-5892-4899-8487-071967579551\" (UID: \"75a99162-5892-4899-8487-071967579551\") "
Oct 30 13:10:18.207826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.207828    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istio-token\" (UniqueName: \"kubernetes.io/projected/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istio-token\") pod \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") "
Oct 30 13:10:18.208908 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.207881    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istio-envoy\") pod \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\" (UID: \"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3\") "
Oct 30 13:10:18.208908 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.207968    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75a99162-5892-4899-8487-071967579551-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a99162-5892-4899-8487-071967579551-istio-podinfo\") pod \"75a99162-5892-4899-8487-071967579551\" (UID: \"75a99162-5892-4899-8487-071967579551\") "
Oct 30 13:10:18.208908 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.208016    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-credential-socket\") pod \"75a99162-5892-4899-8487-071967579551\" (UID: \"75a99162-5892-4899-8487-071967579551\") "
Oct 30 13:10:18.221577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.209527    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istio-data" pod "b72c176a-dc4d-4b4e-82e9-b1e15fceaed3" (UID: "b72c176a-dc4d-4b4e-82e9-b1e15fceaed3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:18.221577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.211732    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istiod-ca-cert" pod "b72c176a-dc4d-4b4e-82e9-b1e15fceaed3" (UID: "b72c176a-dc4d-4b4e-82e9-b1e15fceaed3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:18.221577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.211971    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-workload-socket" pod "b72c176a-dc4d-4b4e-82e9-b1e15fceaed3" (UID: "b72c176a-dc4d-4b4e-82e9-b1e15fceaed3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:18.221577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.217409    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75a99162-5892-4899-8487-071967579551-istiod-ca-cert" pod "75a99162-5892-4899-8487-071967579551" (UID: "75a99162-5892-4899-8487-071967579551"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:18.221577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.217681    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-workload-socket" pod "75a99162-5892-4899-8487-071967579551" (UID: "75a99162-5892-4899-8487-071967579551"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:18.221577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.219304    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-credential-socket" pod "75a99162-5892-4899-8487-071967579551" (UID: "75a99162-5892-4899-8487-071967579551"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:18.250015 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.222105    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-workload-certs" pod "75a99162-5892-4899-8487-071967579551" (UID: "75a99162-5892-4899-8487-071967579551"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:18.250015 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.223297    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-workload-certs" pod "b72c176a-dc4d-4b4e-82e9-b1e15fceaed3" (UID: "b72c176a-dc4d-4b4e-82e9-b1e15fceaed3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:18.250015 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.240046    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a99162-5892-4899-8487-071967579551-kube-api-access-cztw2" pod "75a99162-5892-4899-8487-071967579551" (UID: "75a99162-5892-4899-8487-071967579551"). InnerVolumeSpecName "kube-api-access-cztw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:18.280246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.276379    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-istio-data" pod "75a99162-5892-4899-8487-071967579551" (UID: "75a99162-5892-4899-8487-071967579551"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:18.280246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.277235    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-credential-socket" pod "b72c176a-dc4d-4b4e-82e9-b1e15fceaed3" (UID: "b72c176a-dc4d-4b4e-82e9-b1e15fceaed3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:18.285184 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.284923    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istio-podinfo" pod "b72c176a-dc4d-4b4e-82e9-b1e15fceaed3" (UID: "b72c176a-dc4d-4b4e-82e9-b1e15fceaed3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:10:18.285392 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.285357    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istio-envoy" pod "b72c176a-dc4d-4b4e-82e9-b1e15fceaed3" (UID: "b72c176a-dc4d-4b4e-82e9-b1e15fceaed3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:18.285824 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.285793    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-istio-envoy" pod "75a99162-5892-4899-8487-071967579551" (UID: "75a99162-5892-4899-8487-071967579551"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:18.285916 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.285854    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a99162-5892-4899-8487-071967579551-istio-token" pod "75a99162-5892-4899-8487-071967579551" (UID: "75a99162-5892-4899-8487-071967579551"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:18.286655 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.286627    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istio-token" pod "b72c176a-dc4d-4b4e-82e9-b1e15fceaed3" (UID: "b72c176a-dc4d-4b4e-82e9-b1e15fceaed3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:18.286993 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.286974    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-kube-api-access-44tv2" pod "b72c176a-dc4d-4b4e-82e9-b1e15fceaed3" (UID: "b72c176a-dc4d-4b4e-82e9-b1e15fceaed3"). InnerVolumeSpecName "kube-api-access-44tv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:18.287279 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.287248    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75a99162-5892-4899-8487-071967579551-istio-podinfo" pod "75a99162-5892-4899-8487-071967579551" (UID: "75a99162-5892-4899-8487-071967579551"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:10:18.311562 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.311369    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a99162-5892-4899-8487-071967579551-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:18.311562 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.311395    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:18.311562 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.311404    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:18.311562 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.311413    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:18.311562 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.311422    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a99162-5892-4899-8487-071967579551-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:18.311562 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.311430    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:18.311562 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.311439    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:18.311562 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.311448    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:18.311562 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.311458    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:18.311562 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.311465    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:18.311562 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.311473    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75a99162-5892-4899-8487-071967579551-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:18.311562 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.311480    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:18.311562 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.311488    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cztw2\" (UniqueName: \"kubernetes.io/projected/75a99162-5892-4899-8487-071967579551-kube-api-access-cztw2\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:18.311562 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.311495    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:18.311562 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.311503    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44tv2\" (UniqueName: \"kubernetes.io/projected/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-kube-api-access-44tv2\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:18.311562 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.311511    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a99162-5892-4899-8487-071967579551-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:18.311562 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.311521    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:18.311562 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.311533    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:18.552303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.551999    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g"]
Oct 30 13:10:18.563876 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.563841    4544 cpu_manager.go:414] "containerMap: removing container" podUID="75a99162-5892-4899-8487-071967579551" containerName="istio-proxy"
Oct 30 13:10:18.564138 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.564117    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a99162-5892-4899-8487-071967579551" containerName="istio-proxy"
Oct 30 13:10:18.565440 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.565376    4544 cpu_manager.go:414] "containerMap: removing container" podUID="c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a" containerName="istio-proxy"
Oct 30 13:10:18.565440 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.565400    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a" containerName="istio-proxy"
Oct 30 13:10:18.565440 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.565442    4544 cpu_manager.go:414] "containerMap: removing container" podUID="b72c176a-dc4d-4b4e-82e9-b1e15fceaed3" containerName="istio-proxy"
Oct 30 13:10:18.565770 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.565450    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b72c176a-dc4d-4b4e-82e9-b1e15fceaed3" containerName="istio-proxy"
Oct 30 13:10:18.565770 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.565465    4544 cpu_manager.go:414] "containerMap: removing container" podUID="b72c176a-dc4d-4b4e-82e9-b1e15fceaed3" containerName="istio-proxy"
Oct 30 13:10:18.565770 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.565471    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b72c176a-dc4d-4b4e-82e9-b1e15fceaed3" containerName="istio-proxy"
Oct 30 13:10:18.565770 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.565493    4544 cpu_manager.go:414] "containerMap: removing container" podUID="06d5354a-dc17-49c3-af9c-9b756ad02a00" containerName="istio-proxy"
Oct 30 13:10:18.565770 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.565500    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06d5354a-dc17-49c3-af9c-9b756ad02a00" containerName="istio-proxy"
Oct 30 13:10:18.565770 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.565769    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a" containerName="istio-proxy"
Oct 30 13:10:18.566164 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.565793    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="b72c176a-dc4d-4b4e-82e9-b1e15fceaed3" containerName="istio-proxy"
Oct 30 13:10:18.566164 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.565806    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="b72c176a-dc4d-4b4e-82e9-b1e15fceaed3" containerName="istio-proxy"
Oct 30 13:10:18.566164 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.565818    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="06d5354a-dc17-49c3-af9c-9b756ad02a00" containerName="istio-proxy"
Oct 30 13:10:18.566164 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.565832    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a99162-5892-4899-8487-071967579551" containerName="istio-proxy"
Oct 30 13:10:18.571468 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.571448    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g"
Oct 30 13:10:18.578473 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.578446    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g"]
Oct 30 13:10:18.585822 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.585796    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-3e8ee4-serving-cert\""
Oct 30 13:10:18.669297 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.662925    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:18.671130 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.671050    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a19094f-f1fc-4f65-9702-96dc3a0a2f77-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g\" (UID: \"3a19094f-f1fc-4f65-9702-96dc3a0a2f77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g"
Oct 30 13:10:18.671130 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.671090    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z42xr\" (UniqueName: \"kubernetes.io/projected/3a19094f-f1fc-4f65-9702-96dc3a0a2f77-kube-api-access-z42xr\") pod \"maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g\" (UID: \"3a19094f-f1fc-4f65-9702-96dc3a0a2f77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g"
Oct 30 13:10:18.671375 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.671345    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a19094f-f1fc-4f65-9702-96dc3a0a2f77-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g\" (UID: \"3a19094f-f1fc-4f65-9702-96dc3a0a2f77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g"
Oct 30 13:10:18.671545 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.671514    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a19094f-f1fc-4f65-9702-96dc3a0a2f77-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g\" (UID: \"3a19094f-f1fc-4f65-9702-96dc3a0a2f77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g"
Oct 30 13:10:18.706855 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.706829    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:10:18.707765 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.707747    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 13:10:18.707931 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.707912    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h" podUID="565c6710-ac33-437a-b2cc-5572a325523d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 13:10:18.796258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.792813    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:18.796258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.793182    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a19094f-f1fc-4f65-9702-96dc3a0a2f77-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g\" (UID: \"3a19094f-f1fc-4f65-9702-96dc3a0a2f77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g"
Oct 30 13:10:18.796258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.793246    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z42xr\" (UniqueName: \"kubernetes.io/projected/3a19094f-f1fc-4f65-9702-96dc3a0a2f77-kube-api-access-z42xr\") pod \"maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g\" (UID: \"3a19094f-f1fc-4f65-9702-96dc3a0a2f77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g"
Oct 30 13:10:18.796258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.793341    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a19094f-f1fc-4f65-9702-96dc3a0a2f77-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g\" (UID: \"3a19094f-f1fc-4f65-9702-96dc3a0a2f77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g"
Oct 30 13:10:18.796258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.793553    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a19094f-f1fc-4f65-9702-96dc3a0a2f77-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g\" (UID: \"3a19094f-f1fc-4f65-9702-96dc3a0a2f77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g"
Oct 30 13:10:18.796258 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:10:18.793874    4544 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3e8ee4-serving-cert: secret "maas-api-e2e-shared-b-w6-3e8ee4-serving-cert" not found
Oct 30 13:10:18.796258 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:10:18.793940    4544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a19094f-f1fc-4f65-9702-96dc3a0a2f77-maas-api-tls podName:3a19094f-f1fc-4f65-9702-96dc3a0a2f77 nodeName:}" failed. No retries permitted until 2026-10-30 13:10:19.293920866 +0000 UTC m=+1514.341054814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3a19094f-f1fc-4f65-9702-96dc3a0a2f77-maas-api-tls") pod "maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g" (UID: "3a19094f-f1fc-4f65-9702-96dc3a0a2f77") : secret "maas-api-e2e-shared-b-w6-3e8ee4-serving-cert" not found
Oct 30 13:10:18.818307 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.811281    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a19094f-f1fc-4f65-9702-96dc3a0a2f77-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g\" (UID: \"3a19094f-f1fc-4f65-9702-96dc3a0a2f77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g"
Oct 30 13:10:18.885257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.834866    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a19094f-f1fc-4f65-9702-96dc3a0a2f77-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g\" (UID: \"3a19094f-f1fc-4f65-9702-96dc3a0a2f77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g"
Oct 30 13:10:18.919491 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.919055    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z42xr\" (UniqueName: \"kubernetes.io/projected/3a19094f-f1fc-4f65-9702-96dc3a0a2f77-kube-api-access-z42xr\") pod \"maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g\" (UID: \"3a19094f-f1fc-4f65-9702-96dc3a0a2f77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g"
Oct 30 13:10:18.933403 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.933339    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4" event={"ID":"b72c176a-dc4d-4b4e-82e9-b1e15fceaed3","Type":"ContainerDied","Data":"2966b83548118af0b63eea1c965c32210721b021a4f1ee51ad8dce04d4ca0c75"}
Oct 30 13:10:18.933790 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.933723    4544 scope.go:122] "RemoveContainer" containerID="4608c08ab20b056ea6221d80488d26a79c048c7c9841ca6297096f123ffd2d5a"
Oct 30 13:10:18.934180 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.934084    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"
Oct 30 13:10:18.960134 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.960045    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:18.993115 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.973837    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8" event={"ID":"06d5354a-dc17-49c3-af9c-9b756ad02a00","Type":"ContainerDied","Data":"aba5ad3dd5de99a627092c9ab63304d0daef614784b2d124c475b26638c70432"}
Oct 30 13:10:18.993115 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.974127    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"
Oct 30 13:10:19.000145 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:18.999891    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"
Oct 30 13:10:19.000145 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:19.000061    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g" event={"ID":"75a99162-5892-4899-8487-071967579551","Type":"ContainerDied","Data":"f1f59d9690993a1a31a88e07c9584c3eec2a4658e24a40230c9e561c7b2993e5"}
Oct 30 13:10:19.087519 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:19.082420    4544 scope.go:122] "RemoveContainer" containerID="9c4b8f133ff65b8b1f93ca72c3b8c55cb0cd97c69941ae3b7dd25da39f6ceaf8"
Oct 30 13:10:19.101461 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:19.101430    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a" path="/var/lib/kubelet/pods/c07f44cd-a58d-44ce-ad1c-0ce81ca98f3a/volumes"
Oct 30 13:10:19.103646 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:19.103632    4544 scope.go:122] "RemoveContainer" containerID="4bed83bb47acdb31c9bb102d0c03712b21a56e3c53d8d02e8ada1cd9b229dffe"
Oct 30 13:10:19.165650 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:19.165608    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"]
Oct 30 13:10:19.173245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:19.172317    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8"]
Oct 30 13:10:19.206530 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:19.206500    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"]
Oct 30 13:10:19.211032 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:19.211005    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4"]
Oct 30 13:10:19.268239 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:19.268163    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"]
Oct 30 13:10:19.272266 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:19.272242    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g"]
Oct 30 13:10:19.365975 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:19.365873    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a19094f-f1fc-4f65-9702-96dc3a0a2f77-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g\" (UID: \"3a19094f-f1fc-4f65-9702-96dc3a0a2f77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g"
Oct 30 13:10:19.368779 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:19.368751    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a19094f-f1fc-4f65-9702-96dc3a0a2f77-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g\" (UID: \"3a19094f-f1fc-4f65-9702-96dc3a0a2f77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g"
Oct 30 13:10:19.495999 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:19.495965    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g"
Oct 30 13:10:19.773150 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:19.766785    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 13:10:19.773150 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:19.766903    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h" podUID="565c6710-ac33-437a-b2cc-5572a325523d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 13:10:20.085253 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:20.075642    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g"]
Oct 30 13:10:20.505306 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:20.502805    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:20.698375 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:20.698346    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:20.707168 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:20.707144    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 13:10:20.707303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:20.707194    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h" podUID="565c6710-ac33-437a-b2cc-5572a325523d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 13:10:20.840245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:20.839373    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:20.988248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:20.986182    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:21.105134 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:21.102988    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06d5354a-dc17-49c3-af9c-9b756ad02a00" path="/var/lib/kubelet/pods/06d5354a-dc17-49c3-af9c-9b756ad02a00/volumes"
Oct 30 13:10:21.105134 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:21.103798    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75a99162-5892-4899-8487-071967579551" path="/var/lib/kubelet/pods/75a99162-5892-4899-8487-071967579551/volumes"
Oct 30 13:10:21.105134 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:21.104481    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b72c176a-dc4d-4b4e-82e9-b1e15fceaed3" path="/var/lib/kubelet/pods/b72c176a-dc4d-4b4e-82e9-b1e15fceaed3/volumes"
Oct 30 13:10:21.105830 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:21.105806    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g" event={"ID":"3a19094f-f1fc-4f65-9702-96dc3a0a2f77","Type":"ContainerStarted","Data":"50bca92b8622c87555e4c837d5ace1373ea410b1980770fee53d7ff383249a05"}
Oct 30 13:10:21.105991 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:21.105975    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g" event={"ID":"3a19094f-f1fc-4f65-9702-96dc3a0a2f77","Type":"ContainerStarted","Data":"6076272dedd3cfac89c8a51174d182739ac8e6e80bb76b8682d5472eb7501a34"}
Oct 30 13:10:21.106107 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:21.106096    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g"
Oct 30 13:10:21.165718 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:21.165642    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g" podStartSLOduration=2.683219584 podStartE2EDuration="3.165622968s" totalImagesPullingTime="482.403384ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:21.16255598 +0000 UTC m=+1516.209689931" watchObservedRunningTime="2026-10-30 13:10:21.165622968 +0000 UTC m=+1516.212756921"
Oct 30 13:10:21.707325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:21.707287    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 13:10:21.707542 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:21.707339    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h" podUID="565c6710-ac33-437a-b2cc-5572a325523d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 13:10:21.885628 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:21.885585    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:22.042826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:22.042730    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:22.707599 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:22.707561    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 13:10:22.707768 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:22.707641    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h" podUID="565c6710-ac33-437a-b2cc-5572a325523d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 13:10:23.366057 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:23.366011    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:23.673442 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:23.673341    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:10:23.711831 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:23.711799    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:10:24.157299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:24.157250    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:10:25.149909 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:25.149872    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:26.047869 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:26.047834    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"]
Oct 30 13:10:26.048095 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:26.048042    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf" podUID="1fc6a6f0-9e93-41c0-a0db-eee647f4082b" containerName="istio-proxy" containerID="cri-o://446a5bf8d94854172fa8207889d6670f4db80fb8c94de7a6f4fa35442805f1ca" gracePeriod=30
Oct 30 13:10:26.180743 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:26.180711    4544 generic.go:356] "Generic (PLEG): container finished" podID="b56cc33e-52a6-4f72-b0b4-b438e5ea24d6" containerID="2b81707a031b5c82769d5aee00a697487f3780d697793daf3cee864d856c1008" exitCode=0
Oct 30 13:10:26.181138 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:26.180787    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r" event={"ID":"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6","Type":"ContainerDied","Data":"2b81707a031b5c82769d5aee00a697487f3780d697793daf3cee864d856c1008"}
Oct 30 13:10:26.181329 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:26.181313    4544 scope.go:122] "RemoveContainer" containerID="2b81707a031b5c82769d5aee00a697487f3780d697793daf3cee864d856c1008"
Oct 30 13:10:26.183236 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:26.183191    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:10:26.183347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:26.183280    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:10:26.183347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:26.183315    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:10:26.596896 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:26.596864    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc"
Oct 30 13:10:26.597087 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:26.596975    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc"
Oct 30 13:10:26.862488 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:26.862404    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:26.993424 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:26.993354    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"]
Oct 30 13:10:26.993984 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:26.993949    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf" podUID="3c508330-60f7-4a7c-abf6-26bd9c811e53" containerName="istio-proxy" containerID="cri-o://8e34f4759644a58fd0e623065192fc71f5e40fa721e4eacee4059ae33ca73884" gracePeriod=30
Oct 30 13:10:27.011705 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:27.002500    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"]
Oct 30 13:10:27.011705 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:27.002813    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph" podUID="df98822e-ff7b-4c1d-ad2e-1e3b88523780" containerName="istio-proxy" containerID="cri-o://b69d7d866f74dac1bace741db7894472244bbeecd92a111c60b6133cfcda4407" gracePeriod=30
Oct 30 13:10:27.193471 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:27.193376    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r" event={"ID":"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6","Type":"ContainerStarted","Data":"f190536b814f732fbc308b51bd80ac9f38c60d2684e77481bf40df29eec8e6e1"}
Oct 30 13:10:28.067264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:28.067207    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:10:28.067557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:28.067528    4544 patch_prober.go:29] interesting pod/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 30 13:10:28.067627 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:28.067604    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r" podUID="b56cc33e-52a6-4f72-b0b4-b438e5ea24d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 30 13:10:28.154432 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:28.154400    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g"
Oct 30 13:10:28.807117 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:28.807082    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:29.045753 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:10:29.045719    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:10:29.068279 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:29.068177    4544 patch_prober.go:29] interesting pod/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 30 13:10:29.068279 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:29.068255    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r" podUID="b56cc33e-52a6-4f72-b0b4-b438e5ea24d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 30 13:10:30.068117 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:30.068077    4544 patch_prober.go:29] interesting pod/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 30 13:10:30.068546 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:30.068148    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r" podUID="b56cc33e-52a6-4f72-b0b4-b438e5ea24d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 30 13:10:30.696636 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:30.696602    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:31.067689 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:31.067657    4544 patch_prober.go:29] interesting pod/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 30 13:10:31.067841 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:31.067732    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r" podUID="b56cc33e-52a6-4f72-b0b4-b438e5ea24d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 30 13:10:31.266099 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:31.265985    4544 generic.go:356] "Generic (PLEG): container finished" podID="1fc6a6f0-9e93-41c0-a0db-eee647f4082b" containerID="446a5bf8d94854172fa8207889d6670f4db80fb8c94de7a6f4fa35442805f1ca" exitCode=0
Oct 30 13:10:31.266099 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:31.266057    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf" event={"ID":"1fc6a6f0-9e93-41c0-a0db-eee647f4082b","Type":"ContainerDied","Data":"446a5bf8d94854172fa8207889d6670f4db80fb8c94de7a6f4fa35442805f1ca"}
Oct 30 13:10:31.388559 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:31.388535    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:10:31.467441 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:31.467103    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-workload-certs\") pod \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") "
Oct 30 13:10:31.467441 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:31.467162    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istio-token\" (UniqueName: \"kubernetes.io/projected/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istio-token\") pod \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") "
Oct 30 13:10:31.467441 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:31.467237    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istio-data\") pod \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") "
Oct 30 13:10:31.467846 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:31.467707    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istio-data" pod "1fc6a6f0-9e93-41c0-a0db-eee647f4082b" (UID: "1fc6a6f0-9e93-41c0-a0db-eee647f4082b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:31.468187 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:31.467933    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-workload-certs" pod "1fc6a6f0-9e93-41c0-a0db-eee647f4082b" (UID: "1fc6a6f0-9e93-41c0-a0db-eee647f4082b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:31.468534 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:31.468516    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-kube-api-access-lh5gg\" (UniqueName: \"kubernetes.io/projected/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-kube-api-access-lh5gg\") pod \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") "
Oct 30 13:10:31.469038 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:31.469010    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istio-podinfo\") pod \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") "
Oct 30 13:10:31.469507 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:31.469492    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-workload-socket\") pod \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") "
Oct 30 13:10:31.469924 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:31.469908    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istio-envoy\") pod \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") "
Oct 30 13:10:31.470170 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:31.469862    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-workload-socket" pod "1fc6a6f0-9e93-41c0-a0db-eee647f4082b" (UID: "1fc6a6f0-9e93-41c0-a0db-eee647f4082b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:31.471606 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:31.471579    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-credential-socket\") pod \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") "
Oct 30 13:10:31.471984 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:31.471777    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istiod-ca-cert\") pod \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\" (UID: \"1fc6a6f0-9e93-41c0-a0db-eee647f4082b\") "
Oct 30 13:10:31.472537 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:31.472512    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istiod-ca-cert" pod "1fc6a6f0-9e93-41c0-a0db-eee647f4082b" (UID: "1fc6a6f0-9e93-41c0-a0db-eee647f4082b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:31.473256 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:31.473034    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-credential-socket" pod "1fc6a6f0-9e93-41c0-a0db-eee647f4082b" (UID: "1fc6a6f0-9e93-41c0-a0db-eee647f4082b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:31.473697 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:31.473679    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:31.473697 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:31.473698    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:31.473847 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:31.473707    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:31.473847 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:31.473716    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:31.473847 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:31.473724    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:31.477006 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:31.476968    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istio-envoy" pod "1fc6a6f0-9e93-41c0-a0db-eee647f4082b" (UID: "1fc6a6f0-9e93-41c0-a0db-eee647f4082b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:31.477132 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:31.477106    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istio-token" pod "1fc6a6f0-9e93-41c0-a0db-eee647f4082b" (UID: "1fc6a6f0-9e93-41c0-a0db-eee647f4082b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:31.477214 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:31.477198    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istio-podinfo" pod "1fc6a6f0-9e93-41c0-a0db-eee647f4082b" (UID: "1fc6a6f0-9e93-41c0-a0db-eee647f4082b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:10:31.477377 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:31.477356    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-kube-api-access-lh5gg" pod "1fc6a6f0-9e93-41c0-a0db-eee647f4082b" (UID: "1fc6a6f0-9e93-41c0-a0db-eee647f4082b"). InnerVolumeSpecName "kube-api-access-lh5gg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:31.576480 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:31.576380    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:31.576480 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:31.576414    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh5gg\" (UniqueName: \"kubernetes.io/projected/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-kube-api-access-lh5gg\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:31.576480 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:31.576430    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:31.576480 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:31.576443    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fc6a6f0-9e93-41c0-a0db-eee647f4082b-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:32.068179 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.068146    4544 patch_prober.go:29] interesting pod/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 30 13:10:32.068377 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.068202    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r" podUID="b56cc33e-52a6-4f72-b0b4-b438e5ea24d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 30 13:10:32.284115 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.283666    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"
Oct 30 13:10:32.284115 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.283836    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf" event={"ID":"1fc6a6f0-9e93-41c0-a0db-eee647f4082b","Type":"ContainerDied","Data":"ab6e42f87c1b83722edc02593d78c75a50f1cb7636d489ea6b2fa2a2e9c8399f"}
Oct 30 13:10:32.284115 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.283973    4544 scope.go:122] "RemoveContainer" containerID="446a5bf8d94854172fa8207889d6670f4db80fb8c94de7a6f4fa35442805f1ca"
Oct 30 13:10:32.292828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.292691    4544 generic.go:356] "Generic (PLEG): container finished" podID="3c508330-60f7-4a7c-abf6-26bd9c811e53" containerID="8e34f4759644a58fd0e623065192fc71f5e40fa721e4eacee4059ae33ca73884" exitCode=0
Oct 30 13:10:32.292940 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.292891    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf" event={"ID":"3c508330-60f7-4a7c-abf6-26bd9c811e53","Type":"ContainerDied","Data":"8e34f4759644a58fd0e623065192fc71f5e40fa721e4eacee4059ae33ca73884"}
Oct 30 13:10:32.296072 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.295615    4544 generic.go:356] "Generic (PLEG): container finished" podID="df98822e-ff7b-4c1d-ad2e-1e3b88523780" containerID="b69d7d866f74dac1bace741db7894472244bbeecd92a111c60b6133cfcda4407" exitCode=0
Oct 30 13:10:32.296072 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.295853    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph" event={"ID":"df98822e-ff7b-4c1d-ad2e-1e3b88523780","Type":"ContainerDied","Data":"b69d7d866f74dac1bace741db7894472244bbeecd92a111c60b6133cfcda4407"}
Oct 30 13:10:32.369250 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.369202    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"]
Oct 30 13:10:32.378465 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.378439    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf"]
Oct 30 13:10:32.456364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.455920    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:10:32.460703 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.460634    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:10:32.509870 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.509846    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istiod-ca-cert\") pod \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") "
Oct 30 13:10:32.510421 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.510396    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istio-envoy\") pod \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") "
Oct 30 13:10:32.510832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.510680    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istiod-ca-cert" pod "df98822e-ff7b-4c1d-ad2e-1e3b88523780" (UID: "df98822e-ff7b-4c1d-ad2e-1e3b88523780"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:32.510980 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.510966    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istio-token\" (UniqueName: \"kubernetes.io/projected/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istio-token\") pod \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") "
Oct 30 13:10:32.511087 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.511077    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-workload-certs\") pod \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") "
Oct 30 13:10:32.511163 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.511153    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-workload-socket\") pod \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") "
Oct 30 13:10:32.511363 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.511349    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istio-podinfo\") pod \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") "
Oct 30 13:10:32.511499 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.511489    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istio-data\") pod \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") "
Oct 30 13:10:32.511628 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.511618    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df98822e-ff7b-4c1d-ad2e-1e3b88523780-kube-api-access-wmwng\" (UniqueName: \"kubernetes.io/projected/df98822e-ff7b-4c1d-ad2e-1e3b88523780-kube-api-access-wmwng\") pod \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") "
Oct 30 13:10:32.511696 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.511351    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-workload-certs" pod "df98822e-ff7b-4c1d-ad2e-1e3b88523780" (UID: "df98822e-ff7b-4c1d-ad2e-1e3b88523780"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:32.511782 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.511470    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-workload-socket" pod "df98822e-ff7b-4c1d-ad2e-1e3b88523780" (UID: "df98822e-ff7b-4c1d-ad2e-1e3b88523780"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:32.511959 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.511900    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istio-data" pod "df98822e-ff7b-4c1d-ad2e-1e3b88523780" (UID: "df98822e-ff7b-4c1d-ad2e-1e3b88523780"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:32.512048 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.512024    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-credential-socket\") pod \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\" (UID: \"df98822e-ff7b-4c1d-ad2e-1e3b88523780\") "
Oct 30 13:10:32.533979 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.533667    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:32.533979 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.533704    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:32.533979 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.533719    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:32.533979 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.533735    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:32.534457 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.534401    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-credential-socket" pod "df98822e-ff7b-4c1d-ad2e-1e3b88523780" (UID: "df98822e-ff7b-4c1d-ad2e-1e3b88523780"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:32.536738 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.536699    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istio-envoy" pod "df98822e-ff7b-4c1d-ad2e-1e3b88523780" (UID: "df98822e-ff7b-4c1d-ad2e-1e3b88523780"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:32.549307 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.547626    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df98822e-ff7b-4c1d-ad2e-1e3b88523780-kube-api-access-wmwng" pod "df98822e-ff7b-4c1d-ad2e-1e3b88523780" (UID: "df98822e-ff7b-4c1d-ad2e-1e3b88523780"). InnerVolumeSpecName "kube-api-access-wmwng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:32.549307 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.547993    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istio-podinfo" pod "df98822e-ff7b-4c1d-ad2e-1e3b88523780" (UID: "df98822e-ff7b-4c1d-ad2e-1e3b88523780"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:10:32.549307 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.548096    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istio-token" pod "df98822e-ff7b-4c1d-ad2e-1e3b88523780" (UID: "df98822e-ff7b-4c1d-ad2e-1e3b88523780"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:32.637276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.634798    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-workload-socket\") pod \"3c508330-60f7-4a7c-abf6-26bd9c811e53\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") "
Oct 30 13:10:32.637276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.634871    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c508330-60f7-4a7c-abf6-26bd9c811e53-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c508330-60f7-4a7c-abf6-26bd9c811e53-istiod-ca-cert\") pod \"3c508330-60f7-4a7c-abf6-26bd9c811e53\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") "
Oct 30 13:10:32.637276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.634959    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-istio-envoy\") pod \"3c508330-60f7-4a7c-abf6-26bd9c811e53\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") "
Oct 30 13:10:32.637276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.635074    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-istio-data\") pod \"3c508330-60f7-4a7c-abf6-26bd9c811e53\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") "
Oct 30 13:10:32.637276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.635136    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c508330-60f7-4a7c-abf6-26bd9c811e53-kube-api-access-5wlqs\" (UniqueName: \"kubernetes.io/projected/3c508330-60f7-4a7c-abf6-26bd9c811e53-kube-api-access-5wlqs\") pod \"3c508330-60f7-4a7c-abf6-26bd9c811e53\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") "
Oct 30 13:10:32.637276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.635243    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c508330-60f7-4a7c-abf6-26bd9c811e53-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c508330-60f7-4a7c-abf6-26bd9c811e53-istio-podinfo\") pod \"3c508330-60f7-4a7c-abf6-26bd9c811e53\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") "
Oct 30 13:10:32.637276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.635278    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-credential-socket\") pod \"3c508330-60f7-4a7c-abf6-26bd9c811e53\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") "
Oct 30 13:10:32.637276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.635361    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c508330-60f7-4a7c-abf6-26bd9c811e53-istio-token\" (UniqueName: \"kubernetes.io/projected/3c508330-60f7-4a7c-abf6-26bd9c811e53-istio-token\") pod \"3c508330-60f7-4a7c-abf6-26bd9c811e53\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") "
Oct 30 13:10:32.637276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.635465    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-workload-certs\") pod \"3c508330-60f7-4a7c-abf6-26bd9c811e53\" (UID: \"3c508330-60f7-4a7c-abf6-26bd9c811e53\") "
Oct 30 13:10:32.637276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.636342    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c508330-60f7-4a7c-abf6-26bd9c811e53-istiod-ca-cert" pod "3c508330-60f7-4a7c-abf6-26bd9c811e53" (UID: "3c508330-60f7-4a7c-abf6-26bd9c811e53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:32.638052 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.638023    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-workload-socket" pod "3c508330-60f7-4a7c-abf6-26bd9c811e53" (UID: "3c508330-60f7-4a7c-abf6-26bd9c811e53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:32.638325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.638278    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-credential-socket" pod "3c508330-60f7-4a7c-abf6-26bd9c811e53" (UID: "3c508330-60f7-4a7c-abf6-26bd9c811e53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:32.638480 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.638349    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-istio-data" pod "3c508330-60f7-4a7c-abf6-26bd9c811e53" (UID: "3c508330-60f7-4a7c-abf6-26bd9c811e53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:32.638961 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.638933    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-workload-certs" pod "3c508330-60f7-4a7c-abf6-26bd9c811e53" (UID: "3c508330-60f7-4a7c-abf6-26bd9c811e53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:32.639638 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.639612    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:32.639705 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.639660    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmwng\" (UniqueName: \"kubernetes.io/projected/df98822e-ff7b-4c1d-ad2e-1e3b88523780-kube-api-access-wmwng\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:32.639705 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.639677    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:32.639705 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.639691    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c508330-60f7-4a7c-abf6-26bd9c811e53-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:32.639816 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.639708    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:32.639816 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.639722    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:32.639816 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.639734    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:32.639816 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.639746    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:32.639816 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.639759    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:32.639816 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.639772    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df98822e-ff7b-4c1d-ad2e-1e3b88523780-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:32.640721 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.640653    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c508330-60f7-4a7c-abf6-26bd9c811e53-istio-podinfo" pod "3c508330-60f7-4a7c-abf6-26bd9c811e53" (UID: "3c508330-60f7-4a7c-abf6-26bd9c811e53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:10:32.640721 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.640690    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-istio-envoy" pod "3c508330-60f7-4a7c-abf6-26bd9c811e53" (UID: "3c508330-60f7-4a7c-abf6-26bd9c811e53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:32.640832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.640757    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c508330-60f7-4a7c-abf6-26bd9c811e53-istio-token" pod "3c508330-60f7-4a7c-abf6-26bd9c811e53" (UID: "3c508330-60f7-4a7c-abf6-26bd9c811e53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:32.642480 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.642461    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c508330-60f7-4a7c-abf6-26bd9c811e53-kube-api-access-5wlqs" pod "3c508330-60f7-4a7c-abf6-26bd9c811e53" (UID: "3c508330-60f7-4a7c-abf6-26bd9c811e53"). InnerVolumeSpecName "kube-api-access-5wlqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:32.741183 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.741152    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c508330-60f7-4a7c-abf6-26bd9c811e53-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:32.741506 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.741493    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c508330-60f7-4a7c-abf6-26bd9c811e53-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:32.741602 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.741594    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wlqs\" (UniqueName: \"kubernetes.io/projected/3c508330-60f7-4a7c-abf6-26bd9c811e53-kube-api-access-5wlqs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:32.741679 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.741670    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c508330-60f7-4a7c-abf6-26bd9c811e53-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:10:32.880945 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:32.880911    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:33.068239 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:33.068177    4544 patch_prober.go:29] interesting pod/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 30 13:10:33.068458 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:33.068278    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r" podUID="b56cc33e-52a6-4f72-b0b4-b438e5ea24d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 30 13:10:33.090352 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:33.090318    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fc6a6f0-9e93-41c0-a0db-eee647f4082b" path="/var/lib/kubelet/pods/1fc6a6f0-9e93-41c0-a0db-eee647f4082b/volumes"
Oct 30 13:10:33.305864 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:33.305776    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"
Oct 30 13:10:33.305864 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:33.305785    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf" event={"ID":"3c508330-60f7-4a7c-abf6-26bd9c811e53","Type":"ContainerDied","Data":"5dacab7425e384945c0ac6c9b7871bf95ffa9b1643665af57a6b5a9600363411"}
Oct 30 13:10:33.305864 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:33.305859    4544 scope.go:122] "RemoveContainer" containerID="8e34f4759644a58fd0e623065192fc71f5e40fa721e4eacee4059ae33ca73884"
Oct 30 13:10:33.307314 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:33.307300    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"
Oct 30 13:10:33.307402 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:33.307345    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph" event={"ID":"df98822e-ff7b-4c1d-ad2e-1e3b88523780","Type":"ContainerDied","Data":"247b8d2da5d18724d2b531df344b95e8399c982754a7832183f8a8e45214d0be"}
Oct 30 13:10:33.316556 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:33.316539    4544 scope.go:122] "RemoveContainer" containerID="b69d7d866f74dac1bace741db7894472244bbeecd92a111c60b6133cfcda4407"
Oct 30 13:10:33.358593 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:33.358558    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"]
Oct 30 13:10:33.371593 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:33.371559    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf"]
Oct 30 13:10:33.396082 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:33.396049    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"]
Oct 30 13:10:33.406050 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:33.401966    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph"]
Oct 30 13:10:34.072242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:34.072200    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:10:34.072769 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:34.072749    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:10:34.073444 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:34.073425    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:10:34.188417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:34.188357    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:35.028891 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:35.028857    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:35.092770 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:35.092736    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c508330-60f7-4a7c-abf6-26bd9c811e53" path="/var/lib/kubelet/pods/3c508330-60f7-4a7c-abf6-26bd9c811e53/volumes"
Oct 30 13:10:35.093434 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:35.093416    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df98822e-ff7b-4c1d-ad2e-1e3b88523780" path="/var/lib/kubelet/pods/df98822e-ff7b-4c1d-ad2e-1e3b88523780/volumes"
Oct 30 13:10:37.594977 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:37.594880    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:37.740401 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:37.740344    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:37.778481 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:37.752293    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:37.886304 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:37.886062    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:37.971137 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:37.971101    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:38.728417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:38.728294    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:39.148891 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.148854    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"]
Oct 30 13:10:39.149714 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.149696    4544 cpu_manager.go:414] "containerMap: removing container" podUID="df98822e-ff7b-4c1d-ad2e-1e3b88523780" containerName="istio-proxy"
Oct 30 13:10:39.149790 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.149715    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df98822e-ff7b-4c1d-ad2e-1e3b88523780" containerName="istio-proxy"
Oct 30 13:10:39.149790 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.149761    4544 cpu_manager.go:414] "containerMap: removing container" podUID="3c508330-60f7-4a7c-abf6-26bd9c811e53" containerName="istio-proxy"
Oct 30 13:10:39.149790 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.149774    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c508330-60f7-4a7c-abf6-26bd9c811e53" containerName="istio-proxy"
Oct 30 13:10:39.149790 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.149781    4544 cpu_manager.go:414] "containerMap: removing container" podUID="1fc6a6f0-9e93-41c0-a0db-eee647f4082b" containerName="istio-proxy"
Oct 30 13:10:39.149790 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.149786    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fc6a6f0-9e93-41c0-a0db-eee647f4082b" containerName="istio-proxy"
Oct 30 13:10:39.149962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.149906    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c508330-60f7-4a7c-abf6-26bd9c811e53" containerName="istio-proxy"
Oct 30 13:10:39.149962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.149915    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fc6a6f0-9e93-41c0-a0db-eee647f4082b" containerName="istio-proxy"
Oct 30 13:10:39.149962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.149924    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="df98822e-ff7b-4c1d-ad2e-1e3b88523780" containerName="istio-proxy"
Oct 30 13:10:39.157236 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.157203    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:10:39.186040 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.186011    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-31e10b-openshift-default-dockercfg-9j4b6\""
Oct 30 13:10:39.186811 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.186784    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"]
Oct 30 13:10:39.190153 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.190114    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/613b391b-f5e4-4c93-84ba-70d00d691d27-istio-token\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:10:39.190337 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.190215    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/613b391b-f5e4-4c93-84ba-70d00d691d27-istio-podinfo\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:10:39.190414 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.190380    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-istio-data\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:10:39.190478 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.190444    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-workload-certs\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:10:39.190807 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.190779    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-istio-envoy\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:10:39.191078 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.190927    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-credential-socket\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:10:39.191078 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.190981    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/613b391b-f5e4-4c93-84ba-70d00d691d27-istiod-ca-cert\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:10:39.191078 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.191013    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqcsq\" (UniqueName: \"kubernetes.io/projected/613b391b-f5e4-4c93-84ba-70d00d691d27-kube-api-access-gqcsq\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:10:39.191732 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.191709    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-workload-socket\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:10:39.220826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.220772    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:39.293830 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.293791    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/613b391b-f5e4-4c93-84ba-70d00d691d27-istio-podinfo\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:10:39.294049 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.293854    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-istio-data\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:10:39.294049 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.293898    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-workload-certs\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:10:39.294049 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.294004    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-istio-envoy\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:10:39.294198 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.294080    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-credential-socket\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:10:39.294198 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.294136    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/613b391b-f5e4-4c93-84ba-70d00d691d27-istiod-ca-cert\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:10:39.294198 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.294167    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqcsq\" (UniqueName: \"kubernetes.io/projected/613b391b-f5e4-4c93-84ba-70d00d691d27-kube-api-access-gqcsq\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:10:39.294353 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.294327    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-workload-socket\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:10:39.294433 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.294412    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-istio-data\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:10:39.294490 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.294447    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-workload-certs\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:10:39.294534 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.294522    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/613b391b-f5e4-4c93-84ba-70d00d691d27-istio-token\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:10:39.294709 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.294690    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-credential-socket\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:10:39.294822 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.294802    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-workload-socket\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:10:39.295113 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.295092    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/613b391b-f5e4-4c93-84ba-70d00d691d27-istiod-ca-cert\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:10:39.296763 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.296726    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-istio-envoy\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:10:39.297112 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.297087    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/613b391b-f5e4-4c93-84ba-70d00d691d27-istio-podinfo\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:10:39.318381 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.318335    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqcsq\" (UniqueName: \"kubernetes.io/projected/613b391b-f5e4-4c93-84ba-70d00d691d27-kube-api-access-gqcsq\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:10:39.319258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.319209    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/613b391b-f5e4-4c93-84ba-70d00d691d27-istio-token\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:10:39.353776 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.353716    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:39.501295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:39.487344    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:10:40.135283 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:40.135166    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"]
Oct 30 13:10:40.139145 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:10:40.139111    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod613b391b_f5e4_4c93_84ba_70d00d691d27.slice/crio-01af02814413c9477fc2a80bf558a94da0bbf93388d32aaa3d02d75e44b07ab7 WatchSource:0}: Error finding container 01af02814413c9477fc2a80bf558a94da0bbf93388d32aaa3d02d75e44b07ab7: Status 404 returned error can't find the container with id 01af02814413c9477fc2a80bf558a94da0bbf93388d32aaa3d02d75e44b07ab7
Oct 30 13:10:40.142481 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:40.142442    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:10:40.142585 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:40.142520    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:10:40.142585 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:40.142550    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:10:40.207022 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:40.206983    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:40.707611 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:40.707566    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz" event={"ID":"613b391b-f5e4-4c93-84ba-70d00d691d27","Type":"ContainerStarted","Data":"fdd8a7f89d9f07b8ba011c6087fbbadce69c965e14cb3f316853059a30a8b02a"}
Oct 30 13:10:40.707837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:40.707627    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz" event={"ID":"613b391b-f5e4-4c93-84ba-70d00d691d27","Type":"ContainerStarted","Data":"01af02814413c9477fc2a80bf558a94da0bbf93388d32aaa3d02d75e44b07ab7"}
Oct 30 13:10:40.822637 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:40.822579    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz" podStartSLOduration=1.822559822 podStartE2EDuration="1.822559822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:40.80961426 +0000 UTC m=+1535.856748210" watchObservedRunningTime="2026-10-30 13:10:40.822559822 +0000 UTC m=+1535.869693775"
Oct 30 13:10:41.509835 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:41.509793    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:10:41.516408 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:41.516377    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:10:41.716372 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:41.716338    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:10:41.717400 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:41.717379    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:10:42.348041 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:42.347997    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:42.573257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:42.573168    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:43.313885 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:43.313850    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:43.451902 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:43.451867    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:44.570498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:44.570454    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:45.129976 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:45.129914    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:45.259011 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:45.258973    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:45.303504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:45.303468    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b"]
Oct 30 13:10:45.309601 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:45.309568    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b"
Oct 30 13:10:45.330791 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:45.330753    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b"]
Oct 30 13:10:45.337249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:45.337198    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-31e10b-dockercfg-m4tjb\""
Oct 30 13:10:45.337589 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:45.337569    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-31e10b\""
Oct 30 13:10:45.409458 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:45.409364    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phcqq\" (UniqueName: \"kubernetes.io/projected/b97f8cd4-df63-413d-8955-bfd05fcf2ac8-kube-api-access-phcqq\") pod \"payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b\" (UID: \"b97f8cd4-df63-413d-8955-bfd05fcf2ac8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b"
Oct 30 13:10:45.410334 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:45.410300    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b97f8cd4-df63-413d-8955-bfd05fcf2ac8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b\" (UID: \"b97f8cd4-df63-413d-8955-bfd05fcf2ac8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b"
Oct 30 13:10:45.428430 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:45.428361    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc"]
Oct 30 13:10:45.440551 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:45.440517    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc"
Oct 30 13:10:45.487423 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:45.487384    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc"]
Oct 30 13:10:45.556535 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:45.520761    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmq88\" (UniqueName: \"kubernetes.io/projected/97315c62-dbfc-4e18-aca2-34ff9e6d2a75-kube-api-access-bmq88\") pod \"payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc\" (UID: \"97315c62-dbfc-4e18-aca2-34ff9e6d2a75\") " pod="openshift-ingress/payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc"
Oct 30 13:10:45.556535 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:45.554130    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b97f8cd4-df63-413d-8955-bfd05fcf2ac8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b\" (UID: \"b97f8cd4-df63-413d-8955-bfd05fcf2ac8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b"
Oct 30 13:10:45.556535 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:45.554239    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97315c62-dbfc-4e18-aca2-34ff9e6d2a75-plugins-config-volume\") pod \"payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc\" (UID: \"97315c62-dbfc-4e18-aca2-34ff9e6d2a75\") " pod="openshift-ingress/payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc"
Oct 30 13:10:45.556535 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:45.554302    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phcqq\" (UniqueName: \"kubernetes.io/projected/b97f8cd4-df63-413d-8955-bfd05fcf2ac8-kube-api-access-phcqq\") pod \"payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b\" (UID: \"b97f8cd4-df63-413d-8955-bfd05fcf2ac8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b"
Oct 30 13:10:45.568806 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:45.568537    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b97f8cd4-df63-413d-8955-bfd05fcf2ac8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b\" (UID: \"b97f8cd4-df63-413d-8955-bfd05fcf2ac8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b"
Oct 30 13:10:45.686309 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:45.637979    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phcqq\" (UniqueName: \"kubernetes.io/projected/b97f8cd4-df63-413d-8955-bfd05fcf2ac8-kube-api-access-phcqq\") pod \"payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b\" (UID: \"b97f8cd4-df63-413d-8955-bfd05fcf2ac8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b"
Oct 30 13:10:45.686309 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:45.656694    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97315c62-dbfc-4e18-aca2-34ff9e6d2a75-plugins-config-volume\") pod \"payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc\" (UID: \"97315c62-dbfc-4e18-aca2-34ff9e6d2a75\") " pod="openshift-ingress/payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc"
Oct 30 13:10:45.686309 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:45.656839    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmq88\" (UniqueName: \"kubernetes.io/projected/97315c62-dbfc-4e18-aca2-34ff9e6d2a75-kube-api-access-bmq88\") pod \"payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc\" (UID: \"97315c62-dbfc-4e18-aca2-34ff9e6d2a75\") " pod="openshift-ingress/payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc"
Oct 30 13:10:45.727569 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:45.727532    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97315c62-dbfc-4e18-aca2-34ff9e6d2a75-plugins-config-volume\") pod \"payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc\" (UID: \"97315c62-dbfc-4e18-aca2-34ff9e6d2a75\") " pod="openshift-ingress/payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc"
Oct 30 13:10:45.792301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:45.788554    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmq88\" (UniqueName: \"kubernetes.io/projected/97315c62-dbfc-4e18-aca2-34ff9e6d2a75-kube-api-access-bmq88\") pod \"payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc\" (UID: \"97315c62-dbfc-4e18-aca2-34ff9e6d2a75\") " pod="openshift-ingress/payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc"
Oct 30 13:10:45.876285 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:45.876197    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc"
Oct 30 13:10:45.926003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:45.925969    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b"
Oct 30 13:10:46.187464 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:46.187439    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc"]
Oct 30 13:10:46.190109 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:10:46.190043    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97315c62_dbfc_4e18_aca2_34ff9e6d2a75.slice/crio-e9b38ad4b0792e92e5da50b3a8ba1923feb83ca79807323e3d26597c0214364c WatchSource:0}: Error finding container e9b38ad4b0792e92e5da50b3a8ba1923feb83ca79807323e3d26597c0214364c: Status 404 returned error can't find the container with id e9b38ad4b0792e92e5da50b3a8ba1923feb83ca79807323e3d26597c0214364c
Oct 30 13:10:46.262497 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:46.262474    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b"]
Oct 30 13:10:46.264366 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:10:46.264338    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb97f8cd4_df63_413d_8955_bfd05fcf2ac8.slice/crio-f9579c330811e3cf6f8646ecd19db2338fea5ed57e26da47e8fd5d7441d917f0 WatchSource:0}: Error finding container f9579c330811e3cf6f8646ecd19db2338fea5ed57e26da47e8fd5d7441d917f0: Status 404 returned error can't find the container with id f9579c330811e3cf6f8646ecd19db2338fea5ed57e26da47e8fd5d7441d917f0
Oct 30 13:10:46.738655 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:46.738558    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7"]
Oct 30 13:10:46.743866 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:46.743837    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7"
Oct 30 13:10:46.758502 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:46.758463    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f3160a-serving-cert\""
Oct 30 13:10:46.771753 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:46.771717    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7"]
Oct 30 13:10:46.788809 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:46.787985    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75449fe5-8d9e-4c97-97b3-59929a10f0d5-openshift-service-ca\") pod \"maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7\" (UID: \"75449fe5-8d9e-4c97-97b3-59929a10f0d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7"
Oct 30 13:10:46.788809 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:46.788208    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frrgk\" (UniqueName: \"kubernetes.io/projected/75449fe5-8d9e-4c97-97b3-59929a10f0d5-kube-api-access-frrgk\") pod \"maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7\" (UID: \"75449fe5-8d9e-4c97-97b3-59929a10f0d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7"
Oct 30 13:10:46.788809 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:46.788353    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75449fe5-8d9e-4c97-97b3-59929a10f0d5-metrics-certs\") pod \"maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7\" (UID: \"75449fe5-8d9e-4c97-97b3-59929a10f0d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7"
Oct 30 13:10:46.788809 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:46.788379    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75449fe5-8d9e-4c97-97b3-59929a10f0d5-maas-api-tls\") pod \"maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7\" (UID: \"75449fe5-8d9e-4c97-97b3-59929a10f0d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7"
Oct 30 13:10:46.813167 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:46.813126    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b" event={"ID":"b97f8cd4-df63-413d-8955-bfd05fcf2ac8","Type":"ContainerStarted","Data":"f987e9d33d927333f9d83f889ec8b06b9b4f5c2b9e3bebac763dcf6e025859d4"}
Oct 30 13:10:46.813391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:46.813191    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b" event={"ID":"b97f8cd4-df63-413d-8955-bfd05fcf2ac8","Type":"ContainerStarted","Data":"f9579c330811e3cf6f8646ecd19db2338fea5ed57e26da47e8fd5d7441d917f0"}
Oct 30 13:10:46.813391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:46.813257    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b"
Oct 30 13:10:46.815274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:46.815244    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc" event={"ID":"97315c62-dbfc-4e18-aca2-34ff9e6d2a75","Type":"ContainerStarted","Data":"3f0cc60569923437bc0eb61280c122ed608cfe2170074765925d202265dab217"}
Oct 30 13:10:46.815407 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:46.815298    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc"
Oct 30 13:10:46.815407 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:46.815316    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc" event={"ID":"97315c62-dbfc-4e18-aca2-34ff9e6d2a75","Type":"ContainerStarted","Data":"e9b38ad4b0792e92e5da50b3a8ba1923feb83ca79807323e3d26597c0214364c"}
Oct 30 13:10:46.837588 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:46.837554    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:46.855467 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:46.855411    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b" podStartSLOduration=1.855397495 podStartE2EDuration="1.855397495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:46.855177391 +0000 UTC m=+1541.902311346" watchObservedRunningTime="2026-10-30 13:10:46.855397495 +0000 UTC m=+1541.902531448"
Oct 30 13:10:46.890472 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:46.890434    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75449fe5-8d9e-4c97-97b3-59929a10f0d5-openshift-service-ca\") pod \"maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7\" (UID: \"75449fe5-8d9e-4c97-97b3-59929a10f0d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7"
Oct 30 13:10:46.890663 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:46.890639    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frrgk\" (UniqueName: \"kubernetes.io/projected/75449fe5-8d9e-4c97-97b3-59929a10f0d5-kube-api-access-frrgk\") pod \"maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7\" (UID: \"75449fe5-8d9e-4c97-97b3-59929a10f0d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7"
Oct 30 13:10:46.890772 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:46.890754    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75449fe5-8d9e-4c97-97b3-59929a10f0d5-metrics-certs\") pod \"maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7\" (UID: \"75449fe5-8d9e-4c97-97b3-59929a10f0d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7"
Oct 30 13:10:46.890837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:46.890787    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75449fe5-8d9e-4c97-97b3-59929a10f0d5-maas-api-tls\") pod \"maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7\" (UID: \"75449fe5-8d9e-4c97-97b3-59929a10f0d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7"
Oct 30 13:10:46.891258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:46.891213    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75449fe5-8d9e-4c97-97b3-59929a10f0d5-openshift-service-ca\") pod \"maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7\" (UID: \"75449fe5-8d9e-4c97-97b3-59929a10f0d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7"
Oct 30 13:10:46.893336 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:46.893317    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75449fe5-8d9e-4c97-97b3-59929a10f0d5-metrics-certs\") pod \"maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7\" (UID: \"75449fe5-8d9e-4c97-97b3-59929a10f0d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7"
Oct 30 13:10:46.893460 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:46.893444    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75449fe5-8d9e-4c97-97b3-59929a10f0d5-maas-api-tls\") pod \"maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7\" (UID: \"75449fe5-8d9e-4c97-97b3-59929a10f0d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7"
Oct 30 13:10:46.896034 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:46.895986    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc" podStartSLOduration=1.895968741 podStartE2EDuration="1.895968741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:46.894470056 +0000 UTC m=+1541.941604010" watchObservedRunningTime="2026-10-30 13:10:46.895968741 +0000 UTC m=+1541.943102694"
Oct 30 13:10:46.914133 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:46.914104    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frrgk\" (UniqueName: \"kubernetes.io/projected/75449fe5-8d9e-4c97-97b3-59929a10f0d5-kube-api-access-frrgk\") pod \"maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7\" (UID: \"75449fe5-8d9e-4c97-97b3-59929a10f0d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7"
Oct 30 13:10:46.952364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:46.952329    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:47.064194 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:47.062532    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7"
Oct 30 13:10:47.488924 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:47.488898    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7"]
Oct 30 13:10:47.490977 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:10:47.490953    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75449fe5_8d9e_4c97_97b3_59929a10f0d5.slice/crio-a7210faf5e02de18c75b4b691f4e8b5be1dd8d2f805614774300b75b115a2a2b WatchSource:0}: Error finding container a7210faf5e02de18c75b4b691f4e8b5be1dd8d2f805614774300b75b115a2a2b: Status 404 returned error can't find the container with id a7210faf5e02de18c75b4b691f4e8b5be1dd8d2f805614774300b75b115a2a2b
Oct 30 13:10:47.844401 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:47.844358    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7" event={"ID":"75449fe5-8d9e-4c97-97b3-59929a10f0d5","Type":"ContainerStarted","Data":"a7210faf5e02de18c75b4b691f4e8b5be1dd8d2f805614774300b75b115a2a2b"}
Oct 30 13:10:48.610172 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:48.610132    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:48.857520 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:48.857481    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7" event={"ID":"75449fe5-8d9e-4c97-97b3-59929a10f0d5","Type":"ContainerStarted","Data":"0686d74ee925184433845cfebf937318f1c649ceff7c61869696c6ac5d3833e2"}
Oct 30 13:10:48.857986 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:48.857752    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7"
Oct 30 13:10:48.905929 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:48.905831    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7" podStartSLOduration=2.472214936 podStartE2EDuration="2.905816293s" totalImagesPullingTime="433.601357ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:48.901759893 +0000 UTC m=+1543.948893845" watchObservedRunningTime="2026-10-30 13:10:48.905816293 +0000 UTC m=+1543.952950245"
Oct 30 13:10:48.996962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:48.996919    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:49.163761 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:49.163640    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:49.553755 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:49.551615    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"]
Oct 30 13:10:49.559357 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:49.559328    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:10:49.584337 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:49.584288    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"]
Oct 30 13:10:49.587493 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:49.587458    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-e9d641-openshift-default-dockercfg-jhrsg\""
Oct 30 13:10:49.612677 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:49.612639    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:49.624377 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:49.623590    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-istio-envoy\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:10:49.624377 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:49.623712    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d79e94-7094-44da-a04a-824f9328125e-istiod-ca-cert\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:10:49.624377 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:49.623765    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-istio-data\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:10:49.624377 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:49.623935    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-credential-socket\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:10:49.624377 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:49.623989    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdhgk\" (UniqueName: \"kubernetes.io/projected/02d79e94-7094-44da-a04a-824f9328125e-kube-api-access-xdhgk\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:10:49.624377 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:49.624058    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-workload-socket\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:10:49.624377 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:49.624132    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02d79e94-7094-44da-a04a-824f9328125e-istio-token\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:10:49.624377 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:49.624163    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-workload-certs\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:10:49.624377 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:49.624350    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d79e94-7094-44da-a04a-824f9328125e-istio-podinfo\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:10:49.740246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:49.740026    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d79e94-7094-44da-a04a-824f9328125e-istio-podinfo\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:10:49.740246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:49.740113    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-istio-envoy\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:10:49.740246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:49.740182    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d79e94-7094-44da-a04a-824f9328125e-istiod-ca-cert\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:10:49.740246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:49.740245    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-istio-data\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:10:49.740623 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:49.740364    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-credential-socket\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:10:49.740623 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:49.740406    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdhgk\" (UniqueName: \"kubernetes.io/projected/02d79e94-7094-44da-a04a-824f9328125e-kube-api-access-xdhgk\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:10:49.740623 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:49.740475    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-workload-socket\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:10:49.740623 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:49.740546    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02d79e94-7094-44da-a04a-824f9328125e-istio-token\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:10:49.740623 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:49.740574    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-workload-certs\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:10:49.764247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:49.741139    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-workload-certs\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:10:49.767781 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:49.766138    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d79e94-7094-44da-a04a-824f9328125e-istiod-ca-cert\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:10:49.783676 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:49.768658    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-credential-socket\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:10:49.784723 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:49.784638    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-istio-data\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:10:49.784941 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:49.784843    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-workload-socket\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:10:49.810810 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:49.810678    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-istio-envoy\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:10:49.864734 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:49.836932    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d79e94-7094-44da-a04a-824f9328125e-istio-podinfo\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:10:49.906239 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:49.894054    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02d79e94-7094-44da-a04a-824f9328125e-istio-token\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:10:49.907148 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:49.907119    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdhgk\" (UniqueName: \"kubernetes.io/projected/02d79e94-7094-44da-a04a-824f9328125e-kube-api-access-xdhgk\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:10:50.052145 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:50.052104    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:50.175443 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:50.175403    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:10:50.512025 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:50.511964    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"]
Oct 30 13:10:50.512852 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:10:50.512827    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02d79e94_7094_44da_a04a_824f9328125e.slice/crio-12beb64402e22d718bd6336218c16253c884fb4b0d7d0645cea8925637a6677b WatchSource:0}: Error finding container 12beb64402e22d718bd6336218c16253c884fb4b0d7d0645cea8925637a6677b: Status 404 returned error can't find the container with id 12beb64402e22d718bd6336218c16253c884fb4b0d7d0645cea8925637a6677b
Oct 30 13:10:50.515782 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:50.515740    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:10:50.515884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:50.515819    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:10:50.515884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:50.515848    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:10:50.952387 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:50.952346    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj" event={"ID":"02d79e94-7094-44da-a04a-824f9328125e","Type":"ContainerStarted","Data":"f8dcec6e0c46f9f66b78c1fd2a67d5a55c7b87e3a7423e251b6da5ef84d7fdec"}
Oct 30 13:10:50.952888 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:50.952415    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj" event={"ID":"02d79e94-7094-44da-a04a-824f9328125e","Type":"ContainerStarted","Data":"12beb64402e22d718bd6336218c16253c884fb4b0d7d0645cea8925637a6677b"}
Oct 30 13:10:51.008883 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:51.008827    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj" podStartSLOduration=2.008810378 podStartE2EDuration="2.008810378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:51.007140879 +0000 UTC m=+1546.054274833" watchObservedRunningTime="2026-10-30 13:10:51.008810378 +0000 UTC m=+1546.055944332"
Oct 30 13:10:51.180246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:51.179059    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:10:51.202048 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:51.192553    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:10:51.340422 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:51.340385    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:51.960195 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:51.960161    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:10:51.961312 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:51.961293    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:10:53.151706 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:53.151664    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:53.444735 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:53.444632    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:53.572634 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:53.572596    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:54.500491 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:54.500448    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:54.948231 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:54.948183    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7"
Oct 30 13:10:55.180781 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:55.180735    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:55.345095 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:55.345057    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:55.358015 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:55.357982    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn"]
Oct 30 13:10:55.364488 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:55.364460    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn"
Oct 30 13:10:55.385479 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:55.385442    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-e9d641\""
Oct 30 13:10:55.385479 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:55.385477    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-e9d641-dockercfg-9bcfw\""
Oct 30 13:10:55.393732 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:55.393693    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn"]
Oct 30 13:10:55.493099 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:55.492561    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blc6p\" (UniqueName: \"kubernetes.io/projected/f6d9c074-fc98-4507-822d-7ca79b3770c5-kube-api-access-blc6p\") pod \"payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn\" (UID: \"f6d9c074-fc98-4507-822d-7ca79b3770c5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn"
Oct 30 13:10:55.493099 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:55.492891    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6d9c074-fc98-4507-822d-7ca79b3770c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn\" (UID: \"f6d9c074-fc98-4507-822d-7ca79b3770c5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn"
Oct 30 13:10:55.641308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:55.634606    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47"]
Oct 30 13:10:55.657315 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:55.651574    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blc6p\" (UniqueName: \"kubernetes.io/projected/f6d9c074-fc98-4507-822d-7ca79b3770c5-kube-api-access-blc6p\") pod \"payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn\" (UID: \"f6d9c074-fc98-4507-822d-7ca79b3770c5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn"
Oct 30 13:10:55.657990 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:55.657953    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6d9c074-fc98-4507-822d-7ca79b3770c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn\" (UID: \"f6d9c074-fc98-4507-822d-7ca79b3770c5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn"
Oct 30 13:10:55.685620 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:55.685368    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6d9c074-fc98-4507-822d-7ca79b3770c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn\" (UID: \"f6d9c074-fc98-4507-822d-7ca79b3770c5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn"
Oct 30 13:10:55.751986 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:55.751953    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47"
Oct 30 13:10:55.764156 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:55.764118    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47"]
Oct 30 13:10:55.806245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:55.794138    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blc6p\" (UniqueName: \"kubernetes.io/projected/f6d9c074-fc98-4507-822d-7ca79b3770c5-kube-api-access-blc6p\") pod \"payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn\" (UID: \"f6d9c074-fc98-4507-822d-7ca79b3770c5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn"
Oct 30 13:10:55.868677 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:55.868644    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddlsr\" (UniqueName: \"kubernetes.io/projected/eb920234-9adb-4f7f-a5e4-27faa3603144-kube-api-access-ddlsr\") pod \"payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47\" (UID: \"eb920234-9adb-4f7f-a5e4-27faa3603144\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47"
Oct 30 13:10:55.869114 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:55.869077    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb920234-9adb-4f7f-a5e4-27faa3603144-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47\" (UID: \"eb920234-9adb-4f7f-a5e4-27faa3603144\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47"
Oct 30 13:10:55.970755 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:55.970643    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddlsr\" (UniqueName: \"kubernetes.io/projected/eb920234-9adb-4f7f-a5e4-27faa3603144-kube-api-access-ddlsr\") pod \"payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47\" (UID: \"eb920234-9adb-4f7f-a5e4-27faa3603144\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47"
Oct 30 13:10:55.970948 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:55.970856    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb920234-9adb-4f7f-a5e4-27faa3603144-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47\" (UID: \"eb920234-9adb-4f7f-a5e4-27faa3603144\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47"
Oct 30 13:10:55.971647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:55.971623    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb920234-9adb-4f7f-a5e4-27faa3603144-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47\" (UID: \"eb920234-9adb-4f7f-a5e4-27faa3603144\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47"
Oct 30 13:10:55.981105 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:55.981065    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn"
Oct 30 13:10:55.994030 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:55.993997    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddlsr\" (UniqueName: \"kubernetes.io/projected/eb920234-9adb-4f7f-a5e4-27faa3603144-kube-api-access-ddlsr\") pod \"payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47\" (UID: \"eb920234-9adb-4f7f-a5e4-27faa3603144\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47"
Oct 30 13:10:56.180849 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:56.180808    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47"
Oct 30 13:10:56.306761 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:10:56.306728    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6d9c074_fc98_4507_822d_7ca79b3770c5.slice/crio-293950139e6a0b3706e938d8468598c17fb0c1fdd8718280c5985691db979f5c WatchSource:0}: Error finding container 293950139e6a0b3706e938d8468598c17fb0c1fdd8718280c5985691db979f5c: Status 404 returned error can't find the container with id 293950139e6a0b3706e938d8468598c17fb0c1fdd8718280c5985691db979f5c
Oct 30 13:10:56.308593 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:56.308563    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn"]
Oct 30 13:10:56.501742 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:56.501714    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47"]
Oct 30 13:10:56.503686 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:10:56.503653    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb920234_9adb_4f7f_a5e4_27faa3603144.slice/crio-0f7f0a73b20c944968985f36e6f0402d0451bd37e3d380b0b4c5f55587b0ac3d WatchSource:0}: Error finding container 0f7f0a73b20c944968985f36e6f0402d0451bd37e3d380b0b4c5f55587b0ac3d: Status 404 returned error can't find the container with id 0f7f0a73b20c944968985f36e6f0402d0451bd37e3d380b0b4c5f55587b0ac3d
Oct 30 13:10:57.004072 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:57.004038    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn" event={"ID":"f6d9c074-fc98-4507-822d-7ca79b3770c5","Type":"ContainerStarted","Data":"9a974fb0f58ac4c61fd08b97b7b563886ccc90cc6d32bb856b4c41192bf34794"}
Oct 30 13:10:57.004569 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:57.004101    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn" event={"ID":"f6d9c074-fc98-4507-822d-7ca79b3770c5","Type":"ContainerStarted","Data":"293950139e6a0b3706e938d8468598c17fb0c1fdd8718280c5985691db979f5c"}
Oct 30 13:10:57.004569 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:57.004133    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn"
Oct 30 13:10:57.005878 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:57.005855    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47" event={"ID":"eb920234-9adb-4f7f-a5e4-27faa3603144","Type":"ContainerStarted","Data":"ae684698bd369cffd22ee43a4f483c486f6b6a8c187dcac89bc133c35ade3ec9"}
Oct 30 13:10:57.005995 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:57.005893    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47" event={"ID":"eb920234-9adb-4f7f-a5e4-27faa3603144","Type":"ContainerStarted","Data":"0f7f0a73b20c944968985f36e6f0402d0451bd37e3d380b0b4c5f55587b0ac3d"}
Oct 30 13:10:57.005995 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:57.005914    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47"
Oct 30 13:10:57.045545 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:57.045492    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn" podStartSLOduration=2.045478325 podStartE2EDuration="2.045478325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:57.043250818 +0000 UTC m=+1552.090384770" watchObservedRunningTime="2026-10-30 13:10:57.045478325 +0000 UTC m=+1552.092612278"
Oct 30 13:10:57.094483 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:57.094425    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47" podStartSLOduration=2.094409335 podStartE2EDuration="2.094409335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:57.091751339 +0000 UTC m=+1552.138885295" watchObservedRunningTime="2026-10-30 13:10:57.094409335 +0000 UTC m=+1552.141543328"
Oct 30 13:10:57.255611 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:57.251627    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:10:57.847469 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:57.847439    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b"
Oct 30 13:10:57.847681 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:57.847526    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc"
Oct 30 13:10:59.186557 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:10:59.186511    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:10:59.424345 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:10:59.424302    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:01.522252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:01.507400    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:03.289154 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:03.289122    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:03.443452 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:03.443414    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:03.850861 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:03.850823    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:05.235901 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:05.235105    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:05.369852 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:05.369746    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:05.787858 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:05.787804    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"]
Oct 30 13:11:05.868411 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:05.868245    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:05.898481 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:05.898446    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"]
Oct 30 13:11:05.898817 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:05.898797    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:11:05.931805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:05.931772    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-5b9e8e-openshift-default-dockercfg-qr57z\""
Oct 30 13:11:06.017724 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:06.016440    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57mbt\" (UniqueName: \"kubernetes.io/projected/eaa6149f-3c75-4666-8607-5d2686a65b6d-kube-api-access-57mbt\") pod \"e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") " pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:11:06.018132 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:06.018079    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-istio-envoy\") pod \"e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") " pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:11:06.018348 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:06.018328    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaa6149f-3c75-4666-8607-5d2686a65b6d-istio-token\") pod \"e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") " pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:11:06.018627 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:06.018610    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa6149f-3c75-4666-8607-5d2686a65b6d-istio-podinfo\") pod \"e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") " pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:11:06.018834 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:06.018818    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa6149f-3c75-4666-8607-5d2686a65b6d-istiod-ca-cert\") pod \"e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") " pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:11:06.019309 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:06.019288    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-credential-socket\") pod \"e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") " pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:11:06.019543 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:06.019526    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-istio-data\") pod \"e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") " pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:11:06.019760 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:06.019746    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-workload-certs\") pod \"e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") " pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:11:06.019870 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:06.019857    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-workload-socket\") pod \"e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") " pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:11:06.140289 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:06.138848    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-workload-certs\") pod \"e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") " pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:11:06.140289 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:06.139397    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-workload-certs\") pod \"e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") " pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:11:06.140289 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:06.140060    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-workload-socket\") pod \"e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") " pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:11:06.140289 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:06.139481    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-workload-socket\") pod \"e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") " pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:11:06.140710 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:06.140521    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57mbt\" (UniqueName: \"kubernetes.io/projected/eaa6149f-3c75-4666-8607-5d2686a65b6d-kube-api-access-57mbt\") pod \"e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") " pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:11:06.140710 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:06.140628    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-istio-envoy\") pod \"e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") " pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:11:06.141143 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:06.141010    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaa6149f-3c75-4666-8607-5d2686a65b6d-istio-token\") pod \"e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") " pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:11:06.152942 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:06.141387    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa6149f-3c75-4666-8607-5d2686a65b6d-istio-podinfo\") pod \"e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") " pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:11:06.152942 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:06.141792    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa6149f-3c75-4666-8607-5d2686a65b6d-istiod-ca-cert\") pod \"e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") " pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:11:06.172299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:06.171604    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-istio-envoy\") pod \"e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") " pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:11:06.172299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:06.172251    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa6149f-3c75-4666-8607-5d2686a65b6d-istiod-ca-cert\") pod \"e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") " pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:11:06.175666 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:06.174025    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:06.191023 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:06.190988    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57mbt\" (UniqueName: \"kubernetes.io/projected/eaa6149f-3c75-4666-8607-5d2686a65b6d-kube-api-access-57mbt\") pod \"e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") " pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:11:06.195485 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:06.194142    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-credential-socket\") pod \"e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") " pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:11:06.195485 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:06.194199    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-istio-data\") pod \"e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") " pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:11:06.195485 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:06.194716    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-istio-data\") pod \"e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") " pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:11:06.195893 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:06.195841    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-credential-socket\") pod \"e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") " pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:11:06.202206 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:06.202146    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa6149f-3c75-4666-8607-5d2686a65b6d-istio-podinfo\") pod \"e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") " pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:11:06.205269 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:06.205195    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaa6149f-3c75-4666-8607-5d2686a65b6d-istio-token\") pod \"e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") " pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:11:06.486378 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:06.486347    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:11:07.416511 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:07.416254    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"]
Oct 30 13:11:07.508745 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:07.508627    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:11:07.509241 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:07.508799    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:11:07.509241 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:07.508892    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:11:07.724255 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:07.724192    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:08.026523 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:08.025358    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn"
Oct 30 13:11:08.026759 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:08.026579    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47"
Oct 30 13:11:08.490244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:08.463910    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:08.613604 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:08.613558    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:08.702247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:08.701667    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l" event={"ID":"eaa6149f-3c75-4666-8607-5d2686a65b6d","Type":"ContainerStarted","Data":"59a7cd34b4ea8773e5e0aa9ab6cdf2c6113c07eb072a0a4570382665d7280f58"}
Oct 30 13:11:08.702247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:08.701738    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l" event={"ID":"eaa6149f-3c75-4666-8607-5d2686a65b6d","Type":"ContainerStarted","Data":"c6fa87f045b8e5c52dc31b4d7f52982aa82051a6bf30cfc7c1c7d343aa5f7900"}
Oct 30 13:11:08.910257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:08.908722    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l" podStartSLOduration=3.908700307 podStartE2EDuration="3.908700307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:08.779444002 +0000 UTC m=+1563.826577955" watchObservedRunningTime="2026-10-30 13:11:08.908700307 +0000 UTC m=+1563.955834262"
Oct 30 13:11:09.304149 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:09.304054    4544 scope.go:122] "RemoveContainer" containerID="f3b302a80dbbe2d99a9a505b3bf1b06f64e89ce0c895242ce5a2c307e248a79f"
Oct 30 13:11:09.345249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:09.323680    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl"]
Oct 30 13:11:09.577244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:09.576840    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:11:09.685464 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:09.681539    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:09.701256 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:09.691690    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:11:09.741649 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:09.710568    4544 scope.go:122] "RemoveContainer" containerID="deb1ac6eb807798d16153741b602c257267cec2dba65e7412b62a97f85e8efa6"
Oct 30 13:11:09.741649 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:09.712192    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:11:09.764622 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:09.764569    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:09.809099 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:09.790651    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:09.854302 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:09.851397    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:09.921127 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:09.920818    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:10.035513 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:10.035465    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:10.064628 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:10.064600    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:11:10.089538 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:10.089498    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:11:10.166504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:10.166455    4544 scope.go:122] "RemoveContainer" containerID="3718035008b80be809c9330b72c1f334c74ea3d4db68be5e51605d5e149a855f"
Oct 30 13:11:10.721420 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:10.721390    4544 scope.go:122] "RemoveContainer" containerID="8a3c2b920964a86803c87c0ad8794f3389e0a9fc2e1cd77342b0ffa4965572c3"
Oct 30 13:11:10.838801 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:10.838772    4544 scope.go:122] "RemoveContainer" containerID="4540eee2a30c7ea7d097659dbc328fb822ed67eaba79a0677b2a6158598e0b1f"
Oct 30 13:11:10.863373 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:10.863320    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:10.874247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:10.864210    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:11:10.968423 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:10.966957    4544 scope.go:122] "RemoveContainer" containerID="8c252ad0a4cfef24a293d860b2ed60a068d2a1be7500423af51e94243bb68c97"
Oct 30 13:11:11.576965 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:11.576934    4544 scope.go:122] "RemoveContainer" containerID="0f3b461daf9ac3ec80b1668f1ba2e968d4a9b92fe608fe1811096f1644d26c34"
Oct 30 13:11:11.633593 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:11.615841    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:11.833511 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:11.757539    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:11.833511 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:11.758330    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:11.871474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:11.871417    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 13:11:11.962244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:11.957918    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:12.014511 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:12.014477    4544 scope.go:122] "RemoveContainer" containerID="e0cdd19d36d34ea35df556a8c71170cf6d9d0640f987d06c3ffb5c1b2f33f4dd"
Oct 30 13:11:12.043247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:12.035984    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:12.129202 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:12.129166    4544 scope.go:122] "RemoveContainer" containerID="a048be130c1f683a1ecf6ca91dbc1ba01d265d2d1b92216ac720836acbd99d6d"
Oct 30 13:11:12.262255 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:12.261755    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:11:13.272974 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:13.272942    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:13.441452 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:13.441413    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:13.449495 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:13.449456    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn"]
Oct 30 13:11:13.457212 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:13.457179    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn"
Oct 30 13:11:13.474300 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:13.474266    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-5b9e8e-dockercfg-mb7sh\""
Oct 30 13:11:13.479607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:13.479573    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn"]
Oct 30 13:11:13.485411 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:13.485379    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-5b9e8e\""
Oct 30 13:11:13.511629 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:13.511591    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc"]
Oct 30 13:11:13.618286 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:13.603744    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc"
Oct 30 13:11:13.909327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:13.718276    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ggp4\" (UniqueName: \"kubernetes.io/projected/caa19642-0691-40fe-aba4-33199f334f45-kube-api-access-7ggp4\") pod \"payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn\" (UID: \"caa19642-0691-40fe-aba4-33199f334f45\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn"
Oct 30 13:11:13.909327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:13.718342    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caa19642-0691-40fe-aba4-33199f334f45-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn\" (UID: \"caa19642-0691-40fe-aba4-33199f334f45\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn"
Oct 30 13:11:13.909327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:13.825132    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s62c2\" (UniqueName: \"kubernetes.io/projected/f57ae3c0-2e03-4252-bc0a-608b3507e96a-kube-api-access-s62c2\") pod \"payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc\" (UID: \"f57ae3c0-2e03-4252-bc0a-608b3507e96a\") " pod="openshift-ingress/payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc"
Oct 30 13:11:13.909327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:13.825196    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ggp4\" (UniqueName: \"kubernetes.io/projected/caa19642-0691-40fe-aba4-33199f334f45-kube-api-access-7ggp4\") pod \"payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn\" (UID: \"caa19642-0691-40fe-aba4-33199f334f45\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn"
Oct 30 13:11:13.909327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:13.825250    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caa19642-0691-40fe-aba4-33199f334f45-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn\" (UID: \"caa19642-0691-40fe-aba4-33199f334f45\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn"
Oct 30 13:11:13.909327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:13.825577    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57ae3c0-2e03-4252-bc0a-608b3507e96a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc\" (UID: \"f57ae3c0-2e03-4252-bc0a-608b3507e96a\") " pod="openshift-ingress/payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc"
Oct 30 13:11:13.909327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:13.867697    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caa19642-0691-40fe-aba4-33199f334f45-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn\" (UID: \"caa19642-0691-40fe-aba4-33199f334f45\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn"
Oct 30 13:11:13.935247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:13.932798    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57ae3c0-2e03-4252-bc0a-608b3507e96a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc\" (UID: \"f57ae3c0-2e03-4252-bc0a-608b3507e96a\") " pod="openshift-ingress/payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc"
Oct 30 13:11:13.935247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:13.932958    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s62c2\" (UniqueName: \"kubernetes.io/projected/f57ae3c0-2e03-4252-bc0a-608b3507e96a-kube-api-access-s62c2\") pod \"payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc\" (UID: \"f57ae3c0-2e03-4252-bc0a-608b3507e96a\") " pod="openshift-ingress/payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc"
Oct 30 13:11:13.964249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:13.955676    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57ae3c0-2e03-4252-bc0a-608b3507e96a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc\" (UID: \"f57ae3c0-2e03-4252-bc0a-608b3507e96a\") " pod="openshift-ingress/payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc"
Oct 30 13:11:13.982997 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:13.982958    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s62c2\" (UniqueName: \"kubernetes.io/projected/f57ae3c0-2e03-4252-bc0a-608b3507e96a-kube-api-access-s62c2\") pod \"payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc\" (UID: \"f57ae3c0-2e03-4252-bc0a-608b3507e96a\") " pod="openshift-ingress/payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc"
Oct 30 13:11:13.988926 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:13.988573    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc"]
Oct 30 13:11:13.988926 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:13.988849    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ggp4\" (UniqueName: \"kubernetes.io/projected/caa19642-0691-40fe-aba4-33199f334f45-kube-api-access-7ggp4\") pod \"payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn\" (UID: \"caa19642-0691-40fe-aba4-33199f334f45\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn"
Oct 30 13:11:14.073104 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:14.073072    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn"
Oct 30 13:11:14.189699 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:14.189371    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc"
Oct 30 13:11:14.343444 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:14.343385    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl" podUID="eea3f02b-aece-4e51-b7c2-5eb3c54d1dca" containerName="payload-processing" containerID="cri-o://839b91602c0adb3d5135f332ad05c565516c0b5a9da817b7d1f6725e8ddd6c4b" gracePeriod=25
Oct 30 13:11:14.380437 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:14.380401    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:14.492999 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:14.492909    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn"]
Oct 30 13:11:14.580313 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:14.580281    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc"]
Oct 30 13:11:14.584326 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:11:14.584119    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf57ae3c0_2e03_4252_bc0a_608b3507e96a.slice/crio-043ac1a64db35268970dac9f25ca926c11dab30a5704c368da44299ffff58574 WatchSource:0}: Error finding container 043ac1a64db35268970dac9f25ca926c11dab30a5704c368da44299ffff58574: Status 404 returned error can't find the container with id 043ac1a64db35268970dac9f25ca926c11dab30a5704c368da44299ffff58574
Oct 30 13:11:14.801812 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:14.801772    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl"
Oct 30 13:11:14.963596 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:14.963558    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eea3f02b-aece-4e51-b7c2-5eb3c54d1dca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea3f02b-aece-4e51-b7c2-5eb3c54d1dca-plugins-config-volume\") pod \"eea3f02b-aece-4e51-b7c2-5eb3c54d1dca\" (UID: \"eea3f02b-aece-4e51-b7c2-5eb3c54d1dca\") "
Oct 30 13:11:14.963803 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:14.963749    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eea3f02b-aece-4e51-b7c2-5eb3c54d1dca-kube-api-access-bc29k\" (UniqueName: \"kubernetes.io/projected/eea3f02b-aece-4e51-b7c2-5eb3c54d1dca-kube-api-access-bc29k\") pod \"eea3f02b-aece-4e51-b7c2-5eb3c54d1dca\" (UID: \"eea3f02b-aece-4e51-b7c2-5eb3c54d1dca\") "
Oct 30 13:11:14.964000 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:14.963973    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eea3f02b-aece-4e51-b7c2-5eb3c54d1dca-plugins-config-volume" pod "eea3f02b-aece-4e51-b7c2-5eb3c54d1dca" (UID: "eea3f02b-aece-4e51-b7c2-5eb3c54d1dca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:11:14.964399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:14.964380    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea3f02b-aece-4e51-b7c2-5eb3c54d1dca-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:11:14.966042 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:14.966019    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea3f02b-aece-4e51-b7c2-5eb3c54d1dca-kube-api-access-bc29k" pod "eea3f02b-aece-4e51-b7c2-5eb3c54d1dca" (UID: "eea3f02b-aece-4e51-b7c2-5eb3c54d1dca"). InnerVolumeSpecName "kube-api-access-bc29k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:15.065938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:15.065852    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc29k\" (UniqueName: \"kubernetes.io/projected/eea3f02b-aece-4e51-b7c2-5eb3c54d1dca-kube-api-access-bc29k\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:11:15.173697 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:15.173660    4544 generic.go:356] "Generic (PLEG): container finished" podID="eea3f02b-aece-4e51-b7c2-5eb3c54d1dca" containerID="839b91602c0adb3d5135f332ad05c565516c0b5a9da817b7d1f6725e8ddd6c4b" exitCode=0
Oct 30 13:11:15.173856 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:15.173714    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl"
Oct 30 13:11:15.173856 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:15.173743    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl" event={"ID":"eea3f02b-aece-4e51-b7c2-5eb3c54d1dca","Type":"ContainerDied","Data":"839b91602c0adb3d5135f332ad05c565516c0b5a9da817b7d1f6725e8ddd6c4b"}
Oct 30 13:11:15.173856 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:15.173832    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl" event={"ID":"eea3f02b-aece-4e51-b7c2-5eb3c54d1dca","Type":"ContainerDied","Data":"05e2db94f36473f52ee03be650ad3b2da431392943ba90844ca3844adc2efd6e"}
Oct 30 13:11:15.174011 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:15.173865    4544 scope.go:122] "RemoveContainer" containerID="839b91602c0adb3d5135f332ad05c565516c0b5a9da817b7d1f6725e8ddd6c4b"
Oct 30 13:11:15.178661 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:15.178532    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn" event={"ID":"caa19642-0691-40fe-aba4-33199f334f45","Type":"ContainerStarted","Data":"4e16d9d6137a8423edbfbc61c069c49fe6f2e9ac86730d7aa07e79c34e4ac618"}
Oct 30 13:11:15.178661 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:15.178594    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn"
Oct 30 13:11:15.178661 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:15.178619    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn" event={"ID":"caa19642-0691-40fe-aba4-33199f334f45","Type":"ContainerStarted","Data":"037ca1ddbece7bbeff1a4bd3100a27e0247fa70f1dbc23faa29a0ab52114c40c"}
Oct 30 13:11:15.183305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:15.183276    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc" event={"ID":"f57ae3c0-2e03-4252-bc0a-608b3507e96a","Type":"ContainerStarted","Data":"83dda0c79e593bd8a0ca00d17e15eaedd22ff3ac88e631143618e7c2d1dacfd5"}
Oct 30 13:11:15.183431 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:15.183329    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc" event={"ID":"f57ae3c0-2e03-4252-bc0a-608b3507e96a","Type":"ContainerStarted","Data":"043ac1a64db35268970dac9f25ca926c11dab30a5704c368da44299ffff58574"}
Oct 30 13:11:15.183431 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:15.183353    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc"
Oct 30 13:11:15.190919 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:15.190901    4544 scope.go:122] "RemoveContainer" containerID="839b91602c0adb3d5135f332ad05c565516c0b5a9da817b7d1f6725e8ddd6c4b"
Oct 30 13:11:15.191182 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:11:15.191161    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"839b91602c0adb3d5135f332ad05c565516c0b5a9da817b7d1f6725e8ddd6c4b\": container with ID starting with 839b91602c0adb3d5135f332ad05c565516c0b5a9da817b7d1f6725e8ddd6c4b not found: ID does not exist" containerID="839b91602c0adb3d5135f332ad05c565516c0b5a9da817b7d1f6725e8ddd6c4b"
Oct 30 13:11:15.191272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:15.191192    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"839b91602c0adb3d5135f332ad05c565516c0b5a9da817b7d1f6725e8ddd6c4b"} err="failed to get container status \"839b91602c0adb3d5135f332ad05c565516c0b5a9da817b7d1f6725e8ddd6c4b\": rpc error: code = NotFound desc = could not find container \"839b91602c0adb3d5135f332ad05c565516c0b5a9da817b7d1f6725e8ddd6c4b\": container with ID starting with 839b91602c0adb3d5135f332ad05c565516c0b5a9da817b7d1f6725e8ddd6c4b not found: ID does not exist"
Oct 30 13:11:15.233824 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:15.233739    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn" podStartSLOduration=2.2337172020000002 podStartE2EDuration="2.233717202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:15.231721113 +0000 UTC m=+1570.278855116" watchObservedRunningTime="2026-10-30 13:11:15.233717202 +0000 UTC m=+1570.280851156"
Oct 30 13:11:15.272848 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:15.272788    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc" podStartSLOduration=2.272775374 podStartE2EDuration="2.272775374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:15.270774522 +0000 UTC m=+1570.317908487" watchObservedRunningTime="2026-10-30 13:11:15.272775374 +0000 UTC m=+1570.319909326"
Oct 30 13:11:15.310205 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:15.310160    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl"]
Oct 30 13:11:15.343125 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:15.343086    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl"]
Oct 30 13:11:16.587471 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:16.587413    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:17.090264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:17.090229    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eea3f02b-aece-4e51-b7c2-5eb3c54d1dca" path="/var/lib/kubelet/pods/eea3f02b-aece-4e51-b7c2-5eb3c54d1dca/volumes"
Oct 30 13:11:18.144970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:18.144930    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:18.953992 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:18.953951    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:19.134761 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:19.134723    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:19.258290 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:19.258155    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:19.682512 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:19.682436    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:21.237301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:21.237186    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:21.265449 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:21.265419    4544 generic.go:356] "Generic (PLEG): container finished" podID="4bbc446d-8807-4613-9d86-b87b0733d2cf" containerID="13fd4d84b6bf12c6d6f835bd82bf43d5eb7acd38a82180c57f66e497c0fea67d" exitCode=0
Oct 30 13:11:21.265636 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:21.265502    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr" event={"ID":"4bbc446d-8807-4613-9d86-b87b0733d2cf","Type":"ContainerDied","Data":"13fd4d84b6bf12c6d6f835bd82bf43d5eb7acd38a82180c57f66e497c0fea67d"}
Oct 30 13:11:21.266157 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:21.266138    4544 scope.go:122] "RemoveContainer" containerID="13fd4d84b6bf12c6d6f835bd82bf43d5eb7acd38a82180c57f66e497c0fea67d"
Oct 30 13:11:21.268215 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:21.268180    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:11:21.268346 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:21.268278    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:11:21.268346 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:21.268334    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:11:21.364200 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:21.364163    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:21.495591 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:21.495477    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:22.275590 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:22.275553    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr" event={"ID":"4bbc446d-8807-4613-9d86-b87b0733d2cf","Type":"ContainerStarted","Data":"63c5590fda351734414b3bea73c33d5731093bf2b3f65a1d9873980323188810"}
Oct 30 13:11:23.146247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:23.134946    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:11:23.171977 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:23.151160    4544 patch_prober.go:29] interesting pod/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 30 13:11:23.171977 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:23.151253    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr" podUID="4bbc446d-8807-4613-9d86-b87b0733d2cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 30 13:11:23.347240 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:23.347184    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc"]
Oct 30 13:11:23.347706 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:23.347456    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc" podUID="91d76077-b236-4f06-a6ab-428c729cfe22" containerName="maas-api" containerID="cri-o://1ab85eb6440b4fba06fe87f118720df43de3d46e9db4eff2b5795352e36419e2" gracePeriod=30
Oct 30 13:11:23.442058 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:23.441886    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:24.021506 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:24.020446    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc"
Oct 30 13:11:24.133248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:24.124125    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91d76077-b236-4f06-a6ab-428c729cfe22-metrics-certs\" (UniqueName: \"kubernetes.io/secret/91d76077-b236-4f06-a6ab-428c729cfe22-metrics-certs\") pod \"91d76077-b236-4f06-a6ab-428c729cfe22\" (UID: \"91d76077-b236-4f06-a6ab-428c729cfe22\") "
Oct 30 13:11:24.133248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:24.124211    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91d76077-b236-4f06-a6ab-428c729cfe22-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d76077-b236-4f06-a6ab-428c729cfe22-openshift-service-ca\") pod \"91d76077-b236-4f06-a6ab-428c729cfe22\" (UID: \"91d76077-b236-4f06-a6ab-428c729cfe22\") "
Oct 30 13:11:24.133248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:24.124323    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91d76077-b236-4f06-a6ab-428c729cfe22-kube-api-access-p2m8z\" (UniqueName: \"kubernetes.io/projected/91d76077-b236-4f06-a6ab-428c729cfe22-kube-api-access-p2m8z\") pod \"91d76077-b236-4f06-a6ab-428c729cfe22\" (UID: \"91d76077-b236-4f06-a6ab-428c729cfe22\") "
Oct 30 13:11:24.133248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:24.124528    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91d76077-b236-4f06-a6ab-428c729cfe22-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91d76077-b236-4f06-a6ab-428c729cfe22-maas-api-tls\") pod \"91d76077-b236-4f06-a6ab-428c729cfe22\" (UID: \"91d76077-b236-4f06-a6ab-428c729cfe22\") "
Oct 30 13:11:24.133248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:24.124807    4544 patch_prober.go:29] interesting pod/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 30 13:11:24.133248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:24.124860    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr" podUID="4bbc446d-8807-4613-9d86-b87b0733d2cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 30 13:11:24.133248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:24.125538    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91d76077-b236-4f06-a6ab-428c729cfe22-openshift-service-ca" pod "91d76077-b236-4f06-a6ab-428c729cfe22" (UID: "91d76077-b236-4f06-a6ab-428c729cfe22"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:11:24.177612 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:24.173631    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:11:24.195261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:24.193605    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91d76077-b236-4f06-a6ab-428c729cfe22-kube-api-access-p2m8z" pod "91d76077-b236-4f06-a6ab-428c729cfe22" (UID: "91d76077-b236-4f06-a6ab-428c729cfe22"). InnerVolumeSpecName "kube-api-access-p2m8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:24.246406 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:24.245322    4544 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d76077-b236-4f06-a6ab-428c729cfe22-openshift-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:11:24.246406 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:24.245361    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2m8z\" (UniqueName: \"kubernetes.io/projected/91d76077-b236-4f06-a6ab-428c729cfe22-kube-api-access-p2m8z\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:11:24.247398 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:24.247345    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91d76077-b236-4f06-a6ab-428c729cfe22-maas-api-tls" pod "91d76077-b236-4f06-a6ab-428c729cfe22" (UID: "91d76077-b236-4f06-a6ab-428c729cfe22"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:11:24.264317 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:24.261065    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91d76077-b236-4f06-a6ab-428c729cfe22-metrics-certs" pod "91d76077-b236-4f06-a6ab-428c729cfe22" (UID: "91d76077-b236-4f06-a6ab-428c729cfe22"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:11:24.279096 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:24.265573    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:24.363245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:24.361341    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91d76077-b236-4f06-a6ab-428c729cfe22-maas-api-tls\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:11:24.363245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:24.361384    4544 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91d76077-b236-4f06-a6ab-428c729cfe22-metrics-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:11:24.370683 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:24.370587    4544 generic.go:356] "Generic (PLEG): container finished" podID="91d76077-b236-4f06-a6ab-428c729cfe22" containerID="1ab85eb6440b4fba06fe87f118720df43de3d46e9db4eff2b5795352e36419e2" exitCode=0
Oct 30 13:11:24.370683 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:24.370650    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc"
Oct 30 13:11:24.370683 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:24.370651    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc" event={"ID":"91d76077-b236-4f06-a6ab-428c729cfe22","Type":"ContainerDied","Data":"1ab85eb6440b4fba06fe87f118720df43de3d46e9db4eff2b5795352e36419e2"}
Oct 30 13:11:24.370808 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:24.370726    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc" event={"ID":"91d76077-b236-4f06-a6ab-428c729cfe22","Type":"ContainerDied","Data":"d699124d38d7a96b15295764a76e6a7426008f811f32bf6da756e73eb2f6d6e1"}
Oct 30 13:11:24.370808 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:24.370759    4544 scope.go:122] "RemoveContainer" containerID="1ab85eb6440b4fba06fe87f118720df43de3d46e9db4eff2b5795352e36419e2"
Oct 30 13:11:24.383810 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:24.383785    4544 scope.go:122] "RemoveContainer" containerID="1ab85eb6440b4fba06fe87f118720df43de3d46e9db4eff2b5795352e36419e2"
Oct 30 13:11:24.384158 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:11:24.384135    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ab85eb6440b4fba06fe87f118720df43de3d46e9db4eff2b5795352e36419e2\": container with ID starting with 1ab85eb6440b4fba06fe87f118720df43de3d46e9db4eff2b5795352e36419e2 not found: ID does not exist" containerID="1ab85eb6440b4fba06fe87f118720df43de3d46e9db4eff2b5795352e36419e2"
Oct 30 13:11:24.384213 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:24.384171    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ab85eb6440b4fba06fe87f118720df43de3d46e9db4eff2b5795352e36419e2"} err="failed to get container status \"1ab85eb6440b4fba06fe87f118720df43de3d46e9db4eff2b5795352e36419e2\": rpc error: code = NotFound desc = could not find container \"1ab85eb6440b4fba06fe87f118720df43de3d46e9db4eff2b5795352e36419e2\": container with ID starting with 1ab85eb6440b4fba06fe87f118720df43de3d46e9db4eff2b5795352e36419e2 not found: ID does not exist"
Oct 30 13:11:24.431256 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:24.430620    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc"]
Oct 30 13:11:24.440475 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:24.438362    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-nh7vc"]
Oct 30 13:11:24.538905 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:24.538867    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:24.679034 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:24.678418    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-2rblx"]
Oct 30 13:11:24.683982 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:24.683945    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c0e6c42-2rblx"]
Oct 30 13:11:24.878256 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:24.877518    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:25.102117 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:25.102080    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00130766-10c7-4dab-ab14-99afeb886aa6" path="/var/lib/kubelet/pods/00130766-10c7-4dab-ab14-99afeb886aa6/volumes"
Oct 30 13:11:25.102791 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:25.102773    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91d76077-b236-4f06-a6ab-428c729cfe22" path="/var/lib/kubelet/pods/91d76077-b236-4f06-a6ab-428c729cfe22/volumes"
Oct 30 13:11:25.139362 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:25.139327    4544 patch_prober.go:29] interesting pod/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 30 13:11:25.139637 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:25.139598    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr" podUID="4bbc446d-8807-4613-9d86-b87b0733d2cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 30 13:11:26.133076 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:26.132608    4544 patch_prober.go:29] interesting pod/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 30 13:11:26.133076 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:26.132683    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr" podUID="4bbc446d-8807-4613-9d86-b87b0733d2cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 30 13:11:26.255970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:26.254402    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn"
Oct 30 13:11:26.267248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:26.256329    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc"
Oct 30 13:11:26.776248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:26.761898    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:26.971246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:26.958975    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:27.007263 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:27.005615    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"]
Oct 30 13:11:27.007263 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:27.005874    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz" podUID="1bf8120f-fb85-451c-922f-7e1cafe165df" containerName="istio-proxy" containerID="cri-o://fe312d96c23fe5970bde1e3b8ead4c545fcbd408773972722bc15d7cb23fbd46" gracePeriod=30
Oct 30 13:11:27.211382 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:27.209720    4544 patch_prober.go:29] interesting pod/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 30 13:11:27.211382 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:27.209813    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr" podUID="4bbc446d-8807-4613-9d86-b87b0733d2cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 30 13:11:27.350320 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:27.347150    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:27.454392 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:27.454330    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:27.770499 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:27.770462    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:28.052418 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:28.052290    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:28.120243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:28.118214    4544 patch_prober.go:29] interesting pod/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 30 13:11:28.120243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:28.118302    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr" podUID="4bbc446d-8807-4613-9d86-b87b0733d2cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 30 13:11:29.019812 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:11:29.019775    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:11:29.021203 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:29.021175    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:29.316715 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:29.316634    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:11:29.319891 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:29.319863    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:11:29.350922 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:29.350888    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:29.629973 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:29.629932    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:30.196459 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:30.196424    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:32.164823 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:11:32.164788    4544 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf8120f_fb85_451c_922f_7e1cafe165df.slice/crio-conmon-fe312d96c23fe5970bde1e3b8ead4c545fcbd408773972722bc15d7cb23fbd46.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 13:11:32.332504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.332475    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:11:32.373982 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.373911    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bf8120f-fb85-451c-922f-7e1cafe165df-kube-api-access-8w9xl\" (UniqueName: \"kubernetes.io/projected/1bf8120f-fb85-451c-922f-7e1cafe165df-kube-api-access-8w9xl\") pod \"1bf8120f-fb85-451c-922f-7e1cafe165df\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") "
Oct 30 13:11:32.373982 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.373967    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1bf8120f-fb85-451c-922f-7e1cafe165df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bf8120f-fb85-451c-922f-7e1cafe165df-istio-podinfo\") pod \"1bf8120f-fb85-451c-922f-7e1cafe165df\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") "
Oct 30 13:11:32.374182 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.374044    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-istio-envoy\") pod \"1bf8120f-fb85-451c-922f-7e1cafe165df\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") "
Oct 30 13:11:32.374182 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.374135    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-workload-certs\") pod \"1bf8120f-fb85-451c-922f-7e1cafe165df\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") "
Oct 30 13:11:32.374349 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.374185    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-workload-socket\") pod \"1bf8120f-fb85-451c-922f-7e1cafe165df\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") "
Oct 30 13:11:32.374349 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.374333    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-credential-socket\") pod \"1bf8120f-fb85-451c-922f-7e1cafe165df\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") "
Oct 30 13:11:32.374504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.374455    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-workload-certs" pod "1bf8120f-fb85-451c-922f-7e1cafe165df" (UID: "1bf8120f-fb85-451c-922f-7e1cafe165df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:32.374504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.374488    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-workload-socket" pod "1bf8120f-fb85-451c-922f-7e1cafe165df" (UID: "1bf8120f-fb85-451c-922f-7e1cafe165df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:32.374504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.374492    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-istio-data\") pod \"1bf8120f-fb85-451c-922f-7e1cafe165df\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") "
Oct 30 13:11:32.374712 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.374587    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bf8120f-fb85-451c-922f-7e1cafe165df-istio-token\" (UniqueName: \"kubernetes.io/projected/1bf8120f-fb85-451c-922f-7e1cafe165df-istio-token\") pod \"1bf8120f-fb85-451c-922f-7e1cafe165df\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") "
Oct 30 13:11:32.374712 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.374603    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-credential-socket" pod "1bf8120f-fb85-451c-922f-7e1cafe165df" (UID: "1bf8120f-fb85-451c-922f-7e1cafe165df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:32.374712 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.374690    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-istio-data" pod "1bf8120f-fb85-451c-922f-7e1cafe165df" (UID: "1bf8120f-fb85-451c-922f-7e1cafe165df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:32.374874 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.374728    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bf8120f-fb85-451c-922f-7e1cafe165df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bf8120f-fb85-451c-922f-7e1cafe165df-istiod-ca-cert\") pod \"1bf8120f-fb85-451c-922f-7e1cafe165df\" (UID: \"1bf8120f-fb85-451c-922f-7e1cafe165df\") "
Oct 30 13:11:32.375115 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.375089    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bf8120f-fb85-451c-922f-7e1cafe165df-istiod-ca-cert" pod "1bf8120f-fb85-451c-922f-7e1cafe165df" (UID: "1bf8120f-fb85-451c-922f-7e1cafe165df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:11:32.375581 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.375558    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bf8120f-fb85-451c-922f-7e1cafe165df-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.375693 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.375586    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.375693 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.375599    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.375693 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.375614    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.375693 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.375627    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.376459 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.376439    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1bf8120f-fb85-451c-922f-7e1cafe165df-istio-podinfo" pod "1bf8120f-fb85-451c-922f-7e1cafe165df" (UID: "1bf8120f-fb85-451c-922f-7e1cafe165df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:11:32.376539 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.376499    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-istio-envoy" pod "1bf8120f-fb85-451c-922f-7e1cafe165df" (UID: "1bf8120f-fb85-451c-922f-7e1cafe165df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:32.376710 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.376689    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf8120f-fb85-451c-922f-7e1cafe165df-kube-api-access-8w9xl" pod "1bf8120f-fb85-451c-922f-7e1cafe165df" (UID: "1bf8120f-fb85-451c-922f-7e1cafe165df"). InnerVolumeSpecName "kube-api-access-8w9xl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:32.376926 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.376908    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf8120f-fb85-451c-922f-7e1cafe165df-istio-token" pod "1bf8120f-fb85-451c-922f-7e1cafe165df" (UID: "1bf8120f-fb85-451c-922f-7e1cafe165df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:32.477641 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.477601    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bf8120f-fb85-451c-922f-7e1cafe165df-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.477641 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.477642    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w9xl\" (UniqueName: \"kubernetes.io/projected/1bf8120f-fb85-451c-922f-7e1cafe165df-kube-api-access-8w9xl\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.477883 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.477657    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bf8120f-fb85-451c-922f-7e1cafe165df-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.477883 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.477672    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bf8120f-fb85-451c-922f-7e1cafe165df-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.518506 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.518475    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:32.663244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.659196    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:32.927615 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.927518    4544 generic.go:356] "Generic (PLEG): container finished" podID="1bf8120f-fb85-451c-922f-7e1cafe165df" containerID="fe312d96c23fe5970bde1e3b8ead4c545fcbd408773972722bc15d7cb23fbd46" exitCode=0
Oct 30 13:11:32.927615 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.927606    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz" event={"ID":"1bf8120f-fb85-451c-922f-7e1cafe165df","Type":"ContainerDied","Data":"fe312d96c23fe5970bde1e3b8ead4c545fcbd408773972722bc15d7cb23fbd46"}
Oct 30 13:11:32.927868 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.927693    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz" event={"ID":"1bf8120f-fb85-451c-922f-7e1cafe165df","Type":"ContainerDied","Data":"ad8d41b9ea4fe1e61beac2eb88fdaf2830215ce3aa5e02970b15fd0572e12ab1"}
Oct 30 13:11:32.927868 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.927730    4544 scope.go:122] "RemoveContainer" containerID="fe312d96c23fe5970bde1e3b8ead4c545fcbd408773972722bc15d7cb23fbd46"
Oct 30 13:11:32.927868 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.927774    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"
Oct 30 13:11:32.953713 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.953687    4544 scope.go:122] "RemoveContainer" containerID="fe312d96c23fe5970bde1e3b8ead4c545fcbd408773972722bc15d7cb23fbd46"
Oct 30 13:11:32.954082 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:11:32.954055    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe312d96c23fe5970bde1e3b8ead4c545fcbd408773972722bc15d7cb23fbd46\": container with ID starting with fe312d96c23fe5970bde1e3b8ead4c545fcbd408773972722bc15d7cb23fbd46 not found: ID does not exist" containerID="fe312d96c23fe5970bde1e3b8ead4c545fcbd408773972722bc15d7cb23fbd46"
Oct 30 13:11:32.954156 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.954096    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe312d96c23fe5970bde1e3b8ead4c545fcbd408773972722bc15d7cb23fbd46"} err="failed to get container status \"fe312d96c23fe5970bde1e3b8ead4c545fcbd408773972722bc15d7cb23fbd46\": rpc error: code = NotFound desc = could not find container \"fe312d96c23fe5970bde1e3b8ead4c545fcbd408773972722bc15d7cb23fbd46\": container with ID starting with fe312d96c23fe5970bde1e3b8ead4c545fcbd408773972722bc15d7cb23fbd46 not found: ID does not exist"
Oct 30 13:11:32.988970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:32.988942    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"]
Oct 30 13:11:33.000263 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:33.000237    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz"]
Oct 30 13:11:33.100399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:33.100366    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bf8120f-fb85-451c-922f-7e1cafe165df" path="/var/lib/kubelet/pods/1bf8120f-fb85-451c-922f-7e1cafe165df/volumes"
Oct 30 13:11:33.338837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:33.338803    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:33.640818 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:33.640727    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:34.067979 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:34.067868    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:34.273506 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:34.273470    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:35.000476 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:35.000441    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:36.115566 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:36.115534    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:36.233062 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:36.233020    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:36.968690 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:36.968651    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:37.118243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:37.099492    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:37.687909 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:37.687879    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:38.447165 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:38.447123    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:38.899828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:38.899795    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:39.264143 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:39.264049    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:39.580017 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:39.579984    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:41.451598 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:41.451568    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:43.284666 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:43.284629    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:43.498546 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:43.498496    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:43.641315 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:43.641269    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:44.992518 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:44.992478    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:45.113411 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:45.113383    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:45.140883 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:45.140853    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh"]
Oct 30 13:11:45.141669 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:45.141654    4544 cpu_manager.go:414] "containerMap: removing container" podUID="eea3f02b-aece-4e51-b7c2-5eb3c54d1dca" containerName="payload-processing"
Oct 30 13:11:45.141731 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:45.141669    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eea3f02b-aece-4e51-b7c2-5eb3c54d1dca" containerName="payload-processing"
Oct 30 13:11:45.141731 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:45.141685    4544 cpu_manager.go:414] "containerMap: removing container" podUID="1bf8120f-fb85-451c-922f-7e1cafe165df" containerName="istio-proxy"
Oct 30 13:11:45.141731 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:45.141690    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bf8120f-fb85-451c-922f-7e1cafe165df" containerName="istio-proxy"
Oct 30 13:11:45.141731 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:45.141711    4544 cpu_manager.go:414] "containerMap: removing container" podUID="91d76077-b236-4f06-a6ab-428c729cfe22" containerName="maas-api"
Oct 30 13:11:45.141731 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:45.141716    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91d76077-b236-4f06-a6ab-428c729cfe22" containerName="maas-api"
Oct 30 13:11:45.141926 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:45.141825    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="eea3f02b-aece-4e51-b7c2-5eb3c54d1dca" containerName="payload-processing"
Oct 30 13:11:45.141926 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:45.141838    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bf8120f-fb85-451c-922f-7e1cafe165df" containerName="istio-proxy"
Oct 30 13:11:45.141926 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:45.141848    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="91d76077-b236-4f06-a6ab-428c729cfe22" containerName="maas-api"
Oct 30 13:11:45.147411 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:45.147391    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh"
Oct 30 13:11:45.171397 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:45.171374    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh"]
Oct 30 13:11:45.177690 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:45.177667    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-716b4c\""
Oct 30 13:11:45.177690 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:45.177676    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-716b4c-dockercfg-zw46q\""
Oct 30 13:11:45.460297 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:45.416994    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gdwp\" (UniqueName: \"kubernetes.io/projected/b4c5666f-6f23-4001-a5d4-c305bb5cd950-kube-api-access-2gdwp\") pod \"payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh\" (UID: \"b4c5666f-6f23-4001-a5d4-c305bb5cd950\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh"
Oct 30 13:11:45.551753 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:45.551716    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4c5666f-6f23-4001-a5d4-c305bb5cd950-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh\" (UID: \"b4c5666f-6f23-4001-a5d4-c305bb5cd950\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh"
Oct 30 13:11:45.653071 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:45.653035    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gdwp\" (UniqueName: \"kubernetes.io/projected/b4c5666f-6f23-4001-a5d4-c305bb5cd950-kube-api-access-2gdwp\") pod \"payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh\" (UID: \"b4c5666f-6f23-4001-a5d4-c305bb5cd950\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh"
Oct 30 13:11:45.653267 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:45.653178    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4c5666f-6f23-4001-a5d4-c305bb5cd950-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh\" (UID: \"b4c5666f-6f23-4001-a5d4-c305bb5cd950\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh"
Oct 30 13:11:45.653917 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:45.653896    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4c5666f-6f23-4001-a5d4-c305bb5cd950-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh\" (UID: \"b4c5666f-6f23-4001-a5d4-c305bb5cd950\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh"
Oct 30 13:11:45.785564 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:45.785483    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gdwp\" (UniqueName: \"kubernetes.io/projected/b4c5666f-6f23-4001-a5d4-c305bb5cd950-kube-api-access-2gdwp\") pod \"payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh\" (UID: \"b4c5666f-6f23-4001-a5d4-c305bb5cd950\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh"
Oct 30 13:11:45.785944 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:45.785922    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh"
Oct 30 13:11:46.259062 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:46.259027    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh"]
Oct 30 13:11:46.261039 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:11:46.261011    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c5666f_6f23_4001_a5d4_c305bb5cd950.slice/crio-f0b935340c80d700a7af211a71af7e36a9972a3a498dcb7b403b31fe421d76a5 WatchSource:0}: Error finding container f0b935340c80d700a7af211a71af7e36a9972a3a498dcb7b403b31fe421d76a5: Status 404 returned error can't find the container with id f0b935340c80d700a7af211a71af7e36a9972a3a498dcb7b403b31fe421d76a5
Oct 30 13:11:46.411539 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:46.411504    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:47.264628 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:47.264583    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh" event={"ID":"b4c5666f-6f23-4001-a5d4-c305bb5cd950","Type":"ContainerStarted","Data":"a667e13629e69fe46bbdbe3cbeec458b169f69e48118cbb3fb5da833fd017c09"}
Oct 30 13:11:47.265266 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:47.265208    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh" event={"ID":"b4c5666f-6f23-4001-a5d4-c305bb5cd950","Type":"ContainerStarted","Data":"f0b935340c80d700a7af211a71af7e36a9972a3a498dcb7b403b31fe421d76a5"}
Oct 30 13:11:47.266822 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:47.266798    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh"
Oct 30 13:11:48.693461 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:48.693429    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:50.695738 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:50.695703    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:51.730187 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:51.730150    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:52.538004 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:52.537939    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:54.477637 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:54.477599    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:54.663587 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:54.663553    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:56.066276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:56.066202    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh" podStartSLOduration=11.066185686 podStartE2EDuration="11.066185686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:47.324764528 +0000 UTC m=+1602.371898491" watchObservedRunningTime="2026-10-30 13:11:56.066185686 +0000 UTC m=+1611.113319639"
Oct 30 13:11:56.068546 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:56.068520    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7"]
Oct 30 13:11:56.074643 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:56.074618    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7"
Oct 30 13:11:56.102670 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:56.102604    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-5b9e8e-serving-cert\""
Oct 30 13:11:56.108348 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:56.108312    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7"]
Oct 30 13:11:56.164137 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:56.164096    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:56.189067 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:56.189023    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/faeaf797-0836-49db-a548-6b7dc6faa766-openshift-service-ca\") pod \"maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7\" (UID: \"faeaf797-0836-49db-a548-6b7dc6faa766\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7"
Oct 30 13:11:56.189250 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:56.189212    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zktcv\" (UniqueName: \"kubernetes.io/projected/faeaf797-0836-49db-a548-6b7dc6faa766-kube-api-access-zktcv\") pod \"maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7\" (UID: \"faeaf797-0836-49db-a548-6b7dc6faa766\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7"
Oct 30 13:11:56.189318 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:56.189300    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/faeaf797-0836-49db-a548-6b7dc6faa766-metrics-certs\") pod \"maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7\" (UID: \"faeaf797-0836-49db-a548-6b7dc6faa766\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7"
Oct 30 13:11:56.189452 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:56.189435    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/faeaf797-0836-49db-a548-6b7dc6faa766-maas-api-tls\") pod \"maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7\" (UID: \"faeaf797-0836-49db-a548-6b7dc6faa766\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7"
Oct 30 13:11:56.294245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:56.292475    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/faeaf797-0836-49db-a548-6b7dc6faa766-maas-api-tls\") pod \"maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7\" (UID: \"faeaf797-0836-49db-a548-6b7dc6faa766\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7"
Oct 30 13:11:56.294245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:56.292666    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/faeaf797-0836-49db-a548-6b7dc6faa766-openshift-service-ca\") pod \"maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7\" (UID: \"faeaf797-0836-49db-a548-6b7dc6faa766\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7"
Oct 30 13:11:56.294245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:56.292827    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zktcv\" (UniqueName: \"kubernetes.io/projected/faeaf797-0836-49db-a548-6b7dc6faa766-kube-api-access-zktcv\") pod \"maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7\" (UID: \"faeaf797-0836-49db-a548-6b7dc6faa766\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7"
Oct 30 13:11:56.294245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:56.292883    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/faeaf797-0836-49db-a548-6b7dc6faa766-metrics-certs\") pod \"maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7\" (UID: \"faeaf797-0836-49db-a548-6b7dc6faa766\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7"
Oct 30 13:11:56.320301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:56.303188    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/faeaf797-0836-49db-a548-6b7dc6faa766-openshift-service-ca\") pod \"maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7\" (UID: \"faeaf797-0836-49db-a548-6b7dc6faa766\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7"
Oct 30 13:11:56.326739 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:56.326702    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:56.335990 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:56.335906    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/faeaf797-0836-49db-a548-6b7dc6faa766-maas-api-tls\") pod \"maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7\" (UID: \"faeaf797-0836-49db-a548-6b7dc6faa766\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7"
Oct 30 13:11:56.353294 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:56.351891    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/faeaf797-0836-49db-a548-6b7dc6faa766-metrics-certs\") pod \"maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7\" (UID: \"faeaf797-0836-49db-a548-6b7dc6faa766\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7"
Oct 30 13:11:56.370290 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:56.365073    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zktcv\" (UniqueName: \"kubernetes.io/projected/faeaf797-0836-49db-a548-6b7dc6faa766-kube-api-access-zktcv\") pod \"maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7\" (UID: \"faeaf797-0836-49db-a548-6b7dc6faa766\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7"
Oct 30 13:11:56.406609 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:56.406546    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7"
Oct 30 13:11:56.755877 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:56.755845    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7"]
Oct 30 13:11:56.757619 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:11:56.757587    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaeaf797_0836_49db_a548_6b7dc6faa766.slice/crio-eae57e37223c3ef5f1633d1eda3d6f523c0378ff99ad9606671fbc6c176ccdbe WatchSource:0}: Error finding container eae57e37223c3ef5f1633d1eda3d6f523c0378ff99ad9606671fbc6c176ccdbe: Status 404 returned error can't find the container with id eae57e37223c3ef5f1633d1eda3d6f523c0378ff99ad9606671fbc6c176ccdbe
Oct 30 13:11:57.379102 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:57.379058    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7" event={"ID":"faeaf797-0836-49db-a548-6b7dc6faa766","Type":"ContainerStarted","Data":"2daff1df0dfa287b042d265ae5830aeb645fb67c961e79964519711fdab9e532"}
Oct 30 13:11:57.381499 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:57.379129    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7" event={"ID":"faeaf797-0836-49db-a548-6b7dc6faa766","Type":"ContainerStarted","Data":"eae57e37223c3ef5f1633d1eda3d6f523c0378ff99ad9606671fbc6c176ccdbe"}
Oct 30 13:11:57.381499 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:57.379160    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7"
Oct 30 13:11:57.423540 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:57.423407    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7" podStartSLOduration=1.022442438 podStartE2EDuration="1.423388768s" totalImagesPullingTime="400.94633ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:57.420605124 +0000 UTC m=+1612.467739077" watchObservedRunningTime="2026-10-30 13:11:57.423388768 +0000 UTC m=+1612.470522721"
Oct 30 13:11:57.604358 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:57.602067    4544 patch_prober.go:29] interesting pod/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": read tcp 10.133.0.2:56186->10.133.0.169:15021: read: connection reset by peer" start-of-body=
Oct 30 13:11:57.604358 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:57.602126    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz" podUID="613b391b-f5e4-4c93-84ba-70d00d691d27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": read tcp 10.133.0.2:56186->10.133.0.169:15021: read: connection reset by peer"
Oct 30 13:11:58.158912 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:58.158882    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:11:58.388911 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:58.388881    4544 generic.go:356] "Generic (PLEG): container finished" podID="613b391b-f5e4-4c93-84ba-70d00d691d27" containerID="fdd8a7f89d9f07b8ba011c6087fbbadce69c965e14cb3f316853059a30a8b02a" exitCode=0
Oct 30 13:11:58.389409 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:58.388955    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz" event={"ID":"613b391b-f5e4-4c93-84ba-70d00d691d27","Type":"ContainerDied","Data":"fdd8a7f89d9f07b8ba011c6087fbbadce69c965e14cb3f316853059a30a8b02a"}
Oct 30 13:11:58.389643 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:58.389626    4544 scope.go:122] "RemoveContainer" containerID="fdd8a7f89d9f07b8ba011c6087fbbadce69c965e14cb3f316853059a30a8b02a"
Oct 30 13:11:58.391376 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:58.391345    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:11:58.391467 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:58.391412    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:11:58.391467 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:58.391445    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:11:59.009393 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:11:59.009353    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:11:59.283605 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:59.283523    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh"
Oct 30 13:11:59.400541 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:59.400504    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz" event={"ID":"613b391b-f5e4-4c93-84ba-70d00d691d27","Type":"ContainerStarted","Data":"dc2ffadd7fb978b8452e516b301d48013661ce31747e3d9fcd1006d39d7bdd1f"}
Oct 30 13:11:59.509875 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:59.509830    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:11:59.510094 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:59.510070    4544 patch_prober.go:29] interesting pod/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 13:11:59.510143 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:11:59.510119    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz" podUID="613b391b-f5e4-4c93-84ba-70d00d691d27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 13:12:00.275935 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:00.275892    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:00.509690 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:00.509654    4544 patch_prober.go:29] interesting pod/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 13:12:00.510099 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:00.509726    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz" podUID="613b391b-f5e4-4c93-84ba-70d00d691d27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 13:12:01.509675 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:01.509637    4544 patch_prober.go:29] interesting pod/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 13:12:01.509885 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:01.509705    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz" podUID="613b391b-f5e4-4c93-84ba-70d00d691d27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 13:12:02.087477 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:02.087418    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:02.509659 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:02.509568    4544 patch_prober.go:29] interesting pod/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 13:12:02.509827 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:02.509652    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz" podUID="613b391b-f5e4-4c93-84ba-70d00d691d27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 13:12:03.395742 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:03.395708    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7"
Oct 30 13:12:03.509146 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:03.509108    4544 patch_prober.go:29] interesting pod/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 13:12:03.509373 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:03.509235    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz" podUID="613b391b-f5e4-4c93-84ba-70d00d691d27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 13:12:03.973018 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:03.972981    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:04.509647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:04.509603    4544 patch_prober.go:29] interesting pod/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 13:12:04.510088 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:04.509665    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz" podUID="613b391b-f5e4-4c93-84ba-70d00d691d27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 13:12:05.513715 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:05.513679    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:12:05.514268 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:05.514248    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:12:05.515177 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:05.515155    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:12:05.886970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:05.886937    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:08.206034 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:08.205998    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:08.487440 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:08.487316    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:08.748902 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:08.748808    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:10.390389 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:10.390350    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:10.830176 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:10.830139    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:13.139133 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:13.139077    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:15.185734 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:15.185699    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:15.307776 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:15.307739    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:17.353182 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:17.353124    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:17.478183 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:17.478115    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:18.642414 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:18.642329    4544 generic.go:356] "Generic (PLEG): container finished" podID="02d79e94-7094-44da-a04a-824f9328125e" containerID="f8dcec6e0c46f9f66b78c1fd2a67d5a55c7b87e3a7423e251b6da5ef84d7fdec" exitCode=0
Oct 30 13:12:18.642414 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:18.642403    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj" event={"ID":"02d79e94-7094-44da-a04a-824f9328125e","Type":"ContainerDied","Data":"f8dcec6e0c46f9f66b78c1fd2a67d5a55c7b87e3a7423e251b6da5ef84d7fdec"}
Oct 30 13:12:18.642936 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:18.642919    4544 scope.go:122] "RemoveContainer" containerID="f8dcec6e0c46f9f66b78c1fd2a67d5a55c7b87e3a7423e251b6da5ef84d7fdec"
Oct 30 13:12:18.644794 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:18.644765    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:12:18.644863 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:18.644838    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:12:18.644908 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:18.644872    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:12:19.436070 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:19.436031    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:19.653517 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:19.653477    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj" event={"ID":"02d79e94-7094-44da-a04a-824f9328125e","Type":"ContainerStarted","Data":"f184a1e7b07b784c4ba5b07513f457f06e9cee0f7fe058a5cf7c524173db7cac"}
Oct 30 13:12:20.176407 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:20.176371    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:12:20.177634 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:20.177599    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 13:12:20.177797 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:20.177658    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj" podUID="02d79e94-7094-44da-a04a-824f9328125e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 13:12:21.176555 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:21.176517    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 13:12:21.176995 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:21.176574    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj" podUID="02d79e94-7094-44da-a04a-824f9328125e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 13:12:21.553349 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:21.553265    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:21.960558 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:21.960516    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:12:22.175833 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:22.175795    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 13:12:22.176008 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:22.175860    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj" podUID="02d79e94-7094-44da-a04a-824f9328125e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 13:12:22.388404 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:22.388367    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:22.642318 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:22.642202    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-75abce-bd9687d56-scc2b"]
Oct 30 13:12:22.783478 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:22.783411    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:22.896381 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:22.896210    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:23.176421 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:23.176313    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 13:12:23.176421 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:23.176393    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj" podUID="02d79e94-7094-44da-a04a-824f9328125e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 13:12:23.579238 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:23.579191    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:24.176868 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:24.176831    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 13:12:24.177075 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:24.176891    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj" podUID="02d79e94-7094-44da-a04a-824f9328125e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 13:12:25.175789 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:25.175749    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 13:12:25.176203 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:25.175834    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj" podUID="02d79e94-7094-44da-a04a-824f9328125e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 13:12:25.543010 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:25.542921    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:26.235333 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:26.235300    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:12:26.238036 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:26.237975    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:12:27.645338 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:27.644286    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-75abce-bd9687d56-scc2b" podUID="e3314158-027a-4c7d-8592-288a8ac98655" containerName="payload-processing" containerID="cri-o://85976aaaf6dcf15bcab73a5c6604ab77301b731bdfd9f29935b50da2f614c8db" gracePeriod=25
Oct 30 13:12:27.854604 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:27.854256    4544 generic.go:356] "Generic (PLEG): container finished" podID="e3314158-027a-4c7d-8592-288a8ac98655" containerID="85976aaaf6dcf15bcab73a5c6604ab77301b731bdfd9f29935b50da2f614c8db" exitCode=0
Oct 30 13:12:27.854604 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:27.854471    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-75abce-bd9687d56-scc2b" event={"ID":"e3314158-027a-4c7d-8592-288a8ac98655","Type":"ContainerDied","Data":"85976aaaf6dcf15bcab73a5c6604ab77301b731bdfd9f29935b50da2f614c8db"}
Oct 30 13:12:27.995738 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:27.995705    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-75abce-bd9687d56-scc2b"
Oct 30 13:12:28.006473 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:28.005593    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3314158-027a-4c7d-8592-288a8ac98655-kube-api-access-mfbf4\" (UniqueName: \"kubernetes.io/projected/e3314158-027a-4c7d-8592-288a8ac98655-kube-api-access-mfbf4\") pod \"e3314158-027a-4c7d-8592-288a8ac98655\" (UID: \"e3314158-027a-4c7d-8592-288a8ac98655\") "
Oct 30 13:12:28.006473 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:28.005751    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3314158-027a-4c7d-8592-288a8ac98655-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3314158-027a-4c7d-8592-288a8ac98655-plugins-config-volume\") pod \"e3314158-027a-4c7d-8592-288a8ac98655\" (UID: \"e3314158-027a-4c7d-8592-288a8ac98655\") "
Oct 30 13:12:28.006683 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:28.006487    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3314158-027a-4c7d-8592-288a8ac98655-plugins-config-volume" pod "e3314158-027a-4c7d-8592-288a8ac98655" (UID: "e3314158-027a-4c7d-8592-288a8ac98655"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:12:28.008191 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:28.008161    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3314158-027a-4c7d-8592-288a8ac98655-kube-api-access-mfbf4" pod "e3314158-027a-4c7d-8592-288a8ac98655" (UID: "e3314158-027a-4c7d-8592-288a8ac98655"). InnerVolumeSpecName "kube-api-access-mfbf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:28.107805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:28.107763    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3314158-027a-4c7d-8592-288a8ac98655-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:12:28.107805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:28.107797    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfbf4\" (UniqueName: \"kubernetes.io/projected/e3314158-027a-4c7d-8592-288a8ac98655-kube-api-access-mfbf4\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:12:28.195842 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:28.195804    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:28.866514 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:28.866475    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-75abce-bd9687d56-scc2b"
Oct 30 13:12:28.867084 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:28.866471    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-75abce-bd9687d56-scc2b" event={"ID":"e3314158-027a-4c7d-8592-288a8ac98655","Type":"ContainerDied","Data":"550ff3ac4ce062234f76c3dd9294508eeedb0c2acdb6482a64a0e953aae8b067"}
Oct 30 13:12:28.867084 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:28.866661    4544 scope.go:122] "RemoveContainer" containerID="85976aaaf6dcf15bcab73a5c6604ab77301b731bdfd9f29935b50da2f614c8db"
Oct 30 13:12:28.944241 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:28.943840    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-75abce-bd9687d56-scc2b"]
Oct 30 13:12:28.949295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:28.949259    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-75abce-bd9687d56-scc2b"]
Oct 30 13:12:29.010065 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:12:29.010027    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:12:29.092080 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:29.092040    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3314158-027a-4c7d-8592-288a8ac98655" path="/var/lib/kubelet/pods/e3314158-027a-4c7d-8592-288a8ac98655/volumes"
Oct 30 13:12:30.266596 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:30.266540    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:30.420932 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:30.420887    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:30.938836 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:30.938788    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:32.279070 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:32.279030    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:32.408893 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:32.408857    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:32.754274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:32.748422    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"]
Oct 30 13:12:32.754274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:32.750199    4544 cpu_manager.go:414] "containerMap: removing container" podUID="e3314158-027a-4c7d-8592-288a8ac98655" containerName="payload-processing"
Oct 30 13:12:32.754274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:32.750234    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3314158-027a-4c7d-8592-288a8ac98655" containerName="payload-processing"
Oct 30 13:12:32.754274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:32.750506    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3314158-027a-4c7d-8592-288a8ac98655" containerName="payload-processing"
Oct 30 13:12:32.761513 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:32.759178    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:12:32.775591 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:32.774292    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"]
Oct 30 13:12:32.786709 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:32.786665    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2a3e01-openshift-default-dockercfg-6vxcz\""
Oct 30 13:12:32.857876 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:32.857833    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:32.930248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:32.921093    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/476505c3-3236-4789-846f-c13caa710b36-istio-podinfo\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:12:32.985276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:32.981339    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-652h2\" (UniqueName: \"kubernetes.io/projected/476505c3-3236-4789-846f-c13caa710b36-kube-api-access-652h2\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:12:32.985276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:32.981419    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-istio-envoy\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:12:32.985276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:32.981692    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/476505c3-3236-4789-846f-c13caa710b36-istiod-ca-cert\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:12:32.985276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:32.981796    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-workload-socket\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:12:32.985276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:32.981830    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/476505c3-3236-4789-846f-c13caa710b36-istio-token\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:12:32.985276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:32.981858    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-istio-data\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:12:32.985276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:32.981899    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-workload-certs\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:12:32.985276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:32.981957    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-credential-socket\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:12:33.023311 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:33.023204    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:33.101032 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:33.100997    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/476505c3-3236-4789-846f-c13caa710b36-istio-podinfo\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:12:33.101363 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:33.101327    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-652h2\" (UniqueName: \"kubernetes.io/projected/476505c3-3236-4789-846f-c13caa710b36-kube-api-access-652h2\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:12:33.101513 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:33.101499    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-istio-envoy\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:12:33.101933 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:33.101916    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/476505c3-3236-4789-846f-c13caa710b36-istiod-ca-cert\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:12:33.102146 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:33.102133    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-workload-socket\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:12:33.102286 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:33.102273    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/476505c3-3236-4789-846f-c13caa710b36-istio-token\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:12:33.102396 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:33.102384    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-istio-data\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:12:33.102509 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:33.102497    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-workload-certs\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:12:33.102655 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:33.102641    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-credential-socket\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:12:33.103333 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:33.103310    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-credential-socket\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:12:33.104643 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:33.104621    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-workload-socket\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:12:33.110317 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:33.104962    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/476505c3-3236-4789-846f-c13caa710b36-istiod-ca-cert\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:12:33.110524 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:33.105602    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-istio-data\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:12:33.110618 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:33.106620    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-workload-certs\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:12:33.146408 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:33.130940    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/476505c3-3236-4789-846f-c13caa710b36-istio-podinfo\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:12:33.172938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:33.164486    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-652h2\" (UniqueName: \"kubernetes.io/projected/476505c3-3236-4789-846f-c13caa710b36-kube-api-access-652h2\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:12:33.189257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:33.185705    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-istio-envoy\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:12:33.196843 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:33.196811    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/476505c3-3236-4789-846f-c13caa710b36-istio-token\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:12:33.388588 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:33.388548    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:12:33.739728 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:33.739696    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"]
Oct 30 13:12:33.741120 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:12:33.741096    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod476505c3_3236_4789_846f_c13caa710b36.slice/crio-ab89f5db184a4fb433af771414977ce9123c8eabfc7800f8e8cad13659eaf3d9 WatchSource:0}: Error finding container ab89f5db184a4fb433af771414977ce9123c8eabfc7800f8e8cad13659eaf3d9: Status 404 returned error can't find the container with id ab89f5db184a4fb433af771414977ce9123c8eabfc7800f8e8cad13659eaf3d9
Oct 30 13:12:33.744624 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:33.744579    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:12:33.744740 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:33.744674    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:12:33.744740 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:33.744714    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:12:34.025022 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:34.024920    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6" event={"ID":"476505c3-3236-4789-846f-c13caa710b36","Type":"ContainerStarted","Data":"56400f2249f611470566cc3ca65eee52ef00ac4e02853170e613fd42d51eb337"}
Oct 30 13:12:34.025022 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:34.024986    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6" event={"ID":"476505c3-3236-4789-846f-c13caa710b36","Type":"ContainerStarted","Data":"ab89f5db184a4fb433af771414977ce9123c8eabfc7800f8e8cad13659eaf3d9"}
Oct 30 13:12:34.071526 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:34.071470    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6" podStartSLOduration=2.071453665 podStartE2EDuration="2.071453665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:34.069762401 +0000 UTC m=+1649.116896357" watchObservedRunningTime="2026-10-30 13:12:34.071453665 +0000 UTC m=+1649.118587617"
Oct 30 13:12:34.389908 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:34.389860    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:12:34.397646 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:34.397614    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:12:35.068185 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:35.068093    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:35.074486 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:35.074407    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:12:35.076704 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:35.076367    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:12:36.575456 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:36.575417    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:36.719800 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:36.719764    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:37.087020 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:37.086986    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:38.101342 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:38.101310    4544 generic.go:356] "Generic (PLEG): container finished" podID="eaa6149f-3c75-4666-8607-5d2686a65b6d" containerID="59a7cd34b4ea8773e5e0aa9ab6cdf2c6113c07eb072a0a4570382665d7280f58" exitCode=0
Oct 30 13:12:38.101829 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:38.101380    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l" event={"ID":"eaa6149f-3c75-4666-8607-5d2686a65b6d","Type":"ContainerDied","Data":"59a7cd34b4ea8773e5e0aa9ab6cdf2c6113c07eb072a0a4570382665d7280f58"}
Oct 30 13:12:38.101909 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:38.101889    4544 scope.go:122] "RemoveContainer" containerID="59a7cd34b4ea8773e5e0aa9ab6cdf2c6113c07eb072a0a4570382665d7280f58"
Oct 30 13:12:38.103850 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:38.103820    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:12:38.103928 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:38.103893    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:12:38.103965 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:38.103925    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:12:38.343603 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:38.343568    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:38.484549 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:38.484461    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:38.535925 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:38.535881    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq"]
Oct 30 13:12:38.541529 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:38.541496    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq"
Oct 30 13:12:38.556659 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:38.556630    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2a3e01-dockercfg-8t2rr\""
Oct 30 13:12:38.574194 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:38.574153    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq"]
Oct 30 13:12:38.599925 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:38.599883    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2a3e01\""
Oct 30 13:12:38.662246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:38.661473    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds745\" (UniqueName: \"kubernetes.io/projected/2d6dcafc-5203-4f02-b8b8-c06fc9eb9353-kube-api-access-ds745\") pod \"payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq\" (UID: \"2d6dcafc-5203-4f02-b8b8-c06fc9eb9353\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq"
Oct 30 13:12:38.662246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:38.661825    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d6dcafc-5203-4f02-b8b8-c06fc9eb9353-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq\" (UID: \"2d6dcafc-5203-4f02-b8b8-c06fc9eb9353\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq"
Oct 30 13:12:38.777352 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:38.769451    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d6dcafc-5203-4f02-b8b8-c06fc9eb9353-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq\" (UID: \"2d6dcafc-5203-4f02-b8b8-c06fc9eb9353\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq"
Oct 30 13:12:38.777352 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:38.769605    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds745\" (UniqueName: \"kubernetes.io/projected/2d6dcafc-5203-4f02-b8b8-c06fc9eb9353-kube-api-access-ds745\") pod \"payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq\" (UID: \"2d6dcafc-5203-4f02-b8b8-c06fc9eb9353\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq"
Oct 30 13:12:38.777352 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:38.770999    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d6dcafc-5203-4f02-b8b8-c06fc9eb9353-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq\" (UID: \"2d6dcafc-5203-4f02-b8b8-c06fc9eb9353\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq"
Oct 30 13:12:38.890301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:38.889379    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds745\" (UniqueName: \"kubernetes.io/projected/2d6dcafc-5203-4f02-b8b8-c06fc9eb9353-kube-api-access-ds745\") pod \"payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq\" (UID: \"2d6dcafc-5203-4f02-b8b8-c06fc9eb9353\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq"
Oct 30 13:12:39.111874 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:39.111829    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l" event={"ID":"eaa6149f-3c75-4666-8607-5d2686a65b6d","Type":"ContainerStarted","Data":"2386d332bfb4479225eef9bbe398aea56a0926ee89a1875c8ab7148cb8c8c474"}
Oct 30 13:12:39.160101 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:39.160050    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq"
Oct 30 13:12:39.389094 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:39.388996    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:39.513138 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:39.511378    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:12:39.513138 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:39.511933    4544 patch_prober.go:29] interesting pod/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 13:12:39.513138 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:39.511984    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l" podUID="eaa6149f-3c75-4666-8607-5d2686a65b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 13:12:39.628267 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:12:39.608130    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d6dcafc_5203_4f02_b8b8_c06fc9eb9353.slice/crio-18b1180297e4524fe065a0904f990da7c3c4248449bb85c6e03a4699cc4cd4dd WatchSource:0}: Error finding container 18b1180297e4524fe065a0904f990da7c3c4248449bb85c6e03a4699cc4cd4dd: Status 404 returned error can't find the container with id 18b1180297e4524fe065a0904f990da7c3c4248449bb85c6e03a4699cc4cd4dd
Oct 30 13:12:39.672243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:39.662350    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq"]
Oct 30 13:12:40.064296 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:40.064181    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:12:40.145393 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:40.145357    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq" event={"ID":"2d6dcafc-5203-4f02-b8b8-c06fc9eb9353","Type":"ContainerStarted","Data":"fb2c21eff2c65186cc0647473c773103f43566a8891cc512c4e99893460de2d2"}
Oct 30 13:12:40.145875 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:40.145414    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq" event={"ID":"2d6dcafc-5203-4f02-b8b8-c06fc9eb9353","Type":"ContainerStarted","Data":"18b1180297e4524fe065a0904f990da7c3c4248449bb85c6e03a4699cc4cd4dd"}
Oct 30 13:12:40.145875 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:40.145681    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq"
Oct 30 13:12:40.190714 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:40.190663    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq" podStartSLOduration=2.190647555 podStartE2EDuration="2.190647555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:40.186860174 +0000 UTC m=+1655.233994152" watchObservedRunningTime="2026-10-30 13:12:40.190647555 +0000 UTC m=+1655.237781507"
Oct 30 13:12:40.486948 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:40.486916    4544 patch_prober.go:29] interesting pod/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 13:12:40.487157 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:40.486969    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l" podUID="eaa6149f-3c75-4666-8607-5d2686a65b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 13:12:41.379922 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:41.379881    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:41.488541 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:41.488503    4544 patch_prober.go:29] interesting pod/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 13:12:41.488791 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:41.488764    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l" podUID="eaa6149f-3c75-4666-8607-5d2686a65b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 13:12:42.487014 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:42.486977    4544 patch_prober.go:29] interesting pod/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 13:12:42.487499 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:42.487037    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l" podUID="eaa6149f-3c75-4666-8607-5d2686a65b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 13:12:43.233247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:43.231160    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:43.487148 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:43.487042    4544 patch_prober.go:29] interesting pod/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 13:12:43.487148 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:43.487107    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l" podUID="eaa6149f-3c75-4666-8607-5d2686a65b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 13:12:44.486818 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:44.486777    4544 patch_prober.go:29] interesting pod/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 13:12:44.486986 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:44.486839    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l" podUID="eaa6149f-3c75-4666-8607-5d2686a65b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 13:12:45.180775 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:45.180744    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:45.492057 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:45.491973    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:12:45.493587 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:45.493559    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:12:46.989323 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:46.986700    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:48.698108 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:48.698072    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:49.313077 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:49.313044    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:49.557180 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:49.557147    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:50.090205 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:50.090171    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:50.279726 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:50.279690    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:51.213559 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:51.213520    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq"
Oct 30 13:12:52.388244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:52.371728    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:54.307673 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:54.307632    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:54.783761 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:54.783729    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:55.179312 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:55.179278    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:55.635088 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:55.635048    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:55.783887 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:55.783849    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:56.490531 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:56.490496    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:57.685904 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:57.685867    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:57.815080 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:57.815044    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:57.874991 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:57.874960    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g"]
Oct 30 13:12:57.879932 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:57.879909    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g"
Oct 30 13:12:57.901170 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:57.901139    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g"]
Oct 30 13:12:57.906420 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:57.906399    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-14e8a4\""
Oct 30 13:12:57.913856 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:57.913821    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-14e8a4-dockercfg-wv67l\""
Oct 30 13:12:58.080253 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:58.076998    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad4d3ca-4f7d-418f-b703-cb9c10e101f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g\" (UID: \"aad4d3ca-4f7d-418f-b703-cb9c10e101f9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g"
Oct 30 13:12:58.080253 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:58.077109    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s2v4\" (UniqueName: \"kubernetes.io/projected/aad4d3ca-4f7d-418f-b703-cb9c10e101f9-kube-api-access-4s2v4\") pod \"payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g\" (UID: \"aad4d3ca-4f7d-418f-b703-cb9c10e101f9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g"
Oct 30 13:12:58.226839 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:58.226790    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad4d3ca-4f7d-418f-b703-cb9c10e101f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g\" (UID: \"aad4d3ca-4f7d-418f-b703-cb9c10e101f9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g"
Oct 30 13:12:58.228174 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:58.228139    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s2v4\" (UniqueName: \"kubernetes.io/projected/aad4d3ca-4f7d-418f-b703-cb9c10e101f9-kube-api-access-4s2v4\") pod \"payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g\" (UID: \"aad4d3ca-4f7d-418f-b703-cb9c10e101f9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g"
Oct 30 13:12:58.229589 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:58.229565    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad4d3ca-4f7d-418f-b703-cb9c10e101f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g\" (UID: \"aad4d3ca-4f7d-418f-b703-cb9c10e101f9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g"
Oct 30 13:12:58.301274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:58.297126    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s2v4\" (UniqueName: \"kubernetes.io/projected/aad4d3ca-4f7d-418f-b703-cb9c10e101f9-kube-api-access-4s2v4\") pod \"payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g\" (UID: \"aad4d3ca-4f7d-418f-b703-cb9c10e101f9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g"
Oct 30 13:12:58.503743 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:58.496630    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g"
Oct 30 13:12:59.009208 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:12:59.009172    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:12:59.098279 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:59.098252    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g"]
Oct 30 13:12:59.100099 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:12:59.100074    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad4d3ca_4f7d_418f_b703_cb9c10e101f9.slice/crio-0de77be0c4e78e4d50bae7ed47f5ffd5d7bb1743fa43c89625820a4bbce6e68e WatchSource:0}: Error finding container 0de77be0c4e78e4d50bae7ed47f5ffd5d7bb1743fa43c89625820a4bbce6e68e: Status 404 returned error can't find the container with id 0de77be0c4e78e4d50bae7ed47f5ffd5d7bb1743fa43c89625820a4bbce6e68e
Oct 30 13:12:59.203708 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:59.203678    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:12:59.614041 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:59.614003    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g" event={"ID":"aad4d3ca-4f7d-418f-b703-cb9c10e101f9","Type":"ContainerStarted","Data":"fa5d5e079221541266f2904a9a1038a8e53af569fedf3c71d51e4ba94f9e889c"}
Oct 30 13:12:59.614319 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:59.614066    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g" event={"ID":"aad4d3ca-4f7d-418f-b703-cb9c10e101f9","Type":"ContainerStarted","Data":"0de77be0c4e78e4d50bae7ed47f5ffd5d7bb1743fa43c89625820a4bbce6e68e"}
Oct 30 13:12:59.614319 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:59.614113    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g"
Oct 30 13:12:59.655305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:12:59.655243    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g" podStartSLOduration=2.655216459 podStartE2EDuration="2.655216459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:59.653457755 +0000 UTC m=+1674.700591709" watchObservedRunningTime="2026-10-30 13:12:59.655216459 +0000 UTC m=+1674.702350413"
Oct 30 13:13:01.285045 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:01.285002    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:03.214623 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:03.214516    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:05.087198 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:05.087158    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:07.261855 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:07.261799    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:09.359126 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:09.359093    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:10.420706 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:10.419327    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:10.626252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:10.626187    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g"
Oct 30 13:13:11.105682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:11.105641    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:11.442609 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:11.442522    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:11.672339 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:11.672292    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:11.812312 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:11.810768    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:13.770651 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:13.770611    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:16.183242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:16.178807    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:16.561632 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:16.561546    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:17.853481 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:17.853449    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:18.024476 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:18.024445    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:18.059631 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:18.059593    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps"]
Oct 30 13:13:18.066160 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:18.066134    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps"
Oct 30 13:13:18.099401 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:18.099368    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps"]
Oct 30 13:13:18.105188 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:18.105107    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-11cfa6\""
Oct 30 13:13:18.115855 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:18.115818    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxwxt\" (UniqueName: \"kubernetes.io/projected/b885ada7-2fc8-4596-900d-6da86c038051-kube-api-access-xxwxt\") pod \"payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps\" (UID: \"b885ada7-2fc8-4596-900d-6da86c038051\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps"
Oct 30 13:13:18.116300 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:18.116270    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b885ada7-2fc8-4596-900d-6da86c038051-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps\" (UID: \"b885ada7-2fc8-4596-900d-6da86c038051\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps"
Oct 30 13:13:18.280258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:18.265111    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxwxt\" (UniqueName: \"kubernetes.io/projected/b885ada7-2fc8-4596-900d-6da86c038051-kube-api-access-xxwxt\") pod \"payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps\" (UID: \"b885ada7-2fc8-4596-900d-6da86c038051\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps"
Oct 30 13:13:18.280258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:18.265252    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b885ada7-2fc8-4596-900d-6da86c038051-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps\" (UID: \"b885ada7-2fc8-4596-900d-6da86c038051\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps"
Oct 30 13:13:18.280258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:18.266469    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b885ada7-2fc8-4596-900d-6da86c038051-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps\" (UID: \"b885ada7-2fc8-4596-900d-6da86c038051\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps"
Oct 30 13:13:18.435825 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:18.435737    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxwxt\" (UniqueName: \"kubernetes.io/projected/b885ada7-2fc8-4596-900d-6da86c038051-kube-api-access-xxwxt\") pod \"payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps\" (UID: \"b885ada7-2fc8-4596-900d-6da86c038051\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps"
Oct 30 13:13:18.682367 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:18.682330    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps"
Oct 30 13:13:18.987411 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:18.987250    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:19.062244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:19.047020    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps"]
Oct 30 13:13:19.832336 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:19.832288    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps" event={"ID":"b885ada7-2fc8-4596-900d-6da86c038051","Type":"ContainerStarted","Data":"34dd37f9ac4aad6edc1a3f9c6a7967711c01ebfadcd87b9f67fde5764289e6eb"}
Oct 30 13:13:19.832679 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:19.832651    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps" event={"ID":"b885ada7-2fc8-4596-900d-6da86c038051","Type":"ContainerStarted","Data":"ba1edde91787782b7a97e4a7020084a9ce163f90173c930ad229c7f44900be7a"}
Oct 30 13:13:19.834124 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:19.834104    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps"
Oct 30 13:13:21.449738 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:21.449680    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:22.210030 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:22.209996    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:22.350145 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:22.350087    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:24.007052 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:24.006995    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:25.692366 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:25.692320    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:26.099778 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:26.099729    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps" podStartSLOduration=8.099715042 podStartE2EDuration="8.099715042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:19.892066413 +0000 UTC m=+1694.939200375" watchObservedRunningTime="2026-10-30 13:13:26.099715042 +0000 UTC m=+1701.146848994"
Oct 30 13:13:26.104038 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:26.104011    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7"]
Oct 30 13:13:26.109644 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:26.109621    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7"
Oct 30 13:13:26.130099 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:26.130068    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-2a3e01-serving-cert\""
Oct 30 13:13:26.153184 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:26.153134    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7"]
Oct 30 13:13:26.202547 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:26.202510    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:26.263442 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:26.263408    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/040d15f4-178f-41c8-8932-80bdfcfb6780-metrics-certs\") pod \"maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7\" (UID: \"040d15f4-178f-41c8-8932-80bdfcfb6780\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7"
Oct 30 13:13:26.263631 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:26.263557    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qtrf\" (UniqueName: \"kubernetes.io/projected/040d15f4-178f-41c8-8932-80bdfcfb6780-kube-api-access-6qtrf\") pod \"maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7\" (UID: \"040d15f4-178f-41c8-8932-80bdfcfb6780\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7"
Oct 30 13:13:26.263806 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:26.263784    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/040d15f4-178f-41c8-8932-80bdfcfb6780-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7\" (UID: \"040d15f4-178f-41c8-8932-80bdfcfb6780\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7"
Oct 30 13:13:26.263865 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:26.263850    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/040d15f4-178f-41c8-8932-80bdfcfb6780-maas-api-tls\") pod \"maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7\" (UID: \"040d15f4-178f-41c8-8932-80bdfcfb6780\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7"
Oct 30 13:13:26.344197 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:26.343274    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:26.367884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:26.367749    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/040d15f4-178f-41c8-8932-80bdfcfb6780-metrics-certs\") pod \"maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7\" (UID: \"040d15f4-178f-41c8-8932-80bdfcfb6780\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7"
Oct 30 13:13:26.368092 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:26.367895    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qtrf\" (UniqueName: \"kubernetes.io/projected/040d15f4-178f-41c8-8932-80bdfcfb6780-kube-api-access-6qtrf\") pod \"maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7\" (UID: \"040d15f4-178f-41c8-8932-80bdfcfb6780\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7"
Oct 30 13:13:26.368179 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:26.368142    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/040d15f4-178f-41c8-8932-80bdfcfb6780-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7\" (UID: \"040d15f4-178f-41c8-8932-80bdfcfb6780\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7"
Oct 30 13:13:26.368332 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:26.368248    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/040d15f4-178f-41c8-8932-80bdfcfb6780-maas-api-tls\") pod \"maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7\" (UID: \"040d15f4-178f-41c8-8932-80bdfcfb6780\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7"
Oct 30 13:13:26.398758 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:26.374010    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/040d15f4-178f-41c8-8932-80bdfcfb6780-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7\" (UID: \"040d15f4-178f-41c8-8932-80bdfcfb6780\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7"
Oct 30 13:13:26.399136 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:26.399108    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/040d15f4-178f-41c8-8932-80bdfcfb6780-maas-api-tls\") pod \"maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7\" (UID: \"040d15f4-178f-41c8-8932-80bdfcfb6780\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7"
Oct 30 13:13:26.416252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:26.404752    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qtrf\" (UniqueName: \"kubernetes.io/projected/040d15f4-178f-41c8-8932-80bdfcfb6780-kube-api-access-6qtrf\") pod \"maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7\" (UID: \"040d15f4-178f-41c8-8932-80bdfcfb6780\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7"
Oct 30 13:13:26.435939 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:26.435901    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/040d15f4-178f-41c8-8932-80bdfcfb6780-metrics-certs\") pod \"maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7\" (UID: \"040d15f4-178f-41c8-8932-80bdfcfb6780\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7"
Oct 30 13:13:26.485533 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:26.485493    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:26.726974 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:26.726896    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7"
Oct 30 13:13:27.056696 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:27.056659    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7"]
Oct 30 13:13:27.058384 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:13:27.058333    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod040d15f4_178f_41c8_8932_80bdfcfb6780.slice/crio-62ea5b4469b587425cb13de12465311532ec9e43a1e56e71b85661835230a434 WatchSource:0}: Error finding container 62ea5b4469b587425cb13de12465311532ec9e43a1e56e71b85661835230a434: Status 404 returned error can't find the container with id 62ea5b4469b587425cb13de12465311532ec9e43a1e56e71b85661835230a434
Oct 30 13:13:28.003675 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:28.003578    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7" event={"ID":"040d15f4-178f-41c8-8932-80bdfcfb6780","Type":"ContainerStarted","Data":"22b47f8343f1b706ff113d7a6a56ca27fcfec0bd651d42ddf36ab8b4b54fb156"}
Oct 30 13:13:28.003675 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:28.003654    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7" event={"ID":"040d15f4-178f-41c8-8932-80bdfcfb6780","Type":"ContainerStarted","Data":"62ea5b4469b587425cb13de12465311532ec9e43a1e56e71b85661835230a434"}
Oct 30 13:13:28.004235 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:28.003695    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7"
Oct 30 13:13:28.047359 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:28.047287    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7" podStartSLOduration=1.717829723 podStartE2EDuration="2.047265581s" totalImagesPullingTime="329.435858ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:28.04574698 +0000 UTC m=+1703.092880932" watchObservedRunningTime="2026-10-30 13:13:28.047265581 +0000 UTC m=+1703.094399535"
Oct 30 13:13:28.408295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:28.405850    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:29.009097 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:13:29.009054    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:13:30.514599 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:30.514563    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:31.928515 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:31.928484    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps"
Oct 30 13:13:32.378905 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:32.378842    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:34.017973 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:34.017943    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7"
Oct 30 13:13:34.211850 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:34.211815    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:36.618308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:36.618272    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:38.399543 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:38.399508    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc"]
Oct 30 13:13:38.433646 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:38.433615    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc"]
Oct 30 13:13:38.564612 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:38.564572    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:38.786797 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:38.786716    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:38.993545 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:38.993504    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:40.574712 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:40.574674    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv"]
Oct 30 13:13:40.576063 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:40.574952    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv" podUID="e2df23af-713d-4c5c-9d63-5999de49d67e" containerName="maas-api" containerID="cri-o://76669f4398827ed3a4f0d4db40ed05a178f7689b6da92930ab64c335aa2b9590" gracePeriod=30
Oct 30 13:13:40.630989 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:40.630958    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:41.385267 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:41.378643    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv"
Oct 30 13:13:41.385988 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:41.385952    4544 generic.go:356] "Generic (PLEG): container finished" podID="e2df23af-713d-4c5c-9d63-5999de49d67e" containerID="76669f4398827ed3a4f0d4db40ed05a178f7689b6da92930ab64c335aa2b9590" exitCode=0
Oct 30 13:13:41.386112 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:41.386065    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv" event={"ID":"e2df23af-713d-4c5c-9d63-5999de49d67e","Type":"ContainerDied","Data":"76669f4398827ed3a4f0d4db40ed05a178f7689b6da92930ab64c335aa2b9590"}
Oct 30 13:13:41.386178 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:41.386125    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv" event={"ID":"e2df23af-713d-4c5c-9d63-5999de49d67e","Type":"ContainerDied","Data":"f7080cd3e8128a3248033b923a3f12f2659b0f5c471ffed64324aab5c8f00c53"}
Oct 30 13:13:41.386178 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:41.386156    4544 scope.go:122] "RemoveContainer" containerID="76669f4398827ed3a4f0d4db40ed05a178f7689b6da92930ab64c335aa2b9590"
Oct 30 13:13:41.402385 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:41.402352    4544 scope.go:122] "RemoveContainer" containerID="76669f4398827ed3a4f0d4db40ed05a178f7689b6da92930ab64c335aa2b9590"
Oct 30 13:13:41.402854 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:13:41.402826    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76669f4398827ed3a4f0d4db40ed05a178f7689b6da92930ab64c335aa2b9590\": container with ID starting with 76669f4398827ed3a4f0d4db40ed05a178f7689b6da92930ab64c335aa2b9590 not found: ID does not exist" containerID="76669f4398827ed3a4f0d4db40ed05a178f7689b6da92930ab64c335aa2b9590"
Oct 30 13:13:41.402938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:41.402870    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76669f4398827ed3a4f0d4db40ed05a178f7689b6da92930ab64c335aa2b9590"} err="failed to get container status \"76669f4398827ed3a4f0d4db40ed05a178f7689b6da92930ab64c335aa2b9590\": rpc error: code = NotFound desc = could not find container \"76669f4398827ed3a4f0d4db40ed05a178f7689b6da92930ab64c335aa2b9590\": container with ID starting with 76669f4398827ed3a4f0d4db40ed05a178f7689b6da92930ab64c335aa2b9590 not found: ID does not exist"
Oct 30 13:13:41.454852 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:41.454812    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2df23af-713d-4c5c-9d63-5999de49d67e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2df23af-713d-4c5c-9d63-5999de49d67e-metrics-certs\") pod \"e2df23af-713d-4c5c-9d63-5999de49d67e\" (UID: \"e2df23af-713d-4c5c-9d63-5999de49d67e\") "
Oct 30 13:13:41.455064 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:41.454995    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2df23af-713d-4c5c-9d63-5999de49d67e-kube-api-access-6f778\" (UniqueName: \"kubernetes.io/projected/e2df23af-713d-4c5c-9d63-5999de49d67e-kube-api-access-6f778\") pod \"e2df23af-713d-4c5c-9d63-5999de49d67e\" (UID: \"e2df23af-713d-4c5c-9d63-5999de49d67e\") "
Oct 30 13:13:41.455150 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:41.455137    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2df23af-713d-4c5c-9d63-5999de49d67e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2df23af-713d-4c5c-9d63-5999de49d67e-maas-api-tls\") pod \"e2df23af-713d-4c5c-9d63-5999de49d67e\" (UID: \"e2df23af-713d-4c5c-9d63-5999de49d67e\") "
Oct 30 13:13:41.461356 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:41.455199    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2df23af-713d-4c5c-9d63-5999de49d67e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2df23af-713d-4c5c-9d63-5999de49d67e-openshift-service-ca\") pod \"e2df23af-713d-4c5c-9d63-5999de49d67e\" (UID: \"e2df23af-713d-4c5c-9d63-5999de49d67e\") "
Oct 30 13:13:41.461356 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:41.456395    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2df23af-713d-4c5c-9d63-5999de49d67e-openshift-service-ca" pod "e2df23af-713d-4c5c-9d63-5999de49d67e" (UID: "e2df23af-713d-4c5c-9d63-5999de49d67e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:13:41.461356 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:41.459552    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2df23af-713d-4c5c-9d63-5999de49d67e-maas-api-tls" pod "e2df23af-713d-4c5c-9d63-5999de49d67e" (UID: "e2df23af-713d-4c5c-9d63-5999de49d67e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:13:41.461592 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:41.461562    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2df23af-713d-4c5c-9d63-5999de49d67e-metrics-certs" pod "e2df23af-713d-4c5c-9d63-5999de49d67e" (UID: "e2df23af-713d-4c5c-9d63-5999de49d67e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:13:41.465895 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:41.465865    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2df23af-713d-4c5c-9d63-5999de49d67e-kube-api-access-6f778" pod "e2df23af-713d-4c5c-9d63-5999de49d67e" (UID: "e2df23af-713d-4c5c-9d63-5999de49d67e"). InnerVolumeSpecName "kube-api-access-6f778". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:13:41.519044 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:41.518953    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:41.557890 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:41.557860    4544 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2df23af-713d-4c5c-9d63-5999de49d67e-metrics-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:13:41.557890 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:41.557886    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f778\" (UniqueName: \"kubernetes.io/projected/e2df23af-713d-4c5c-9d63-5999de49d67e-kube-api-access-6f778\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:13:41.557890 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:41.557896    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2df23af-713d-4c5c-9d63-5999de49d67e-maas-api-tls\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:13:41.558131 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:41.557907    4544 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2df23af-713d-4c5c-9d63-5999de49d67e-openshift-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:13:41.743775 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:41.743726    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:41.867777 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:41.867695    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:42.322027 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:42.321939    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:42.394693 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:42.394660    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv"
Oct 30 13:13:42.448952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:42.448917    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv"]
Oct 30 13:13:42.455651 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:42.455623    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-75abce-578cc55f9f-9kvpv"]
Oct 30 13:13:43.089942 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:43.089902    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2df23af-713d-4c5c-9d63-5999de49d67e" path="/var/lib/kubelet/pods/e2df23af-713d-4c5c-9d63-5999de49d67e/volumes"
Oct 30 13:13:43.399937 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:43.399853    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc" podUID="29a6d4c7-d13a-41ed-a022-57ab28b1ff00" containerName="payload-processing" containerID="cri-o://f73fce2f09d11365a4a8eff5399781854cb2e359b90c9048a4f3172d296cccf4" gracePeriod=25
Oct 30 13:13:43.434864 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:43.434830    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc" podUID="5a040f35-d461-49fb-8e95-d10adcb01a0a" containerName="payload-pre-processing" containerID="cri-o://06bb03f970e991635db89c0aed9aad7f5d40c5628e5e38ca779fa46551298f8d" gracePeriod=25
Oct 30 13:13:43.740768 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:43.740731    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc"
Oct 30 13:13:43.784896 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:43.784867    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc"
Oct 30 13:13:43.803824 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:43.803790    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29a6d4c7-d13a-41ed-a022-57ab28b1ff00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29a6d4c7-d13a-41ed-a022-57ab28b1ff00-plugins-config-volume\") pod \"29a6d4c7-d13a-41ed-a022-57ab28b1ff00\" (UID: \"29a6d4c7-d13a-41ed-a022-57ab28b1ff00\") "
Oct 30 13:13:43.804005 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:43.803878    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29a6d4c7-d13a-41ed-a022-57ab28b1ff00-kube-api-access-pz8gw\" (UniqueName: \"kubernetes.io/projected/29a6d4c7-d13a-41ed-a022-57ab28b1ff00-kube-api-access-pz8gw\") pod \"29a6d4c7-d13a-41ed-a022-57ab28b1ff00\" (UID: \"29a6d4c7-d13a-41ed-a022-57ab28b1ff00\") "
Oct 30 13:13:43.804179 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:43.804157    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a6d4c7-d13a-41ed-a022-57ab28b1ff00-plugins-config-volume" pod "29a6d4c7-d13a-41ed-a022-57ab28b1ff00" (UID: "29a6d4c7-d13a-41ed-a022-57ab28b1ff00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:13:43.804602 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:43.804578    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29a6d4c7-d13a-41ed-a022-57ab28b1ff00-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:13:43.806100 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:43.806080    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a6d4c7-d13a-41ed-a022-57ab28b1ff00-kube-api-access-pz8gw" pod "29a6d4c7-d13a-41ed-a022-57ab28b1ff00" (UID: "29a6d4c7-d13a-41ed-a022-57ab28b1ff00"). InnerVolumeSpecName "kube-api-access-pz8gw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:13:43.905234 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:43.905194    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a040f35-d461-49fb-8e95-d10adcb01a0a-kube-api-access-qbvgc\" (UniqueName: \"kubernetes.io/projected/5a040f35-d461-49fb-8e95-d10adcb01a0a-kube-api-access-qbvgc\") pod \"5a040f35-d461-49fb-8e95-d10adcb01a0a\" (UID: \"5a040f35-d461-49fb-8e95-d10adcb01a0a\") "
Oct 30 13:13:43.905494 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:43.905477    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a040f35-d461-49fb-8e95-d10adcb01a0a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a040f35-d461-49fb-8e95-d10adcb01a0a-plugins-config-volume\") pod \"5a040f35-d461-49fb-8e95-d10adcb01a0a\" (UID: \"5a040f35-d461-49fb-8e95-d10adcb01a0a\") "
Oct 30 13:13:43.905607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:43.905545    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:43.906070 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:43.906047    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz8gw\" (UniqueName: \"kubernetes.io/projected/29a6d4c7-d13a-41ed-a022-57ab28b1ff00-kube-api-access-pz8gw\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:13:43.906070 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:43.906057    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a040f35-d461-49fb-8e95-d10adcb01a0a-plugins-config-volume" pod "5a040f35-d461-49fb-8e95-d10adcb01a0a" (UID: "5a040f35-d461-49fb-8e95-d10adcb01a0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:13:43.907640 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:43.907614    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a040f35-d461-49fb-8e95-d10adcb01a0a-kube-api-access-qbvgc" pod "5a040f35-d461-49fb-8e95-d10adcb01a0a" (UID: "5a040f35-d461-49fb-8e95-d10adcb01a0a"). InnerVolumeSpecName "kube-api-access-qbvgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:13:44.017299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:44.008599    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a040f35-d461-49fb-8e95-d10adcb01a0a-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:13:44.017299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:44.008638    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbvgc\" (UniqueName: \"kubernetes.io/projected/5a040f35-d461-49fb-8e95-d10adcb01a0a-kube-api-access-qbvgc\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:13:44.412010 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:44.411974    4544 generic.go:356] "Generic (PLEG): container finished" podID="5a040f35-d461-49fb-8e95-d10adcb01a0a" containerID="06bb03f970e991635db89c0aed9aad7f5d40c5628e5e38ca779fa46551298f8d" exitCode=0
Oct 30 13:13:44.412542 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:44.412027    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc"
Oct 30 13:13:44.412542 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:44.412062    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc" event={"ID":"5a040f35-d461-49fb-8e95-d10adcb01a0a","Type":"ContainerDied","Data":"06bb03f970e991635db89c0aed9aad7f5d40c5628e5e38ca779fa46551298f8d"}
Oct 30 13:13:44.412542 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:44.412126    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc" event={"ID":"5a040f35-d461-49fb-8e95-d10adcb01a0a","Type":"ContainerDied","Data":"c9a4ce30bbf07f2e354c85234a9ce4cd885f9501970ed6c6972ac1979f1a1159"}
Oct 30 13:13:44.412542 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:44.412157    4544 scope.go:122] "RemoveContainer" containerID="06bb03f970e991635db89c0aed9aad7f5d40c5628e5e38ca779fa46551298f8d"
Oct 30 13:13:44.414551 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:44.413822    4544 generic.go:356] "Generic (PLEG): container finished" podID="29a6d4c7-d13a-41ed-a022-57ab28b1ff00" containerID="f73fce2f09d11365a4a8eff5399781854cb2e359b90c9048a4f3172d296cccf4" exitCode=0
Oct 30 13:13:44.414551 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:44.413855    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc" event={"ID":"29a6d4c7-d13a-41ed-a022-57ab28b1ff00","Type":"ContainerDied","Data":"f73fce2f09d11365a4a8eff5399781854cb2e359b90c9048a4f3172d296cccf4"}
Oct 30 13:13:44.414551 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:44.413881    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc"
Oct 30 13:13:44.414551 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:44.413918    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc" event={"ID":"29a6d4c7-d13a-41ed-a022-57ab28b1ff00","Type":"ContainerDied","Data":"b5d7da5eda68523c1f837601bad6c5a59daa6703eeb5f538435ac3185ccc9782"}
Oct 30 13:13:44.435044 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:44.435025    4544 scope.go:122] "RemoveContainer" containerID="06bb03f970e991635db89c0aed9aad7f5d40c5628e5e38ca779fa46551298f8d"
Oct 30 13:13:44.435520 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:13:44.435497    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06bb03f970e991635db89c0aed9aad7f5d40c5628e5e38ca779fa46551298f8d\": container with ID starting with 06bb03f970e991635db89c0aed9aad7f5d40c5628e5e38ca779fa46551298f8d not found: ID does not exist" containerID="06bb03f970e991635db89c0aed9aad7f5d40c5628e5e38ca779fa46551298f8d"
Oct 30 13:13:44.435595 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:44.435528    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06bb03f970e991635db89c0aed9aad7f5d40c5628e5e38ca779fa46551298f8d"} err="failed to get container status \"06bb03f970e991635db89c0aed9aad7f5d40c5628e5e38ca779fa46551298f8d\": rpc error: code = NotFound desc = could not find container \"06bb03f970e991635db89c0aed9aad7f5d40c5628e5e38ca779fa46551298f8d\": container with ID starting with 06bb03f970e991635db89c0aed9aad7f5d40c5628e5e38ca779fa46551298f8d not found: ID does not exist"
Oct 30 13:13:44.435595 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:44.435547    4544 scope.go:122] "RemoveContainer" containerID="f73fce2f09d11365a4a8eff5399781854cb2e359b90c9048a4f3172d296cccf4"
Oct 30 13:13:44.446421 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:44.446401    4544 scope.go:122] "RemoveContainer" containerID="f73fce2f09d11365a4a8eff5399781854cb2e359b90c9048a4f3172d296cccf4"
Oct 30 13:13:44.446757 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:13:44.446733    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f73fce2f09d11365a4a8eff5399781854cb2e359b90c9048a4f3172d296cccf4\": container with ID starting with f73fce2f09d11365a4a8eff5399781854cb2e359b90c9048a4f3172d296cccf4 not found: ID does not exist" containerID="f73fce2f09d11365a4a8eff5399781854cb2e359b90c9048a4f3172d296cccf4"
Oct 30 13:13:44.446817 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:44.446768    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f73fce2f09d11365a4a8eff5399781854cb2e359b90c9048a4f3172d296cccf4"} err="failed to get container status \"f73fce2f09d11365a4a8eff5399781854cb2e359b90c9048a4f3172d296cccf4\": rpc error: code = NotFound desc = could not find container \"f73fce2f09d11365a4a8eff5399781854cb2e359b90c9048a4f3172d296cccf4\": container with ID starting with f73fce2f09d11365a4a8eff5399781854cb2e359b90c9048a4f3172d296cccf4 not found: ID does not exist"
Oct 30 13:13:44.475265 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:44.475216    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc"]
Oct 30 13:13:44.487005 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:44.486968    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc"]
Oct 30 13:13:44.515061 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:44.515023    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc"]
Oct 30 13:13:44.523997 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:44.521302    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc"]
Oct 30 13:13:45.090597 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:45.090555    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29a6d4c7-d13a-41ed-a022-57ab28b1ff00" path="/var/lib/kubelet/pods/29a6d4c7-d13a-41ed-a022-57ab28b1ff00/volumes"
Oct 30 13:13:45.091056 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:45.091040    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a040f35-d461-49fb-8e95-d10adcb01a0a" path="/var/lib/kubelet/pods/5a040f35-d461-49fb-8e95-d10adcb01a0a/volumes"
Oct 30 13:13:45.768385 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:45.768354    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:46.195952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.195919    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx"]
Oct 30 13:13:46.196969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.196944    4544 cpu_manager.go:414] "containerMap: removing container" podUID="29a6d4c7-d13a-41ed-a022-57ab28b1ff00" containerName="payload-processing"
Oct 30 13:13:46.196969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.196967    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29a6d4c7-d13a-41ed-a022-57ab28b1ff00" containerName="payload-processing"
Oct 30 13:13:46.197161 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.196999    4544 cpu_manager.go:414] "containerMap: removing container" podUID="5a040f35-d461-49fb-8e95-d10adcb01a0a" containerName="payload-pre-processing"
Oct 30 13:13:46.197161 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.197007    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a040f35-d461-49fb-8e95-d10adcb01a0a" containerName="payload-pre-processing"
Oct 30 13:13:46.197161 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.197030    4544 cpu_manager.go:414] "containerMap: removing container" podUID="e2df23af-713d-4c5c-9d63-5999de49d67e" containerName="maas-api"
Oct 30 13:13:46.197161 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.197038    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2df23af-713d-4c5c-9d63-5999de49d67e" containerName="maas-api"
Oct 30 13:13:46.197417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.197323    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="29a6d4c7-d13a-41ed-a022-57ab28b1ff00" containerName="payload-processing"
Oct 30 13:13:46.197417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.197344    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a040f35-d461-49fb-8e95-d10adcb01a0a" containerName="payload-pre-processing"
Oct 30 13:13:46.197417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.197357    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2df23af-713d-4c5c-9d63-5999de49d67e" containerName="maas-api"
Oct 30 13:13:46.203347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.203325    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx"
Oct 30 13:13:46.225953 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.225921    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx"]
Oct 30 13:13:46.230027 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.229972    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-14e8a4-serving-cert\""
Oct 30 13:13:46.325168 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.325108    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"]
Oct 30 13:13:46.334629 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.334574    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:13:46.348467 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.348433    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2phj\" (UniqueName: \"kubernetes.io/projected/a1a9ed47-9838-43d5-bf54-7133d03c25fd-kube-api-access-l2phj\") pod \"maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx\" (UID: \"a1a9ed47-9838-43d5-bf54-7133d03c25fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx"
Oct 30 13:13:46.348832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.348810    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a9ed47-9838-43d5-bf54-7133d03c25fd-metrics-certs\") pod \"maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx\" (UID: \"a1a9ed47-9838-43d5-bf54-7133d03c25fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx"
Oct 30 13:13:46.348934 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.348919    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a9ed47-9838-43d5-bf54-7133d03c25fd-openshift-service-ca\") pod \"maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx\" (UID: \"a1a9ed47-9838-43d5-bf54-7133d03c25fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx"
Oct 30 13:13:46.349018 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.348956    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a9ed47-9838-43d5-bf54-7133d03c25fd-maas-api-tls\") pod \"maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx\" (UID: \"a1a9ed47-9838-43d5-bf54-7133d03c25fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx"
Oct 30 13:13:46.367636 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.367599    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4a8ee1-openshift-default-dockercfg-btzbp\""
Oct 30 13:13:46.372712 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.372671    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"]
Oct 30 13:13:46.377665 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.377631    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:46.451000 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.450912    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istio-podinfo\") pod \"e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") " pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:13:46.451000 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.450961    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-credential-socket\") pod \"e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") " pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:13:46.451000 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.450980    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-workload-certs\") pod \"e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") " pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:13:46.451241 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.451078    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a9ed47-9838-43d5-bf54-7133d03c25fd-openshift-service-ca\") pod \"maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx\" (UID: \"a1a9ed47-9838-43d5-bf54-7133d03c25fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx"
Oct 30 13:13:46.451241 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.451162    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a9ed47-9838-43d5-bf54-7133d03c25fd-maas-api-tls\") pod \"maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx\" (UID: \"a1a9ed47-9838-43d5-bf54-7133d03c25fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx"
Oct 30 13:13:46.451241 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.451181    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istio-envoy\") pod \"e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") " pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:13:46.451343 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.451266    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2phj\" (UniqueName: \"kubernetes.io/projected/a1a9ed47-9838-43d5-bf54-7133d03c25fd-kube-api-access-l2phj\") pod \"maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx\" (UID: \"a1a9ed47-9838-43d5-bf54-7133d03c25fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx"
Oct 30 13:13:46.451396 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.451374    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istio-data\") pod \"e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") " pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:13:46.451438 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.451424    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istio-token\") pod \"e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") " pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:13:46.451643 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.451626    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-workload-socket\") pod \"e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") " pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:13:46.451691 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.451671    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istiod-ca-cert\") pod \"e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") " pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:13:46.451821 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.451799    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9lpb\" (UniqueName: \"kubernetes.io/projected/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-kube-api-access-w9lpb\") pod \"e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") " pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:13:46.451821 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.451798    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a9ed47-9838-43d5-bf54-7133d03c25fd-openshift-service-ca\") pod \"maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx\" (UID: \"a1a9ed47-9838-43d5-bf54-7133d03c25fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx"
Oct 30 13:13:46.451889 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.451844    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a9ed47-9838-43d5-bf54-7133d03c25fd-metrics-certs\") pod \"maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx\" (UID: \"a1a9ed47-9838-43d5-bf54-7133d03c25fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx"
Oct 30 13:13:46.453696 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.453675    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a9ed47-9838-43d5-bf54-7133d03c25fd-maas-api-tls\") pod \"maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx\" (UID: \"a1a9ed47-9838-43d5-bf54-7133d03c25fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx"
Oct 30 13:13:46.454378 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.454359    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a9ed47-9838-43d5-bf54-7133d03c25fd-metrics-certs\") pod \"maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx\" (UID: \"a1a9ed47-9838-43d5-bf54-7133d03c25fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx"
Oct 30 13:13:46.558572 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.558527    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-workload-socket\") pod \"e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") " pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:13:46.558827 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.558809    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istiod-ca-cert\") pod \"e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") " pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:13:46.559007 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.558991    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9lpb\" (UniqueName: \"kubernetes.io/projected/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-kube-api-access-w9lpb\") pod \"e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") " pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:13:46.559148 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.559133    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istio-podinfo\") pod \"e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") " pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:13:46.559288 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.559273    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-credential-socket\") pod \"e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") " pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:13:46.559395 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.559379    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-workload-certs\") pod \"e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") " pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:13:46.559560 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.559543    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istio-envoy\") pod \"e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") " pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:13:46.559801 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.559786    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istio-data\") pod \"e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") " pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:13:46.559940 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.559926    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istio-token\") pod \"e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") " pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:13:46.576242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.560918    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-workload-socket\") pod \"e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") " pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:13:46.577300 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.577266    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istiod-ca-cert\") pod \"e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") " pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:13:46.618073 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.618038    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istio-podinfo\") pod \"e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") " pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:13:46.618639 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.618610    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istio-data\") pod \"e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") " pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:13:46.618889 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.618863    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2phj\" (UniqueName: \"kubernetes.io/projected/a1a9ed47-9838-43d5-bf54-7133d03c25fd-kube-api-access-l2phj\") pod \"maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx\" (UID: \"a1a9ed47-9838-43d5-bf54-7133d03c25fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx"
Oct 30 13:13:46.620741 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.620349    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-credential-socket\") pod \"e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") " pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:13:46.621878 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.621139    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-workload-certs\") pod \"e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") " pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:13:46.645840 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.645455    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:46.671247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.648342    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istio-envoy\") pod \"e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") " pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:13:46.696252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.690885    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istio-token\") pod \"e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") " pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:13:46.729377 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.706206    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9lpb\" (UniqueName: \"kubernetes.io/projected/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-kube-api-access-w9lpb\") pod \"e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") " pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:13:46.818847 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.818562    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx"
Oct 30 13:13:46.964793 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:46.964744    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:13:47.193792 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:47.193752    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx"]
Oct 30 13:13:47.195018 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:13:47.194970    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a9ed47_9838_43d5_bf54_7133d03c25fd.slice/crio-208b12cf3a7fd2334e1b489e69ebc79eba8ee7e01c499bb233d285df1f82d3df WatchSource:0}: Error finding container 208b12cf3a7fd2334e1b489e69ebc79eba8ee7e01c499bb233d285df1f82d3df: Status 404 returned error can't find the container with id 208b12cf3a7fd2334e1b489e69ebc79eba8ee7e01c499bb233d285df1f82d3df
Oct 30 13:13:47.333790 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:47.333748    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"]
Oct 30 13:13:47.335096 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:13:47.335067    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc60c82_a658_4c5d_9dd4_a4eae3af4846.slice/crio-aedfbd547f5a4f42a6be4ae98e5166ba31ae708a287648754ec9c98ac344240e WatchSource:0}: Error finding container aedfbd547f5a4f42a6be4ae98e5166ba31ae708a287648754ec9c98ac344240e: Status 404 returned error can't find the container with id aedfbd547f5a4f42a6be4ae98e5166ba31ae708a287648754ec9c98ac344240e
Oct 30 13:13:47.338514 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:47.338471    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:13:47.338641 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:47.338566    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:13:47.338641 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:47.338613    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:13:47.464864 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:47.464825    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v" event={"ID":"4fc60c82-a658-4c5d-9dd4-a4eae3af4846","Type":"ContainerStarted","Data":"aedfbd547f5a4f42a6be4ae98e5166ba31ae708a287648754ec9c98ac344240e"}
Oct 30 13:13:47.466737 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:47.466708    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx" event={"ID":"a1a9ed47-9838-43d5-bf54-7133d03c25fd","Type":"ContainerStarted","Data":"208b12cf3a7fd2334e1b489e69ebc79eba8ee7e01c499bb233d285df1f82d3df"}
Oct 30 13:13:47.521376 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:47.521309    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v" podStartSLOduration=1.521291841 podStartE2EDuration="1.521291841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:47.519453847 +0000 UTC m=+1722.566587804" watchObservedRunningTime="2026-10-30 13:13:47.521291841 +0000 UTC m=+1722.568425793"
Oct 30 13:13:47.965546 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:47.965457    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:13:48.476075 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:48.476032    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v" event={"ID":"4fc60c82-a658-4c5d-9dd4-a4eae3af4846","Type":"ContainerStarted","Data":"8eb07ad10148fc577f31de8b2c4715f2668e48fbb32527af0083cbf48d637553"}
Oct 30 13:13:48.477989 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:48.477949    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx" event={"ID":"a1a9ed47-9838-43d5-bf54-7133d03c25fd","Type":"ContainerStarted","Data":"b17317c2ca8a9df6dffe220eedbcd2e2b088e4f662197ea57e7742e183a63f45"}
Oct 30 13:13:48.478135 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:48.478046    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx"
Oct 30 13:13:48.566245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:48.560002    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx" podStartSLOduration=2.181948544 podStartE2EDuration="2.559978261s" totalImagesPullingTime="378.029717ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:48.523767668 +0000 UTC m=+1723.570901622" watchObservedRunningTime="2026-10-30 13:13:48.559978261 +0000 UTC m=+1723.607112217"
Oct 30 13:13:48.857203 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:48.857170    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:48.987196 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:48.987164    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:13:49.486567 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:49.486529    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:13:49.487574 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:49.487552    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:13:50.078267 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:50.078190    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:50.253771 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:50.253726    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:50.751001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:50.750972    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:51.431242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:51.425313    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:51.837533 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:51.817197    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:52.202722 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:52.186703    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"]
Oct 30 13:13:52.202722 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:52.187043    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r" podUID="b56cc33e-52a6-4f72-b0b4-b438e5ea24d6" containerName="istio-proxy" containerID="cri-o://f190536b814f732fbc308b51bd80ac9f38c60d2684e77481bf40df29eec8e6e1" gracePeriod=30
Oct 30 13:13:52.202722 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:52.192286    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:52.434471 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:52.434425    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:52.926888 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:52.926831    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:53.198230 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:53.198078    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:53.624493 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:53.624453    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:54.492466 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:54.492437    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx"
Oct 30 13:13:54.952546 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:54.952510    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:55.311977 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:55.311886    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:55.732330 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:55.732287    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:57.553531 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:57.553461    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:13:57.591024 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:57.590983    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-credential-socket\") pod \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") "
Oct 30 13:13:57.591208 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:57.591063    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istio-podinfo\") pod \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") "
Oct 30 13:13:57.591208 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:57.591118    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istio-token\" (UniqueName: \"kubernetes.io/projected/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istio-token\") pod \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") "
Oct 30 13:13:57.591208 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:57.591180    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-workload-socket\") pod \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") "
Oct 30 13:13:57.591352 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:57.591269    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-workload-certs\") pod \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") "
Oct 30 13:13:57.591396 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:57.591366    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-credential-socket" pod "b56cc33e-52a6-4f72-b0b4-b438e5ea24d6" (UID: "b56cc33e-52a6-4f72-b0b4-b438e5ea24d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:13:57.591452 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:57.591435    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istio-data\") pod \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") "
Oct 30 13:13:57.591519 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:57.591502    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istiod-ca-cert\") pod \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") "
Oct 30 13:13:57.591600 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:57.591540    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-workload-socket" pod "b56cc33e-52a6-4f72-b0b4-b438e5ea24d6" (UID: "b56cc33e-52a6-4f72-b0b4-b438e5ea24d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:13:57.591669 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:57.591586    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-kube-api-access-vpmwk\" (UniqueName: \"kubernetes.io/projected/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-kube-api-access-vpmwk\") pod \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") "
Oct 30 13:13:57.591669 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:57.591599    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-workload-certs" pod "b56cc33e-52a6-4f72-b0b4-b438e5ea24d6" (UID: "b56cc33e-52a6-4f72-b0b4-b438e5ea24d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:13:57.591777 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:57.591676    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istio-data" pod "b56cc33e-52a6-4f72-b0b4-b438e5ea24d6" (UID: "b56cc33e-52a6-4f72-b0b4-b438e5ea24d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:13:57.591777 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:57.591683    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istio-envoy\") pod \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\" (UID: \"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6\") "
Oct 30 13:13:57.592018 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:57.591995    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istiod-ca-cert" pod "b56cc33e-52a6-4f72-b0b4-b438e5ea24d6" (UID: "b56cc33e-52a6-4f72-b0b4-b438e5ea24d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:13:57.592764 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:57.592738    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:13:57.592889 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:57.592773    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:13:57.592889 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:57.592789    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:13:57.592889 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:57.592805    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:13:57.592889 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:57.592819    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:13:57.593989 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:57.593963    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istio-envoy" pod "b56cc33e-52a6-4f72-b0b4-b438e5ea24d6" (UID: "b56cc33e-52a6-4f72-b0b4-b438e5ea24d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:13:57.594091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:57.594066    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-kube-api-access-vpmwk" pod "b56cc33e-52a6-4f72-b0b4-b438e5ea24d6" (UID: "b56cc33e-52a6-4f72-b0b4-b438e5ea24d6"). InnerVolumeSpecName "kube-api-access-vpmwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:13:57.594091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:57.594080    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istio-podinfo" pod "b56cc33e-52a6-4f72-b0b4-b438e5ea24d6" (UID: "b56cc33e-52a6-4f72-b0b4-b438e5ea24d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:13:57.594215 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:57.594195    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istio-token" pod "b56cc33e-52a6-4f72-b0b4-b438e5ea24d6" (UID: "b56cc33e-52a6-4f72-b0b4-b438e5ea24d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:13:57.694779 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:57.694738    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpmwk\" (UniqueName: \"kubernetes.io/projected/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-kube-api-access-vpmwk\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:13:57.694779 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:57.694772    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:13:57.694779 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:57.694782    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:13:57.695116 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:57.694814    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:13:58.075553 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:58.075516    4544 generic.go:356] "Generic (PLEG): container finished" podID="b56cc33e-52a6-4f72-b0b4-b438e5ea24d6" containerID="f190536b814f732fbc308b51bd80ac9f38c60d2684e77481bf40df29eec8e6e1" exitCode=0
Oct 30 13:13:58.075720 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:58.075567    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"
Oct 30 13:13:58.075720 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:58.075602    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r" event={"ID":"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6","Type":"ContainerDied","Data":"f190536b814f732fbc308b51bd80ac9f38c60d2684e77481bf40df29eec8e6e1"}
Oct 30 13:13:58.075720 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:58.075669    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r" event={"ID":"b56cc33e-52a6-4f72-b0b4-b438e5ea24d6","Type":"ContainerDied","Data":"b10d9d0462f62a060b5d188f28fa10ef6a7e9f86c6b8b5c39a137cc376f0a16d"}
Oct 30 13:13:58.075720 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:58.075695    4544 scope.go:122] "RemoveContainer" containerID="f190536b814f732fbc308b51bd80ac9f38c60d2684e77481bf40df29eec8e6e1"
Oct 30 13:13:58.090958 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:58.090935    4544 scope.go:122] "RemoveContainer" containerID="2b81707a031b5c82769d5aee00a697487f3780d697793daf3cee864d856c1008"
Oct 30 13:13:58.102422 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:58.102398    4544 scope.go:122] "RemoveContainer" containerID="f190536b814f732fbc308b51bd80ac9f38c60d2684e77481bf40df29eec8e6e1"
Oct 30 13:13:58.102760 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:13:58.102736    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f190536b814f732fbc308b51bd80ac9f38c60d2684e77481bf40df29eec8e6e1\": container with ID starting with f190536b814f732fbc308b51bd80ac9f38c60d2684e77481bf40df29eec8e6e1 not found: ID does not exist" containerID="f190536b814f732fbc308b51bd80ac9f38c60d2684e77481bf40df29eec8e6e1"
Oct 30 13:13:58.102822 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:58.102767    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f190536b814f732fbc308b51bd80ac9f38c60d2684e77481bf40df29eec8e6e1"} err="failed to get container status \"f190536b814f732fbc308b51bd80ac9f38c60d2684e77481bf40df29eec8e6e1\": rpc error: code = NotFound desc = could not find container \"f190536b814f732fbc308b51bd80ac9f38c60d2684e77481bf40df29eec8e6e1\": container with ID starting with f190536b814f732fbc308b51bd80ac9f38c60d2684e77481bf40df29eec8e6e1 not found: ID does not exist"
Oct 30 13:13:58.102822 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:58.102785    4544 scope.go:122] "RemoveContainer" containerID="2b81707a031b5c82769d5aee00a697487f3780d697793daf3cee864d856c1008"
Oct 30 13:13:58.103100 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:13:58.103072    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b81707a031b5c82769d5aee00a697487f3780d697793daf3cee864d856c1008\": container with ID starting with 2b81707a031b5c82769d5aee00a697487f3780d697793daf3cee864d856c1008 not found: ID does not exist" containerID="2b81707a031b5c82769d5aee00a697487f3780d697793daf3cee864d856c1008"
Oct 30 13:13:58.103157 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:58.103110    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b81707a031b5c82769d5aee00a697487f3780d697793daf3cee864d856c1008"} err="failed to get container status \"2b81707a031b5c82769d5aee00a697487f3780d697793daf3cee864d856c1008\": rpc error: code = NotFound desc = could not find container \"2b81707a031b5c82769d5aee00a697487f3780d697793daf3cee864d856c1008\": container with ID starting with 2b81707a031b5c82769d5aee00a697487f3780d697793daf3cee864d856c1008 not found: ID does not exist"
Oct 30 13:13:58.150895 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:58.150857    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"]
Oct 30 13:13:58.157481 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:58.157433    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r"]
Oct 30 13:13:58.468379 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:58.468297    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:58.768571 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:58.768487    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:13:59.010202 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:13:59.010165    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:13:59.090062 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:13:59.090019    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b56cc33e-52a6-4f72-b0b4-b438e5ea24d6" path="/var/lib/kubelet/pods/b56cc33e-52a6-4f72-b0b4-b438e5ea24d6/volumes"
Oct 30 13:14:00.463125 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:00.463045    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:02.538406 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:02.538196    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:04.111843 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:04.111803    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc"]
Oct 30 13:14:04.146477 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:04.146440    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b"]
Oct 30 13:14:04.306122 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:04.306082    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:04.493288 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:04.492389    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:04.732030 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:04.731935    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:05.115837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:05.115794    4544 patch_prober.go:29] interesting pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 13:14:05.116376 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:05.115874    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6" podUID="476505c3-3236-4789-846f-c13caa710b36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 13:14:05.164779 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:05.164676    4544 generic.go:356] "Generic (PLEG): container finished" podID="476505c3-3236-4789-846f-c13caa710b36" containerID="56400f2249f611470566cc3ca65eee52ef00ac4e02853170e613fd42d51eb337" exitCode=0
Oct 30 13:14:05.164960 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:05.164741    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6" event={"ID":"476505c3-3236-4789-846f-c13caa710b36","Type":"ContainerDied","Data":"56400f2249f611470566cc3ca65eee52ef00ac4e02853170e613fd42d51eb337"}
Oct 30 13:14:05.169304 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:05.165953    4544 scope.go:122] "RemoveContainer" containerID="56400f2249f611470566cc3ca65eee52ef00ac4e02853170e613fd42d51eb337"
Oct 30 13:14:05.181097 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:05.181048    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:14:05.181319 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:05.181144    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:14:05.181319 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:05.181182    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:14:06.178142 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:06.178106    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6" event={"ID":"476505c3-3236-4789-846f-c13caa710b36","Type":"ContainerStarted","Data":"b0b7fe58678bf51c96306fd7784d2a86c1e89d59d71c9f0c54901f3be76c79f4"}
Oct 30 13:14:06.389981 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:06.389936    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:14:06.390288 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:06.390264    4544 patch_prober.go:29] interesting pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 13:14:06.390350 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:06.390330    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6" podUID="476505c3-3236-4789-846f-c13caa710b36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 13:14:06.593646 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:06.593607    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:07.390692 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:07.390655    4544 patch_prober.go:29] interesting pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 13:14:07.391101 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:07.390712    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6" podUID="476505c3-3236-4789-846f-c13caa710b36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 13:14:08.390033 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:08.389993    4544 patch_prober.go:29] interesting pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 13:14:08.390256 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:08.390069    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6" podUID="476505c3-3236-4789-846f-c13caa710b36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 13:14:08.449435 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:08.449398    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:08.670866 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:08.670770    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:09.112319 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:09.112274    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc" podUID="97315c62-dbfc-4e18-aca2-34ff9e6d2a75" containerName="payload-processing" containerID="cri-o://3f0cc60569923437bc0eb61280c122ed608cfe2170074765925d202265dab217" gracePeriod=25
Oct 30 13:14:09.147307 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:09.147253    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b" podUID="b97f8cd4-df63-413d-8955-bfd05fcf2ac8" containerName="payload-pre-processing" containerID="cri-o://f987e9d33d927333f9d83f889ec8b06b9b4f5c2b9e3bebac763dcf6e025859d4" gracePeriod=25
Oct 30 13:14:09.394024 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:09.393985    4544 patch_prober.go:29] interesting pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 13:14:09.394187 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:09.394065    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6" podUID="476505c3-3236-4789-846f-c13caa710b36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 13:14:09.593711 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:09.593673    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc"
Oct 30 13:14:09.677891 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:09.677854    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b"
Oct 30 13:14:09.699365 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:09.699334    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97315c62-dbfc-4e18-aca2-34ff9e6d2a75-kube-api-access-bmq88\" (UniqueName: \"kubernetes.io/projected/97315c62-dbfc-4e18-aca2-34ff9e6d2a75-kube-api-access-bmq88\") pod \"97315c62-dbfc-4e18-aca2-34ff9e6d2a75\" (UID: \"97315c62-dbfc-4e18-aca2-34ff9e6d2a75\") "
Oct 30 13:14:09.699707 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:09.699683    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97315c62-dbfc-4e18-aca2-34ff9e6d2a75-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97315c62-dbfc-4e18-aca2-34ff9e6d2a75-plugins-config-volume\") pod \"97315c62-dbfc-4e18-aca2-34ff9e6d2a75\" (UID: \"97315c62-dbfc-4e18-aca2-34ff9e6d2a75\") "
Oct 30 13:14:09.700094 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:09.700068    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97315c62-dbfc-4e18-aca2-34ff9e6d2a75-plugins-config-volume" pod "97315c62-dbfc-4e18-aca2-34ff9e6d2a75" (UID: "97315c62-dbfc-4e18-aca2-34ff9e6d2a75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:14:09.700382 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:09.700365    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97315c62-dbfc-4e18-aca2-34ff9e6d2a75-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:14:09.702382 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:09.702355    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97315c62-dbfc-4e18-aca2-34ff9e6d2a75-kube-api-access-bmq88" pod "97315c62-dbfc-4e18-aca2-34ff9e6d2a75" (UID: "97315c62-dbfc-4e18-aca2-34ff9e6d2a75"). InnerVolumeSpecName "kube-api-access-bmq88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:14:09.819697 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:09.806734    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b97f8cd4-df63-413d-8955-bfd05fcf2ac8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b97f8cd4-df63-413d-8955-bfd05fcf2ac8-plugins-config-volume\") pod \"b97f8cd4-df63-413d-8955-bfd05fcf2ac8\" (UID: \"b97f8cd4-df63-413d-8955-bfd05fcf2ac8\") "
Oct 30 13:14:09.819697 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:09.806982    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b97f8cd4-df63-413d-8955-bfd05fcf2ac8-kube-api-access-phcqq\" (UniqueName: \"kubernetes.io/projected/b97f8cd4-df63-413d-8955-bfd05fcf2ac8-kube-api-access-phcqq\") pod \"b97f8cd4-df63-413d-8955-bfd05fcf2ac8\" (UID: \"b97f8cd4-df63-413d-8955-bfd05fcf2ac8\") "
Oct 30 13:14:09.819697 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:09.808003    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmq88\" (UniqueName: \"kubernetes.io/projected/97315c62-dbfc-4e18-aca2-34ff9e6d2a75-kube-api-access-bmq88\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:14:09.819697 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:09.808996    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b97f8cd4-df63-413d-8955-bfd05fcf2ac8-plugins-config-volume" pod "b97f8cd4-df63-413d-8955-bfd05fcf2ac8" (UID: "b97f8cd4-df63-413d-8955-bfd05fcf2ac8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:14:09.897325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:09.896465    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b97f8cd4-df63-413d-8955-bfd05fcf2ac8-kube-api-access-phcqq" pod "b97f8cd4-df63-413d-8955-bfd05fcf2ac8" (UID: "b97f8cd4-df63-413d-8955-bfd05fcf2ac8"). InnerVolumeSpecName "kube-api-access-phcqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:14:09.912289 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:09.912114    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phcqq\" (UniqueName: \"kubernetes.io/projected/b97f8cd4-df63-413d-8955-bfd05fcf2ac8-kube-api-access-phcqq\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:14:09.912289 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:09.912155    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b97f8cd4-df63-413d-8955-bfd05fcf2ac8-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:14:10.224247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:10.221018    4544 generic.go:356] "Generic (PLEG): container finished" podID="97315c62-dbfc-4e18-aca2-34ff9e6d2a75" containerID="3f0cc60569923437bc0eb61280c122ed608cfe2170074765925d202265dab217" exitCode=0
Oct 30 13:14:10.224247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:10.221279    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc" event={"ID":"97315c62-dbfc-4e18-aca2-34ff9e6d2a75","Type":"ContainerDied","Data":"3f0cc60569923437bc0eb61280c122ed608cfe2170074765925d202265dab217"}
Oct 30 13:14:10.224247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:10.221348    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc" event={"ID":"97315c62-dbfc-4e18-aca2-34ff9e6d2a75","Type":"ContainerDied","Data":"e9b38ad4b0792e92e5da50b3a8ba1923feb83ca79807323e3d26597c0214364c"}
Oct 30 13:14:10.224247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:10.221387    4544 scope.go:122] "RemoveContainer" containerID="3f0cc60569923437bc0eb61280c122ed608cfe2170074765925d202265dab217"
Oct 30 13:14:10.224247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:10.221574    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc"
Oct 30 13:14:10.242806 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:10.242766    4544 generic.go:356] "Generic (PLEG): container finished" podID="b97f8cd4-df63-413d-8955-bfd05fcf2ac8" containerID="f987e9d33d927333f9d83f889ec8b06b9b4f5c2b9e3bebac763dcf6e025859d4" exitCode=0
Oct 30 13:14:10.242960 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:10.242881    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b" event={"ID":"b97f8cd4-df63-413d-8955-bfd05fcf2ac8","Type":"ContainerDied","Data":"f987e9d33d927333f9d83f889ec8b06b9b4f5c2b9e3bebac763dcf6e025859d4"}
Oct 30 13:14:10.243002 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:10.242962    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b" event={"ID":"b97f8cd4-df63-413d-8955-bfd05fcf2ac8","Type":"ContainerDied","Data":"f9579c330811e3cf6f8646ecd19db2338fea5ed57e26da47e8fd5d7441d917f0"}
Oct 30 13:14:10.243053 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:10.243039    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b"
Oct 30 13:14:10.253373 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:10.253351    4544 scope.go:122] "RemoveContainer" containerID="3f0cc60569923437bc0eb61280c122ed608cfe2170074765925d202265dab217"
Oct 30 13:14:10.253714 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:14:10.253693    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f0cc60569923437bc0eb61280c122ed608cfe2170074765925d202265dab217\": container with ID starting with 3f0cc60569923437bc0eb61280c122ed608cfe2170074765925d202265dab217 not found: ID does not exist" containerID="3f0cc60569923437bc0eb61280c122ed608cfe2170074765925d202265dab217"
Oct 30 13:14:10.253803 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:10.253728    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f0cc60569923437bc0eb61280c122ed608cfe2170074765925d202265dab217"} err="failed to get container status \"3f0cc60569923437bc0eb61280c122ed608cfe2170074765925d202265dab217\": rpc error: code = NotFound desc = could not find container \"3f0cc60569923437bc0eb61280c122ed608cfe2170074765925d202265dab217\": container with ID starting with 3f0cc60569923437bc0eb61280c122ed608cfe2170074765925d202265dab217 not found: ID does not exist"
Oct 30 13:14:10.253803 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:10.253755    4544 scope.go:122] "RemoveContainer" containerID="f987e9d33d927333f9d83f889ec8b06b9b4f5c2b9e3bebac763dcf6e025859d4"
Oct 30 13:14:10.264904 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:10.264882    4544 scope.go:122] "RemoveContainer" containerID="f987e9d33d927333f9d83f889ec8b06b9b4f5c2b9e3bebac763dcf6e025859d4"
Oct 30 13:14:10.265316 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:14:10.265286    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f987e9d33d927333f9d83f889ec8b06b9b4f5c2b9e3bebac763dcf6e025859d4\": container with ID starting with f987e9d33d927333f9d83f889ec8b06b9b4f5c2b9e3bebac763dcf6e025859d4 not found: ID does not exist" containerID="f987e9d33d927333f9d83f889ec8b06b9b4f5c2b9e3bebac763dcf6e025859d4"
Oct 30 13:14:10.265420 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:10.265329    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f987e9d33d927333f9d83f889ec8b06b9b4f5c2b9e3bebac763dcf6e025859d4"} err="failed to get container status \"f987e9d33d927333f9d83f889ec8b06b9b4f5c2b9e3bebac763dcf6e025859d4\": rpc error: code = NotFound desc = could not find container \"f987e9d33d927333f9d83f889ec8b06b9b4f5c2b9e3bebac763dcf6e025859d4\": container with ID starting with f987e9d33d927333f9d83f889ec8b06b9b4f5c2b9e3bebac763dcf6e025859d4 not found: ID does not exist"
Oct 30 13:14:10.371319 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:10.371283    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b"]
Oct 30 13:14:10.386817 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:10.386740    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b"]
Oct 30 13:14:10.391682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:10.391641    4544 patch_prober.go:29] interesting pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 13:14:10.391926 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:10.391712    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6" podUID="476505c3-3236-4789-846f-c13caa710b36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 13:14:10.405541 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:10.405496    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc"]
Oct 30 13:14:10.410145 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:10.410062    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc"]
Oct 30 13:14:10.688570 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:10.688479    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:11.094142 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:11.094106    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97315c62-dbfc-4e18-aca2-34ff9e6d2a75" path="/var/lib/kubelet/pods/97315c62-dbfc-4e18-aca2-34ff9e6d2a75/volumes"
Oct 30 13:14:11.094660 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:11.094641    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b97f8cd4-df63-413d-8955-bfd05fcf2ac8" path="/var/lib/kubelet/pods/b97f8cd4-df63-413d-8955-bfd05fcf2ac8/volumes"
Oct 30 13:14:11.390702 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:11.390611    4544 patch_prober.go:29] interesting pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 13:14:11.390702 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:11.390667    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6" podUID="476505c3-3236-4789-846f-c13caa710b36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 13:14:12.396324 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:12.396291    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:14:12.396816 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:12.396799    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:14:12.397563 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:12.397546    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:14:12.709006 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:12.706815    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:14.609571 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:14.609532    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:16.320860 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:16.320821    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:16.456781 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:16.456750    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:16.607564 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:16.607525    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:18.585309 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:18.585271    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:19.031282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:19.031157    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22cb3314-jt7gp"]
Oct 30 13:14:19.042320 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:19.042286    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22cb3314-jt7gp"]
Oct 30 13:14:19.092667 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:19.092632    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad9b8e49-10ba-43ae-a17e-20ee84fef5bc" path="/var/lib/kubelet/pods/ad9b8e49-10ba-43ae-a17e-20ee84fef5bc/volumes"
Oct 30 13:14:20.487148 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:20.487112    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:22.703796 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:22.703762    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:24.437014 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:24.436978    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:24.546361 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:24.546330    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:24.916394 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:24.916346    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:25.514246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:25.504018    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:25.717243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:25.716959    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:26.726929 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:26.726864    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:29.008697 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:14:29.008657    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:14:29.370137 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:29.370102    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:30.425667 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:30.425632    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:30.616527 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:30.616491    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:31.493614 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:31.493532    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:32.514322 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:32.514290    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:32.603699 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:32.603659    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w"]
Oct 30 13:14:32.604973 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:32.604939    4544 cpu_manager.go:414] "containerMap: removing container" podUID="b97f8cd4-df63-413d-8955-bfd05fcf2ac8" containerName="payload-pre-processing"
Oct 30 13:14:32.604973 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:32.604967    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b97f8cd4-df63-413d-8955-bfd05fcf2ac8" containerName="payload-pre-processing"
Oct 30 13:14:32.605129 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:32.604995    4544 cpu_manager.go:414] "containerMap: removing container" podUID="b56cc33e-52a6-4f72-b0b4-b438e5ea24d6" containerName="istio-proxy"
Oct 30 13:14:32.605129 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:32.605004    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b56cc33e-52a6-4f72-b0b4-b438e5ea24d6" containerName="istio-proxy"
Oct 30 13:14:32.605129 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:32.605093    4544 cpu_manager.go:414] "containerMap: removing container" podUID="97315c62-dbfc-4e18-aca2-34ff9e6d2a75" containerName="payload-processing"
Oct 30 13:14:32.605129 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:32.605103    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97315c62-dbfc-4e18-aca2-34ff9e6d2a75" containerName="payload-processing"
Oct 30 13:14:32.605368 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:32.605351    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="b97f8cd4-df63-413d-8955-bfd05fcf2ac8" containerName="payload-pre-processing"
Oct 30 13:14:32.605408 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:32.605375    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="b56cc33e-52a6-4f72-b0b4-b438e5ea24d6" containerName="istio-proxy"
Oct 30 13:14:32.605408 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:32.605390    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="b56cc33e-52a6-4f72-b0b4-b438e5ea24d6" containerName="istio-proxy"
Oct 30 13:14:32.605473 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:32.605413    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="97315c62-dbfc-4e18-aca2-34ff9e6d2a75" containerName="payload-processing"
Oct 30 13:14:32.610053 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:32.610034    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w"
Oct 30 13:14:32.631557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:32.631521    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-649303-dockercfg-8prrk\""
Oct 30 13:14:32.632523 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:32.632492    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w"]
Oct 30 13:14:32.642272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:32.639031    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-649303\""
Oct 30 13:14:32.642272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:32.641013    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dfe63f2-8812-4b0b-a4e2-e0684d7ba985-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w\" (UID: \"8dfe63f2-8812-4b0b-a4e2-e0684d7ba985\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w"
Oct 30 13:14:32.642272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:32.641083    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jgkg\" (UniqueName: \"kubernetes.io/projected/8dfe63f2-8812-4b0b-a4e2-e0684d7ba985-kube-api-access-5jgkg\") pod \"payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w\" (UID: \"8dfe63f2-8812-4b0b-a4e2-e0684d7ba985\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w"
Oct 30 13:14:32.677290 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:32.677196    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:32.678403 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:32.678372    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-649303-5c6ff5544-429vh"]
Oct 30 13:14:32.680079 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:32.680052    4544 cpu_manager.go:414] "containerMap: removing container" podUID="b56cc33e-52a6-4f72-b0b4-b438e5ea24d6" containerName="istio-proxy"
Oct 30 13:14:32.682287 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:32.682264    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b56cc33e-52a6-4f72-b0b4-b438e5ea24d6" containerName="istio-proxy"
Oct 30 13:14:32.868242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:32.842978    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jgkg\" (UniqueName: \"kubernetes.io/projected/8dfe63f2-8812-4b0b-a4e2-e0684d7ba985-kube-api-access-5jgkg\") pod \"payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w\" (UID: \"8dfe63f2-8812-4b0b-a4e2-e0684d7ba985\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w"
Oct 30 13:14:32.869844 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:32.869795    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dfe63f2-8812-4b0b-a4e2-e0684d7ba985-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w\" (UID: \"8dfe63f2-8812-4b0b-a4e2-e0684d7ba985\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w"
Oct 30 13:14:32.885244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:32.870981    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dfe63f2-8812-4b0b-a4e2-e0684d7ba985-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w\" (UID: \"8dfe63f2-8812-4b0b-a4e2-e0684d7ba985\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w"
Oct 30 13:14:33.014247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:32.997483    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-649303-5c6ff5544-429vh"]
Oct 30 13:14:33.036285 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:33.028575    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-649303-5c6ff5544-429vh"
Oct 30 13:14:33.051596 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:33.039324    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jgkg\" (UniqueName: \"kubernetes.io/projected/8dfe63f2-8812-4b0b-a4e2-e0684d7ba985-kube-api-access-5jgkg\") pod \"payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w\" (UID: \"8dfe63f2-8812-4b0b-a4e2-e0684d7ba985\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w"
Oct 30 13:14:33.132734 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:33.132643    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh2sf\" (UniqueName: \"kubernetes.io/projected/cdc909e9-a85b-4c23-8218-c0a982b13829-kube-api-access-lh2sf\") pod \"payload-processing-e2e-inf-b-649303-5c6ff5544-429vh\" (UID: \"cdc909e9-a85b-4c23-8218-c0a982b13829\") " pod="openshift-ingress/payload-processing-e2e-inf-b-649303-5c6ff5544-429vh"
Oct 30 13:14:33.132734 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:33.132699    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc909e9-a85b-4c23-8218-c0a982b13829-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-649303-5c6ff5544-429vh\" (UID: \"cdc909e9-a85b-4c23-8218-c0a982b13829\") " pod="openshift-ingress/payload-processing-e2e-inf-b-649303-5c6ff5544-429vh"
Oct 30 13:14:33.226698 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:33.226661    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w"
Oct 30 13:14:33.234920 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:33.234881    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh2sf\" (UniqueName: \"kubernetes.io/projected/cdc909e9-a85b-4c23-8218-c0a982b13829-kube-api-access-lh2sf\") pod \"payload-processing-e2e-inf-b-649303-5c6ff5544-429vh\" (UID: \"cdc909e9-a85b-4c23-8218-c0a982b13829\") " pod="openshift-ingress/payload-processing-e2e-inf-b-649303-5c6ff5544-429vh"
Oct 30 13:14:33.235059 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:33.234948    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc909e9-a85b-4c23-8218-c0a982b13829-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-649303-5c6ff5544-429vh\" (UID: \"cdc909e9-a85b-4c23-8218-c0a982b13829\") " pod="openshift-ingress/payload-processing-e2e-inf-b-649303-5c6ff5544-429vh"
Oct 30 13:14:33.235689 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:33.235665    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc909e9-a85b-4c23-8218-c0a982b13829-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-649303-5c6ff5544-429vh\" (UID: \"cdc909e9-a85b-4c23-8218-c0a982b13829\") " pod="openshift-ingress/payload-processing-e2e-inf-b-649303-5c6ff5544-429vh"
Oct 30 13:14:33.259094 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:33.259053    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh2sf\" (UniqueName: \"kubernetes.io/projected/cdc909e9-a85b-4c23-8218-c0a982b13829-kube-api-access-lh2sf\") pod \"payload-processing-e2e-inf-b-649303-5c6ff5544-429vh\" (UID: \"cdc909e9-a85b-4c23-8218-c0a982b13829\") " pod="openshift-ingress/payload-processing-e2e-inf-b-649303-5c6ff5544-429vh"
Oct 30 13:14:33.373492 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:33.373460    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-649303-5c6ff5544-429vh"
Oct 30 13:14:33.549007 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:14:33.548976    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dfe63f2_8812_4b0b_a4e2_e0684d7ba985.slice/crio-c127bab9c40ea13b9fdf4e0c973adca03b9bb7b87e6f45ef0cb0238d808da41c WatchSource:0}: Error finding container c127bab9c40ea13b9fdf4e0c973adca03b9bb7b87e6f45ef0cb0238d808da41c: Status 404 returned error can't find the container with id c127bab9c40ea13b9fdf4e0c973adca03b9bb7b87e6f45ef0cb0238d808da41c
Oct 30 13:14:33.549765 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:33.549739    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w"]
Oct 30 13:14:33.693805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:33.693773    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-649303-5c6ff5544-429vh"]
Oct 30 13:14:33.697416 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:14:33.697368    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc909e9_a85b_4c23_8218_c0a982b13829.slice/crio-6bfdfe15370f68e709f6ad288b906fa7dd406252e7b2320f31f3e8264be46758 WatchSource:0}: Error finding container 6bfdfe15370f68e709f6ad288b906fa7dd406252e7b2320f31f3e8264be46758: Status 404 returned error can't find the container with id 6bfdfe15370f68e709f6ad288b906fa7dd406252e7b2320f31f3e8264be46758
Oct 30 13:14:34.179282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:34.179239    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:34.527528 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:34.527415    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w" event={"ID":"8dfe63f2-8812-4b0b-a4e2-e0684d7ba985","Type":"ContainerStarted","Data":"e0f9c9172ea33a197ff4e79bbc6e634ae2ebaaa34a8d8e6e119c9f7bdf6c0e66"}
Oct 30 13:14:34.527528 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:34.527524    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w"
Oct 30 13:14:34.528290 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:34.527552    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w" event={"ID":"8dfe63f2-8812-4b0b-a4e2-e0684d7ba985","Type":"ContainerStarted","Data":"c127bab9c40ea13b9fdf4e0c973adca03b9bb7b87e6f45ef0cb0238d808da41c"}
Oct 30 13:14:34.529573 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:34.529540    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-649303-5c6ff5544-429vh" event={"ID":"cdc909e9-a85b-4c23-8218-c0a982b13829","Type":"ContainerStarted","Data":"7acb3bd725ca5bcb002422ff88669751cac7784c19650ecf692cbb5cd23fb23e"}
Oct 30 13:14:34.529686 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:34.529590    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-649303-5c6ff5544-429vh" event={"ID":"cdc909e9-a85b-4c23-8218-c0a982b13829","Type":"ContainerStarted","Data":"6bfdfe15370f68e709f6ad288b906fa7dd406252e7b2320f31f3e8264be46758"}
Oct 30 13:14:34.529686 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:34.529612    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-649303-5c6ff5544-429vh"
Oct 30 13:14:34.575769 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:34.575262    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w" podStartSLOduration=2.575241076 podStartE2EDuration="2.575241076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:34.571962603 +0000 UTC m=+1769.619096557" watchObservedRunningTime="2026-10-30 13:14:34.575241076 +0000 UTC m=+1769.622375027"
Oct 30 13:14:34.637552 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:34.637495    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-649303-5c6ff5544-429vh" podStartSLOduration=2.637479582 podStartE2EDuration="2.637479582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:34.633060986 +0000 UTC m=+1769.680194942" watchObservedRunningTime="2026-10-30 13:14:34.637479582 +0000 UTC m=+1769.684613535"
Oct 30 13:14:36.227824 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:36.227785    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:36.756844 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:36.756808    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:37.789735 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:37.789684    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:38.789758 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:38.789705    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:41.245357 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:41.245314    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:41.408837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:41.408758    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:43.489475 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:43.489438    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:43.885250 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:43.885195    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:45.541092 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:45.541058    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-649303-5c6ff5544-429vh"
Oct 30 13:14:45.541540 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:45.541335    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w"
Oct 30 13:14:45.829412 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:45.829381    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:45.987202 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:45.987166    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:46.220973 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:46.220825    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:46.239593 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:46.239557    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"]
Oct 30 13:14:46.239871 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:46.239842    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d" podUID="48119de6-b81b-4f07-b304-a33e16ab2d68" containerName="istio-proxy" containerID="cri-o://3da43ddb550a515febd09ab873c92ec7751ad065f0f75cc82d5ddb90f011fe82" gracePeriod=30
Oct 30 13:14:46.590246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:46.584800    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:48.649772 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:48.649735    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:50.673266 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:50.666968    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:51.548444 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.548412    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:14:51.600130 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.600093    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48119de6-b81b-4f07-b304-a33e16ab2d68-istio-token\" (UniqueName: \"kubernetes.io/projected/48119de6-b81b-4f07-b304-a33e16ab2d68-istio-token\") pod \"48119de6-b81b-4f07-b304-a33e16ab2d68\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") "
Oct 30 13:14:51.600335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.600177    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48119de6-b81b-4f07-b304-a33e16ab2d68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48119de6-b81b-4f07-b304-a33e16ab2d68-istio-podinfo\") pod \"48119de6-b81b-4f07-b304-a33e16ab2d68\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") "
Oct 30 13:14:51.600335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.600202    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-credential-socket\") pod \"48119de6-b81b-4f07-b304-a33e16ab2d68\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") "
Oct 30 13:14:51.600335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.600254    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48119de6-b81b-4f07-b304-a33e16ab2d68-kube-api-access-r84s7\" (UniqueName: \"kubernetes.io/projected/48119de6-b81b-4f07-b304-a33e16ab2d68-kube-api-access-r84s7\") pod \"48119de6-b81b-4f07-b304-a33e16ab2d68\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") "
Oct 30 13:14:51.600500 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.600487    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-workload-socket\") pod \"48119de6-b81b-4f07-b304-a33e16ab2d68\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") "
Oct 30 13:14:51.600556 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.600544    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-istio-data\") pod \"48119de6-b81b-4f07-b304-a33e16ab2d68\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") "
Oct 30 13:14:51.600626 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.600582    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-credential-socket" pod "48119de6-b81b-4f07-b304-a33e16ab2d68" (UID: "48119de6-b81b-4f07-b304-a33e16ab2d68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:51.600626 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.600599    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-istio-envoy\") pod \"48119de6-b81b-4f07-b304-a33e16ab2d68\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") "
Oct 30 13:14:51.600768 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.600740    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-workload-socket" pod "48119de6-b81b-4f07-b304-a33e16ab2d68" (UID: "48119de6-b81b-4f07-b304-a33e16ab2d68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:51.600840 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.600749    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48119de6-b81b-4f07-b304-a33e16ab2d68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48119de6-b81b-4f07-b304-a33e16ab2d68-istiod-ca-cert\") pod \"48119de6-b81b-4f07-b304-a33e16ab2d68\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") "
Oct 30 13:14:51.600932 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.600907    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-istio-data" pod "48119de6-b81b-4f07-b304-a33e16ab2d68" (UID: "48119de6-b81b-4f07-b304-a33e16ab2d68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:51.600932 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.600919    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-workload-certs\") pod \"48119de6-b81b-4f07-b304-a33e16ab2d68\" (UID: \"48119de6-b81b-4f07-b304-a33e16ab2d68\") "
Oct 30 13:14:51.601096 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.601072    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48119de6-b81b-4f07-b304-a33e16ab2d68-istiod-ca-cert" pod "48119de6-b81b-4f07-b304-a33e16ab2d68" (UID: "48119de6-b81b-4f07-b304-a33e16ab2d68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:14:51.601273 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.601246    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-workload-certs" pod "48119de6-b81b-4f07-b304-a33e16ab2d68" (UID: "48119de6-b81b-4f07-b304-a33e16ab2d68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:51.601929 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.601902    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:14:51.602033 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.601932    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:14:51.602033 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.601947    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48119de6-b81b-4f07-b304-a33e16ab2d68-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:14:51.602033 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.601962    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:14:51.602033 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.601974    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:14:51.602718 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.602694    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48119de6-b81b-4f07-b304-a33e16ab2d68-istio-podinfo" pod "48119de6-b81b-4f07-b304-a33e16ab2d68" (UID: "48119de6-b81b-4f07-b304-a33e16ab2d68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:14:51.602805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.602755    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-istio-envoy" pod "48119de6-b81b-4f07-b304-a33e16ab2d68" (UID: "48119de6-b81b-4f07-b304-a33e16ab2d68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:51.602865 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.602842    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48119de6-b81b-4f07-b304-a33e16ab2d68-istio-token" pod "48119de6-b81b-4f07-b304-a33e16ab2d68" (UID: "48119de6-b81b-4f07-b304-a33e16ab2d68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:14:51.602910 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.602867    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48119de6-b81b-4f07-b304-a33e16ab2d68-kube-api-access-r84s7" pod "48119de6-b81b-4f07-b304-a33e16ab2d68" (UID: "48119de6-b81b-4f07-b304-a33e16ab2d68"). InnerVolumeSpecName "kube-api-access-r84s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:14:51.703670 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.703627    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48119de6-b81b-4f07-b304-a33e16ab2d68-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:14:51.703670 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.703660    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48119de6-b81b-4f07-b304-a33e16ab2d68-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:14:51.703670 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.703674    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r84s7\" (UniqueName: \"kubernetes.io/projected/48119de6-b81b-4f07-b304-a33e16ab2d68-kube-api-access-r84s7\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:14:51.704130 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.703688    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48119de6-b81b-4f07-b304-a33e16ab2d68-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:14:51.805658 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.805560    4544 generic.go:356] "Generic (PLEG): container finished" podID="48119de6-b81b-4f07-b304-a33e16ab2d68" containerID="3da43ddb550a515febd09ab873c92ec7751ad065f0f75cc82d5ddb90f011fe82" exitCode=0
Oct 30 13:14:51.805658 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.805628    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d" event={"ID":"48119de6-b81b-4f07-b304-a33e16ab2d68","Type":"ContainerDied","Data":"3da43ddb550a515febd09ab873c92ec7751ad065f0f75cc82d5ddb90f011fe82"}
Oct 30 13:14:51.805884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.805665    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"
Oct 30 13:14:51.805884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.805706    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d" event={"ID":"48119de6-b81b-4f07-b304-a33e16ab2d68","Type":"ContainerDied","Data":"d40973bc0792a36a5e9b66b36871ee77900c97dd52fb82192e27174eb40352bd"}
Oct 30 13:14:51.805884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.805729    4544 scope.go:122] "RemoveContainer" containerID="3da43ddb550a515febd09ab873c92ec7751ad065f0f75cc82d5ddb90f011fe82"
Oct 30 13:14:51.818151 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.818132    4544 scope.go:122] "RemoveContainer" containerID="0ce2a491e46e690ccbac4fa6554b14a117fa4a58145487fe1e086774fa4f2492"
Oct 30 13:14:51.829303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.829276    4544 scope.go:122] "RemoveContainer" containerID="3da43ddb550a515febd09ab873c92ec7751ad065f0f75cc82d5ddb90f011fe82"
Oct 30 13:14:51.829740 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:14:51.829714    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3da43ddb550a515febd09ab873c92ec7751ad065f0f75cc82d5ddb90f011fe82\": container with ID starting with 3da43ddb550a515febd09ab873c92ec7751ad065f0f75cc82d5ddb90f011fe82 not found: ID does not exist" containerID="3da43ddb550a515febd09ab873c92ec7751ad065f0f75cc82d5ddb90f011fe82"
Oct 30 13:14:51.829867 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.829754    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3da43ddb550a515febd09ab873c92ec7751ad065f0f75cc82d5ddb90f011fe82"} err="failed to get container status \"3da43ddb550a515febd09ab873c92ec7751ad065f0f75cc82d5ddb90f011fe82\": rpc error: code = NotFound desc = could not find container \"3da43ddb550a515febd09ab873c92ec7751ad065f0f75cc82d5ddb90f011fe82\": container with ID starting with 3da43ddb550a515febd09ab873c92ec7751ad065f0f75cc82d5ddb90f011fe82 not found: ID does not exist"
Oct 30 13:14:51.829867 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.829781    4544 scope.go:122] "RemoveContainer" containerID="0ce2a491e46e690ccbac4fa6554b14a117fa4a58145487fe1e086774fa4f2492"
Oct 30 13:14:51.830123 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:14:51.830100    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ce2a491e46e690ccbac4fa6554b14a117fa4a58145487fe1e086774fa4f2492\": container with ID starting with 0ce2a491e46e690ccbac4fa6554b14a117fa4a58145487fe1e086774fa4f2492 not found: ID does not exist" containerID="0ce2a491e46e690ccbac4fa6554b14a117fa4a58145487fe1e086774fa4f2492"
Oct 30 13:14:51.830160 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.830134    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ce2a491e46e690ccbac4fa6554b14a117fa4a58145487fe1e086774fa4f2492"} err="failed to get container status \"0ce2a491e46e690ccbac4fa6554b14a117fa4a58145487fe1e086774fa4f2492\": rpc error: code = NotFound desc = could not find container \"0ce2a491e46e690ccbac4fa6554b14a117fa4a58145487fe1e086774fa4f2492\": container with ID starting with 0ce2a491e46e690ccbac4fa6554b14a117fa4a58145487fe1e086774fa4f2492 not found: ID does not exist"
Oct 30 13:14:51.907393 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.907350    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"]
Oct 30 13:14:51.929749 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:51.929716    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d"]
Oct 30 13:14:52.509686 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:52.509653    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:53.090302 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:53.090258    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48119de6-b81b-4f07-b304-a33e16ab2d68" path="/var/lib/kubelet/pods/48119de6-b81b-4f07-b304-a33e16ab2d68/volumes"
Oct 30 13:14:53.967404 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:53.967368    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7"]
Oct 30 13:14:53.967665 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:53.967636    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7" podUID="75449fe5-8d9e-4c97-97b3-59929a10f0d5" containerName="maas-api" containerID="cri-o://0686d74ee925184433845cfebf937318f1c649ceff7c61869696c6ac5d3833e2" gracePeriod=30
Oct 30 13:14:54.043149 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:54.043062    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:54.568617 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:54.567419    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7"
Oct 30 13:14:54.735369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:54.731873    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75449fe5-8d9e-4c97-97b3-59929a10f0d5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/75449fe5-8d9e-4c97-97b3-59929a10f0d5-metrics-certs\") pod \"75449fe5-8d9e-4c97-97b3-59929a10f0d5\" (UID: \"75449fe5-8d9e-4c97-97b3-59929a10f0d5\") "
Oct 30 13:14:54.735369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:54.732022    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75449fe5-8d9e-4c97-97b3-59929a10f0d5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75449fe5-8d9e-4c97-97b3-59929a10f0d5-openshift-service-ca\") pod \"75449fe5-8d9e-4c97-97b3-59929a10f0d5\" (UID: \"75449fe5-8d9e-4c97-97b3-59929a10f0d5\") "
Oct 30 13:14:54.735369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:54.732133    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75449fe5-8d9e-4c97-97b3-59929a10f0d5-kube-api-access-frrgk\" (UniqueName: \"kubernetes.io/projected/75449fe5-8d9e-4c97-97b3-59929a10f0d5-kube-api-access-frrgk\") pod \"75449fe5-8d9e-4c97-97b3-59929a10f0d5\" (UID: \"75449fe5-8d9e-4c97-97b3-59929a10f0d5\") "
Oct 30 13:14:54.735369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:54.732170    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75449fe5-8d9e-4c97-97b3-59929a10f0d5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75449fe5-8d9e-4c97-97b3-59929a10f0d5-maas-api-tls\") pod \"75449fe5-8d9e-4c97-97b3-59929a10f0d5\" (UID: \"75449fe5-8d9e-4c97-97b3-59929a10f0d5\") "
Oct 30 13:14:54.735369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:54.734012    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75449fe5-8d9e-4c97-97b3-59929a10f0d5-openshift-service-ca" pod "75449fe5-8d9e-4c97-97b3-59929a10f0d5" (UID: "75449fe5-8d9e-4c97-97b3-59929a10f0d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:14:54.765241 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:54.736083    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:54.832740 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:54.826784    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75449fe5-8d9e-4c97-97b3-59929a10f0d5-maas-api-tls" pod "75449fe5-8d9e-4c97-97b3-59929a10f0d5" (UID: "75449fe5-8d9e-4c97-97b3-59929a10f0d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:14:54.832740 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:54.826924    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75449fe5-8d9e-4c97-97b3-59929a10f0d5-kube-api-access-frrgk" pod "75449fe5-8d9e-4c97-97b3-59929a10f0d5" (UID: "75449fe5-8d9e-4c97-97b3-59929a10f0d5"). InnerVolumeSpecName "kube-api-access-frrgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:14:54.833445 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:54.833366    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75449fe5-8d9e-4c97-97b3-59929a10f0d5-metrics-certs" pod "75449fe5-8d9e-4c97-97b3-59929a10f0d5" (UID: "75449fe5-8d9e-4c97-97b3-59929a10f0d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:14:54.836914 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:54.836885    4544 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75449fe5-8d9e-4c97-97b3-59929a10f0d5-metrics-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:14:54.837112 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:54.837094    4544 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75449fe5-8d9e-4c97-97b3-59929a10f0d5-openshift-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:14:54.837212 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:54.837201    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frrgk\" (UniqueName: \"kubernetes.io/projected/75449fe5-8d9e-4c97-97b3-59929a10f0d5-kube-api-access-frrgk\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:14:54.837319 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:54.837307    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75449fe5-8d9e-4c97-97b3-59929a10f0d5-maas-api-tls\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:14:54.837942 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:54.837917    4544 generic.go:356] "Generic (PLEG): container finished" podID="75449fe5-8d9e-4c97-97b3-59929a10f0d5" containerID="0686d74ee925184433845cfebf937318f1c649ceff7c61869696c6ac5d3833e2" exitCode=0
Oct 30 13:14:54.838038 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:54.837968    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7" event={"ID":"75449fe5-8d9e-4c97-97b3-59929a10f0d5","Type":"ContainerDied","Data":"0686d74ee925184433845cfebf937318f1c649ceff7c61869696c6ac5d3833e2"}
Oct 30 13:14:54.838038 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:54.838017    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7" event={"ID":"75449fe5-8d9e-4c97-97b3-59929a10f0d5","Type":"ContainerDied","Data":"a7210faf5e02de18c75b4b691f4e8b5be1dd8d2f805614774300b75b115a2a2b"}
Oct 30 13:14:54.838151 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:54.838047    4544 scope.go:122] "RemoveContainer" containerID="0686d74ee925184433845cfebf937318f1c649ceff7c61869696c6ac5d3833e2"
Oct 30 13:14:54.838212 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:54.838158    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7"
Oct 30 13:14:54.853641 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:54.852843    4544 scope.go:122] "RemoveContainer" containerID="0686d74ee925184433845cfebf937318f1c649ceff7c61869696c6ac5d3833e2"
Oct 30 13:14:54.853641 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:14:54.853341    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0686d74ee925184433845cfebf937318f1c649ceff7c61869696c6ac5d3833e2\": container with ID starting with 0686d74ee925184433845cfebf937318f1c649ceff7c61869696c6ac5d3833e2 not found: ID does not exist" containerID="0686d74ee925184433845cfebf937318f1c649ceff7c61869696c6ac5d3833e2"
Oct 30 13:14:54.853641 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:54.853381    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0686d74ee925184433845cfebf937318f1c649ceff7c61869696c6ac5d3833e2"} err="failed to get container status \"0686d74ee925184433845cfebf937318f1c649ceff7c61869696c6ac5d3833e2\": rpc error: code = NotFound desc = could not find container \"0686d74ee925184433845cfebf937318f1c649ceff7c61869696c6ac5d3833e2\": container with ID starting with 0686d74ee925184433845cfebf937318f1c649ceff7c61869696c6ac5d3833e2 not found: ID does not exist"
Oct 30 13:14:54.894110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:54.894078    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7"]
Oct 30 13:14:54.898280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:54.898249    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f3160a-64d55999d7-pwjp7"]
Oct 30 13:14:54.999456 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:54.999189    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:55.090495 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:55.090462    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75449fe5-8d9e-4c97-97b3-59929a10f0d5" path="/var/lib/kubelet/pods/75449fe5-8d9e-4c97-97b3-59929a10f0d5/volumes"
Oct 30 13:14:55.119349 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:55.119312    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:55.240250 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:55.236827    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:55.504604 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:55.504569    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:55.511712 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:55.511676    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc"]
Oct 30 13:14:55.552372 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:55.552335    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn"]
Oct 30 13:14:55.662879 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:55.662838    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:55.812303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:55.799789    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:57.431862 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:57.431825    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:58.808511 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:58.808471    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:59.008933 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:14:59.008898    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:14:59.221532 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.221497    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:59.284492 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.284463    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"]
Oct 30 13:14:59.285331 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.285316    4544 cpu_manager.go:414] "containerMap: removing container" podUID="48119de6-b81b-4f07-b304-a33e16ab2d68" containerName="istio-proxy"
Oct 30 13:14:59.285403 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.285333    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48119de6-b81b-4f07-b304-a33e16ab2d68" containerName="istio-proxy"
Oct 30 13:14:59.285403 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.285350    4544 cpu_manager.go:414] "containerMap: removing container" podUID="48119de6-b81b-4f07-b304-a33e16ab2d68" containerName="istio-proxy"
Oct 30 13:14:59.285403 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.285354    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48119de6-b81b-4f07-b304-a33e16ab2d68" containerName="istio-proxy"
Oct 30 13:14:59.285403 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.285389    4544 cpu_manager.go:414] "containerMap: removing container" podUID="75449fe5-8d9e-4c97-97b3-59929a10f0d5" containerName="maas-api"
Oct 30 13:14:59.285403 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.285394    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75449fe5-8d9e-4c97-97b3-59929a10f0d5" containerName="maas-api"
Oct 30 13:14:59.285554 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.285505    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="75449fe5-8d9e-4c97-97b3-59929a10f0d5" containerName="maas-api"
Oct 30 13:14:59.285554 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.285518    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="48119de6-b81b-4f07-b304-a33e16ab2d68" containerName="istio-proxy"
Oct 30 13:14:59.285554 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.285527    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="48119de6-b81b-4f07-b304-a33e16ab2d68" containerName="istio-proxy"
Oct 30 13:14:59.289518 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.289495    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:14:59.307735 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.307708    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"]
Oct 30 13:14:59.313711 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.313680    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6af611-openshift-default-dockercfg-s4h66\""
Oct 30 13:14:59.355497 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.355454    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istiod-ca-cert\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:14:59.355667 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.355560    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istio-envoy\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:14:59.355667 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.355605    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istio-podinfo\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:14:59.355667 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.355638    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-workload-socket\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:14:59.355772 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.355691    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-workload-certs\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:14:59.355772 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.355758    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-credential-socket\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:14:59.355841 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.355811    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istio-data\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:14:59.355899 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.355883    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istio-token\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:14:59.355968 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.355953    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjtbk\" (UniqueName: \"kubernetes.io/projected/da3e4fe2-4f62-4385-9fc3-102dbda6c781-kube-api-access-vjtbk\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:14:59.472304 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.462868    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:14:59.526243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.525176    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-credential-socket\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:14:59.526243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.525267    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istio-data\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:14:59.526243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.525328    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istio-token\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:14:59.526243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.525386    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjtbk\" (UniqueName: \"kubernetes.io/projected/da3e4fe2-4f62-4385-9fc3-102dbda6c781-kube-api-access-vjtbk\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:14:59.526243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.525578    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istiod-ca-cert\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:14:59.526243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.525685    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istio-envoy\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:14:59.526243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.525723    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istio-podinfo\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:14:59.526243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.525753    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-workload-socket\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:14:59.526243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.525803    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-workload-certs\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:14:59.530705 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.529856    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-credential-socket\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:14:59.530705 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.530123    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istio-data\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:14:59.554251 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.550539    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-workload-socket\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:14:59.554251 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.551194    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istiod-ca-cert\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:14:59.583245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.567843    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-workload-certs\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:14:59.584958 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.584893    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istio-envoy\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:14:59.619264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.612584    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istio-podinfo\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:14:59.619994 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.619930    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istio-token\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:14:59.620756 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.620733    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjtbk\" (UniqueName: \"kubernetes.io/projected/da3e4fe2-4f62-4385-9fc3-102dbda6c781-kube-api-access-vjtbk\") pod \"e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") " pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:14:59.914241 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.909024    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:14:59.927045 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:14:59.926986    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:00.215884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.215798    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx"]
Oct 30 13:15:00.225777 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.224510    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs"]
Oct 30 13:15:00.225983 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.225882    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx"
Oct 30 13:15:00.230644 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.230619    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs"
Oct 30 13:15:00.264671 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.264638    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx"]
Oct 30 13:15:00.276874 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.276835    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs"]
Oct 30 13:15:00.282115 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.282083    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq"]
Oct 30 13:15:00.288009 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.287982    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq"
Oct 30 13:15:00.299938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.299908    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"]
Oct 30 13:15:00.306398 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:15:00.306365    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3e4fe2_4f62_4385_9fc3_102dbda6c781.slice/crio-d987c9e8106a2e0b529768ef0999ca1961226977ea5cfede472967bdb8222416 WatchSource:0}: Error finding container d987c9e8106a2e0b529768ef0999ca1961226977ea5cfede472967bdb8222416: Status 404 returned error can't find the container with id d987c9e8106a2e0b529768ef0999ca1961226977ea5cfede472967bdb8222416
Oct 30 13:15:00.308451 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.308429    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:15:00.310980 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.310946    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:15:00.311095 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.311014    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:15:00.311095 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.311044    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:15:00.312900 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.312879    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l"]
Oct 30 13:15:00.318429 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.318407    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l"
Oct 30 13:15:00.336373 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.334455    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq"]
Oct 30 13:15:00.352570 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.343860    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpgck\" (UniqueName: \"kubernetes.io/projected/d6ae22b0-e9c0-45bc-85e4-76c659da2808-kube-api-access-wpgck\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs\" (UID: \"d6ae22b0-e9c0-45bc-85e4-76c659da2808\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs"
Oct 30 13:15:00.352570 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.343909    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9t4g\" (UniqueName: \"kubernetes.io/projected/674c045d-1ba0-4075-8ce8-84e3b275dbd1-kube-api-access-p9t4g\") pod \"maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq\" (UID: \"674c045d-1ba0-4075-8ce8-84e3b275dbd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq"
Oct 30 13:15:00.352570 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.344092    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/674c045d-1ba0-4075-8ce8-84e3b275dbd1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq\" (UID: \"674c045d-1ba0-4075-8ce8-84e3b275dbd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq"
Oct 30 13:15:00.352570 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.344158    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6ae22b0-e9c0-45bc-85e4-76c659da2808-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs\" (UID: \"d6ae22b0-e9c0-45bc-85e4-76c659da2808\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs"
Oct 30 13:15:00.352570 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.344296    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qqkz\" (UniqueName: \"kubernetes.io/projected/32e6a633-93f2-48be-87d6-5a9521132614-kube-api-access-9qqkz\") pod \"maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx\" (UID: \"32e6a633-93f2-48be-87d6-5a9521132614\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx"
Oct 30 13:15:00.352570 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.344388    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e6a633-93f2-48be-87d6-5a9521132614-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx\" (UID: \"32e6a633-93f2-48be-87d6-5a9521132614\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx"
Oct 30 13:15:00.385484 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.385452    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l"]
Oct 30 13:15:00.392853 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.391338    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7"]
Oct 30 13:15:00.398007 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.397972    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7"
Oct 30 13:15:00.406139 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.406102    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f"]
Oct 30 13:15:00.412210 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.412186    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f"
Oct 30 13:15:00.444283 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.444026    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7"]
Oct 30 13:15:00.445802 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.445758    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7dgl\" (UniqueName: \"kubernetes.io/projected/72837ba1-2075-4553-a926-3716a21270ee-kube-api-access-w7dgl\") pod \"maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f\" (UID: \"72837ba1-2075-4553-a926-3716a21270ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f"
Oct 30 13:15:00.445950 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.445829    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxv5l\" (UniqueName: \"kubernetes.io/projected/03d3f1b3-4d1c-4aca-82d5-f106740125ad-kube-api-access-rxv5l\") pod \"maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l\" (UID: \"03d3f1b3-4d1c-4aca-82d5-f106740125ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l"
Oct 30 13:15:00.445950 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.445864    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19f3a006-19e8-44af-9db0-366d400001f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7\" (UID: \"19f3a006-19e8-44af-9db0-366d400001f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7"
Oct 30 13:15:00.446066 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.446009    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w287\" (UniqueName: \"kubernetes.io/projected/19f3a006-19e8-44af-9db0-366d400001f3-kube-api-access-8w287\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7\" (UID: \"19f3a006-19e8-44af-9db0-366d400001f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7"
Oct 30 13:15:00.446124 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.446068    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpgck\" (UniqueName: \"kubernetes.io/projected/d6ae22b0-e9c0-45bc-85e4-76c659da2808-kube-api-access-wpgck\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs\" (UID: \"d6ae22b0-e9c0-45bc-85e4-76c659da2808\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs"
Oct 30 13:15:00.446124 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.446099    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9t4g\" (UniqueName: \"kubernetes.io/projected/674c045d-1ba0-4075-8ce8-84e3b275dbd1-kube-api-access-p9t4g\") pod \"maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq\" (UID: \"674c045d-1ba0-4075-8ce8-84e3b275dbd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq"
Oct 30 13:15:00.446292 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.446198    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03d3f1b3-4d1c-4aca-82d5-f106740125ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l\" (UID: \"03d3f1b3-4d1c-4aca-82d5-f106740125ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l"
Oct 30 13:15:00.446365 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.446314    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/674c045d-1ba0-4075-8ce8-84e3b275dbd1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq\" (UID: \"674c045d-1ba0-4075-8ce8-84e3b275dbd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq"
Oct 30 13:15:00.446422 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.446365    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6ae22b0-e9c0-45bc-85e4-76c659da2808-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs\" (UID: \"d6ae22b0-e9c0-45bc-85e4-76c659da2808\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs"
Oct 30 13:15:00.446422 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.446396    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72837ba1-2075-4553-a926-3716a21270ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f\" (UID: \"72837ba1-2075-4553-a926-3716a21270ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f"
Oct 30 13:15:00.446542 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.446526    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qqkz\" (UniqueName: \"kubernetes.io/projected/32e6a633-93f2-48be-87d6-5a9521132614-kube-api-access-9qqkz\") pod \"maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx\" (UID: \"32e6a633-93f2-48be-87d6-5a9521132614\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx"
Oct 30 13:15:00.446620 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.446606    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e6a633-93f2-48be-87d6-5a9521132614-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx\" (UID: \"32e6a633-93f2-48be-87d6-5a9521132614\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx"
Oct 30 13:15:00.447295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.447272    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e6a633-93f2-48be-87d6-5a9521132614-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx\" (UID: \"32e6a633-93f2-48be-87d6-5a9521132614\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx"
Oct 30 13:15:00.448171 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.448151    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/674c045d-1ba0-4075-8ce8-84e3b275dbd1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq\" (UID: \"674c045d-1ba0-4075-8ce8-84e3b275dbd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq"
Oct 30 13:15:00.448729 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.448709    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6ae22b0-e9c0-45bc-85e4-76c659da2808-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs\" (UID: \"d6ae22b0-e9c0-45bc-85e4-76c659da2808\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs"
Oct 30 13:15:00.472118 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.472038    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq"]
Oct 30 13:15:00.482498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.482465    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f"]
Oct 30 13:15:00.482683 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.482592    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq"
Oct 30 13:15:00.512238 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.512167    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc" podUID="f57ae3c0-2e03-4252-bc0a-608b3507e96a" containerName="payload-processing" containerID="cri-o://83dda0c79e593bd8a0ca00d17e15eaedd22ff3ac88e631143618e7c2d1dacfd5" gracePeriod=25
Oct 30 13:15:00.525925 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.525883    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qqkz\" (UniqueName: \"kubernetes.io/projected/32e6a633-93f2-48be-87d6-5a9521132614-kube-api-access-9qqkz\") pod \"maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx\" (UID: \"32e6a633-93f2-48be-87d6-5a9521132614\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx"
Oct 30 13:15:00.528877 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.528846    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpgck\" (UniqueName: \"kubernetes.io/projected/d6ae22b0-e9c0-45bc-85e4-76c659da2808-kube-api-access-wpgck\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs\" (UID: \"d6ae22b0-e9c0-45bc-85e4-76c659da2808\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs"
Oct 30 13:15:00.536805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.536718    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9t4g\" (UniqueName: \"kubernetes.io/projected/674c045d-1ba0-4075-8ce8-84e3b275dbd1-kube-api-access-p9t4g\") pod \"maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq\" (UID: \"674c045d-1ba0-4075-8ce8-84e3b275dbd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq"
Oct 30 13:15:00.538931 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.538897    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq"]
Oct 30 13:15:00.549052 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.547787    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7dgl\" (UniqueName: \"kubernetes.io/projected/72837ba1-2075-4553-a926-3716a21270ee-kube-api-access-w7dgl\") pod \"maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f\" (UID: \"72837ba1-2075-4553-a926-3716a21270ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f"
Oct 30 13:15:00.549052 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.547850    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxv5l\" (UniqueName: \"kubernetes.io/projected/03d3f1b3-4d1c-4aca-82d5-f106740125ad-kube-api-access-rxv5l\") pod \"maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l\" (UID: \"03d3f1b3-4d1c-4aca-82d5-f106740125ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l"
Oct 30 13:15:00.549052 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.547864    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq"]
Oct 30 13:15:00.549052 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.547877    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19f3a006-19e8-44af-9db0-366d400001f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7\" (UID: \"19f3a006-19e8-44af-9db0-366d400001f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7"
Oct 30 13:15:00.549052 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.547962    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ddc624d-f00d-4c01-b9e1-877259555186-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq\" (UID: \"1ddc624d-f00d-4c01-b9e1-877259555186\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq"
Oct 30 13:15:00.549052 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.548033    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6xn9\" (UniqueName: \"kubernetes.io/projected/1ddc624d-f00d-4c01-b9e1-877259555186-kube-api-access-s6xn9\") pod \"maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq\" (UID: \"1ddc624d-f00d-4c01-b9e1-877259555186\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq"
Oct 30 13:15:00.549052 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.548081    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w287\" (UniqueName: \"kubernetes.io/projected/19f3a006-19e8-44af-9db0-366d400001f3-kube-api-access-8w287\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7\" (UID: \"19f3a006-19e8-44af-9db0-366d400001f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7"
Oct 30 13:15:00.549052 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.548206    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03d3f1b3-4d1c-4aca-82d5-f106740125ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l\" (UID: \"03d3f1b3-4d1c-4aca-82d5-f106740125ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l"
Oct 30 13:15:00.549052 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.548280    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72837ba1-2075-4553-a926-3716a21270ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f\" (UID: \"72837ba1-2075-4553-a926-3716a21270ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f"
Oct 30 13:15:00.552290 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.550112    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72837ba1-2075-4553-a926-3716a21270ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f\" (UID: \"72837ba1-2075-4553-a926-3716a21270ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f"
Oct 30 13:15:00.552290 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.552003    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19f3a006-19e8-44af-9db0-366d400001f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7\" (UID: \"19f3a006-19e8-44af-9db0-366d400001f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7"
Oct 30 13:15:00.552290 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.552157    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03d3f1b3-4d1c-4aca-82d5-f106740125ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l\" (UID: \"03d3f1b3-4d1c-4aca-82d5-f106740125ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l"
Oct 30 13:15:00.552970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.552782    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx"
Oct 30 13:15:00.553203 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.553177    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn" podUID="caa19642-0691-40fe-aba4-33199f334f45" containerName="payload-pre-processing" containerID="cri-o://4e16d9d6137a8423edbfbc61c069c49fe6f2e9ac86730d7aa07e79c34e4ac618" gracePeriod=25
Oct 30 13:15:00.558136 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.555319    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq"
Oct 30 13:15:00.566902 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.566667    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs"
Oct 30 13:15:00.582556 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.582502    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj"]
Oct 30 13:15:00.592066 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.592034    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxv5l\" (UniqueName: \"kubernetes.io/projected/03d3f1b3-4d1c-4aca-82d5-f106740125ad-kube-api-access-rxv5l\") pod \"maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l\" (UID: \"03d3f1b3-4d1c-4aca-82d5-f106740125ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l"
Oct 30 13:15:00.595108 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.595082    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq"]
Oct 30 13:15:00.595648 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.595269    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj"
Oct 30 13:15:00.595924 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.595900    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w287\" (UniqueName: \"kubernetes.io/projected/19f3a006-19e8-44af-9db0-366d400001f3-kube-api-access-8w287\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7\" (UID: \"19f3a006-19e8-44af-9db0-366d400001f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7"
Oct 30 13:15:00.599125 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.599101    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7dgl\" (UniqueName: \"kubernetes.io/projected/72837ba1-2075-4553-a926-3716a21270ee-kube-api-access-w7dgl\") pod \"maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f\" (UID: \"72837ba1-2075-4553-a926-3716a21270ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f"
Oct 30 13:15:00.616588 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.614824    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq"
Oct 30 13:15:00.625954 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.625916    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj"]
Oct 30 13:15:00.651847 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.650716    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6xn9\" (UniqueName: \"kubernetes.io/projected/1ddc624d-f00d-4c01-b9e1-877259555186-kube-api-access-s6xn9\") pod \"maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq\" (UID: \"1ddc624d-f00d-4c01-b9e1-877259555186\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq"
Oct 30 13:15:00.651847 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.650989    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74f6b\" (UniqueName: \"kubernetes.io/projected/ac81161c-cd40-46f4-9fe0-2490f472393c-kube-api-access-74f6b\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq\" (UID: \"ac81161c-cd40-46f4-9fe0-2490f472393c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq"
Oct 30 13:15:00.651847 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.651081    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/002634ca-d404-4827-b3e7-d007555a2792-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj\" (UID: \"002634ca-d404-4827-b3e7-d007555a2792\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj"
Oct 30 13:15:00.651847 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.651157    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z4kn\" (UniqueName: \"kubernetes.io/projected/002634ca-d404-4827-b3e7-d007555a2792-kube-api-access-5z4kn\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj\" (UID: \"002634ca-d404-4827-b3e7-d007555a2792\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj"
Oct 30 13:15:00.651847 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.651283    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac81161c-cd40-46f4-9fe0-2490f472393c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq\" (UID: \"ac81161c-cd40-46f4-9fe0-2490f472393c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq"
Oct 30 13:15:00.653558 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.652503    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l"
Oct 30 13:15:00.654550 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.654495    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ddc624d-f00d-4c01-b9e1-877259555186-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq\" (UID: \"1ddc624d-f00d-4c01-b9e1-877259555186\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq"
Oct 30 13:15:00.657327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.654678    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ddc624d-f00d-4c01-b9e1-877259555186-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq\" (UID: \"1ddc624d-f00d-4c01-b9e1-877259555186\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq"
Oct 30 13:15:00.699440 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.699372    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6xn9\" (UniqueName: \"kubernetes.io/projected/1ddc624d-f00d-4c01-b9e1-877259555186-kube-api-access-s6xn9\") pod \"maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq\" (UID: \"1ddc624d-f00d-4c01-b9e1-877259555186\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq"
Oct 30 13:15:00.728128 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.727605    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7"
Oct 30 13:15:00.736852 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.736633    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f"
Oct 30 13:15:00.761137 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.760808    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74f6b\" (UniqueName: \"kubernetes.io/projected/ac81161c-cd40-46f4-9fe0-2490f472393c-kube-api-access-74f6b\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq\" (UID: \"ac81161c-cd40-46f4-9fe0-2490f472393c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq"
Oct 30 13:15:00.761137 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.760902    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/002634ca-d404-4827-b3e7-d007555a2792-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj\" (UID: \"002634ca-d404-4827-b3e7-d007555a2792\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj"
Oct 30 13:15:00.761137 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.760967    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z4kn\" (UniqueName: \"kubernetes.io/projected/002634ca-d404-4827-b3e7-d007555a2792-kube-api-access-5z4kn\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj\" (UID: \"002634ca-d404-4827-b3e7-d007555a2792\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj"
Oct 30 13:15:00.761137 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.761025    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac81161c-cd40-46f4-9fe0-2490f472393c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq\" (UID: \"ac81161c-cd40-46f4-9fe0-2490f472393c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq"
Oct 30 13:15:00.764163 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.764058    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/002634ca-d404-4827-b3e7-d007555a2792-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj\" (UID: \"002634ca-d404-4827-b3e7-d007555a2792\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj"
Oct 30 13:15:00.767202 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.767156    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac81161c-cd40-46f4-9fe0-2490f472393c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq\" (UID: \"ac81161c-cd40-46f4-9fe0-2490f472393c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq"
Oct 30 13:15:00.812452 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.812073    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z4kn\" (UniqueName: \"kubernetes.io/projected/002634ca-d404-4827-b3e7-d007555a2792-kube-api-access-5z4kn\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj\" (UID: \"002634ca-d404-4827-b3e7-d007555a2792\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj"
Oct 30 13:15:00.823724 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.822975    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74f6b\" (UniqueName: \"kubernetes.io/projected/ac81161c-cd40-46f4-9fe0-2490f472393c-kube-api-access-74f6b\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq\" (UID: \"ac81161c-cd40-46f4-9fe0-2490f472393c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq"
Oct 30 13:15:00.834827 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.834696    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq"
Oct 30 13:15:00.878150 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.878099    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq"
Oct 30 13:15:00.956004 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.954758    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj"
Oct 30 13:15:00.990515 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.990436    4544 generic.go:356] "Generic (PLEG): container finished" podID="caa19642-0691-40fe-aba4-33199f334f45" containerID="4e16d9d6137a8423edbfbc61c069c49fe6f2e9ac86730d7aa07e79c34e4ac618" exitCode=0
Oct 30 13:15:00.990609 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:00.990553    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn" event={"ID":"caa19642-0691-40fe-aba4-33199f334f45","Type":"ContainerDied","Data":"4e16d9d6137a8423edbfbc61c069c49fe6f2e9ac86730d7aa07e79c34e4ac618"}
Oct 30 13:15:01.002368 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:01.000774    4544 generic.go:356] "Generic (PLEG): container finished" podID="f57ae3c0-2e03-4252-bc0a-608b3507e96a" containerID="83dda0c79e593bd8a0ca00d17e15eaedd22ff3ac88e631143618e7c2d1dacfd5" exitCode=0
Oct 30 13:15:01.002368 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:01.000991    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc" event={"ID":"f57ae3c0-2e03-4252-bc0a-608b3507e96a","Type":"ContainerDied","Data":"83dda0c79e593bd8a0ca00d17e15eaedd22ff3ac88e631143618e7c2d1dacfd5"}
Oct 30 13:15:01.020687 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:01.019457    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648" event={"ID":"da3e4fe2-4f62-4385-9fc3-102dbda6c781","Type":"ContainerStarted","Data":"bc2ef4441794765bc7ae114875739124ce0b10ba8092dee461d869ff0616b0fa"}
Oct 30 13:15:01.020687 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:01.019523    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648" event={"ID":"da3e4fe2-4f62-4385-9fc3-102dbda6c781","Type":"ContainerStarted","Data":"d987c9e8106a2e0b529768ef0999ca1961226977ea5cfede472967bdb8222416"}
Oct 30 13:15:01.098280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:01.097338    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648" podStartSLOduration=2.097316971 podStartE2EDuration="2.097316971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:01.084760406 +0000 UTC m=+1796.131894371" watchObservedRunningTime="2026-10-30 13:15:01.097316971 +0000 UTC m=+1796.144450930"
Oct 30 13:15:01.119441 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:01.119408    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc"
Oct 30 13:15:01.163773 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:01.163537    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx"]
Oct 30 13:15:01.165976 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:15:01.165934    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e6a633_93f2_48be_87d6_5a9521132614.slice/crio-9a6db18cdffee3fdb971cba56cdf31950a4a4cc4ffe5198730673bce39319844 WatchSource:0}: Error finding container 9a6db18cdffee3fdb971cba56cdf31950a4a4cc4ffe5198730673bce39319844: Status 404 returned error can't find the container with id 9a6db18cdffee3fdb971cba56cdf31950a4a4cc4ffe5198730673bce39319844
Oct 30 13:15:01.211268 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:01.211233    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f57ae3c0-2e03-4252-bc0a-608b3507e96a-kube-api-access-s62c2\" (UniqueName: \"kubernetes.io/projected/f57ae3c0-2e03-4252-bc0a-608b3507e96a-kube-api-access-s62c2\") pod \"f57ae3c0-2e03-4252-bc0a-608b3507e96a\" (UID: \"f57ae3c0-2e03-4252-bc0a-608b3507e96a\") "
Oct 30 13:15:01.211427 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:01.211312    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f57ae3c0-2e03-4252-bc0a-608b3507e96a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57ae3c0-2e03-4252-bc0a-608b3507e96a-plugins-config-volume\") pod \"f57ae3c0-2e03-4252-bc0a-608b3507e96a\" (UID: \"f57ae3c0-2e03-4252-bc0a-608b3507e96a\") "
Oct 30 13:15:01.213339 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:01.213307    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f57ae3c0-2e03-4252-bc0a-608b3507e96a-plugins-config-volume" pod "f57ae3c0-2e03-4252-bc0a-608b3507e96a" (UID: "f57ae3c0-2e03-4252-bc0a-608b3507e96a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:01.217316 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:01.217284    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f57ae3c0-2e03-4252-bc0a-608b3507e96a-kube-api-access-s62c2" pod "f57ae3c0-2e03-4252-bc0a-608b3507e96a" (UID: "f57ae3c0-2e03-4252-bc0a-608b3507e96a"). InnerVolumeSpecName "kube-api-access-s62c2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:01.280353 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:01.280244    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn"
Oct 30 13:15:01.322285 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:01.316684    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s62c2\" (UniqueName: \"kubernetes.io/projected/f57ae3c0-2e03-4252-bc0a-608b3507e96a-kube-api-access-s62c2\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:01.322285 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:01.316724    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57ae3c0-2e03-4252-bc0a-608b3507e96a-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:01.662848 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:01.652702    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caa19642-0691-40fe-aba4-33199f334f45-kube-api-access-7ggp4\" (UniqueName: \"kubernetes.io/projected/caa19642-0691-40fe-aba4-33199f334f45-kube-api-access-7ggp4\") pod \"caa19642-0691-40fe-aba4-33199f334f45\" (UID: \"caa19642-0691-40fe-aba4-33199f334f45\") "
Oct 30 13:15:01.662848 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:01.652812    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caa19642-0691-40fe-aba4-33199f334f45-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caa19642-0691-40fe-aba4-33199f334f45-plugins-config-volume\") pod \"caa19642-0691-40fe-aba4-33199f334f45\" (UID: \"caa19642-0691-40fe-aba4-33199f334f45\") "
Oct 30 13:15:01.662848 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:01.661432    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caa19642-0691-40fe-aba4-33199f334f45-plugins-config-volume" pod "caa19642-0691-40fe-aba4-33199f334f45" (UID: "caa19642-0691-40fe-aba4-33199f334f45"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:01.676322 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:01.676275    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caa19642-0691-40fe-aba4-33199f334f45-kube-api-access-7ggp4" pod "caa19642-0691-40fe-aba4-33199f334f45" (UID: "caa19642-0691-40fe-aba4-33199f334f45"). InnerVolumeSpecName "kube-api-access-7ggp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:01.684747 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:15:01.684542    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod674c045d_1ba0_4075_8ce8_84e3b275dbd1.slice/crio-4ee7e89e7b2ad00316904f06e428d8da4a355f66258f5a836c6fe3fbac035646 WatchSource:0}: Error finding container 4ee7e89e7b2ad00316904f06e428d8da4a355f66258f5a836c6fe3fbac035646: Status 404 returned error can't find the container with id 4ee7e89e7b2ad00316904f06e428d8da4a355f66258f5a836c6fe3fbac035646
Oct 30 13:15:01.695456 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:01.695428    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs"]
Oct 30 13:15:01.706629 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:01.706594    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq"]
Oct 30 13:15:01.722013 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:01.721980    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l"]
Oct 30 13:15:01.730360 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:15:01.730334    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03d3f1b3_4d1c_4aca_82d5_f106740125ad.slice/crio-c45482331ad8004146dc46894b9b8556ce1ba7eae995f7af5c9d02817f51df75 WatchSource:0}: Error finding container c45482331ad8004146dc46894b9b8556ce1ba7eae995f7af5c9d02817f51df75: Status 404 returned error can't find the container with id c45482331ad8004146dc46894b9b8556ce1ba7eae995f7af5c9d02817f51df75
Oct 30 13:15:01.758434 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:01.756762    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caa19642-0691-40fe-aba4-33199f334f45-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:01.758434 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:01.756796    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ggp4\" (UniqueName: \"kubernetes.io/projected/caa19642-0691-40fe-aba4-33199f334f45-kube-api-access-7ggp4\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:01.758434 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:01.756882    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7"]
Oct 30 13:15:01.768501 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:01.768477    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq"]
Oct 30 13:15:01.775431 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:01.775407    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj"]
Oct 30 13:15:01.786463 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:01.786434    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f"]
Oct 30 13:15:01.891617 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:01.891589    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq"]
Oct 30 13:15:01.910286 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:01.910252    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:15:01.917582 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:01.917558    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:15:02.038895 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:02.038864    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq" event={"ID":"ac81161c-cd40-46f4-9fe0-2490f472393c","Type":"ContainerStarted","Data":"2cd327a23285d691e02d46ea566355d4c09110ee0cabd55fb9289cd2b03a80be"}
Oct 30 13:15:02.040367 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:02.040343    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l" event={"ID":"03d3f1b3-4d1c-4aca-82d5-f106740125ad","Type":"ContainerStarted","Data":"c45482331ad8004146dc46894b9b8556ce1ba7eae995f7af5c9d02817f51df75"}
Oct 30 13:15:02.041562 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:02.041541    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq" event={"ID":"674c045d-1ba0-4075-8ce8-84e3b275dbd1","Type":"ContainerStarted","Data":"4ee7e89e7b2ad00316904f06e428d8da4a355f66258f5a836c6fe3fbac035646"}
Oct 30 13:15:02.043080 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:02.043063    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7" event={"ID":"19f3a006-19e8-44af-9db0-366d400001f3","Type":"ContainerStarted","Data":"92fe53390561d3191b07716cb5e22d3b98c419bb1c7451800a4c7a99cb8f6445"}
Oct 30 13:15:02.044582 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:02.044563    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f" event={"ID":"72837ba1-2075-4553-a926-3716a21270ee","Type":"ContainerStarted","Data":"b406e7a806749140e7345c7db66c898b3baa3af7bf18e7c5f20ad8f1c5dd7d36"}
Oct 30 13:15:02.046019 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:02.045999    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs" event={"ID":"d6ae22b0-e9c0-45bc-85e4-76c659da2808","Type":"ContainerStarted","Data":"94af8df1e5ade542f2bfa2b03eff14ce889fa3f59e2b101d0536918282b06a6d"}
Oct 30 13:15:02.047373 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:02.047354    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj" event={"ID":"002634ca-d404-4827-b3e7-d007555a2792","Type":"ContainerStarted","Data":"1b5769b5fd03f661c96e72bfee01173548827dbb60899c53cd026e6f7a31e07e"}
Oct 30 13:15:02.049017 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:02.048993    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn" event={"ID":"caa19642-0691-40fe-aba4-33199f334f45","Type":"ContainerDied","Data":"037ca1ddbece7bbeff1a4bd3100a27e0247fa70f1dbc23faa29a0ab52114c40c"}
Oct 30 13:15:02.049119 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:02.049022    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn"
Oct 30 13:15:02.049119 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:02.049040    4544 scope.go:122] "RemoveContainer" containerID="4e16d9d6137a8423edbfbc61c069c49fe6f2e9ac86730d7aa07e79c34e4ac618"
Oct 30 13:15:02.053260 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:02.050803    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc"
Oct 30 13:15:02.053260 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:02.050806    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc" event={"ID":"f57ae3c0-2e03-4252-bc0a-608b3507e96a","Type":"ContainerDied","Data":"043ac1a64db35268970dac9f25ca926c11dab30a5704c368da44299ffff58574"}
Oct 30 13:15:02.053260 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:02.052546    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx" event={"ID":"32e6a633-93f2-48be-87d6-5a9521132614","Type":"ContainerStarted","Data":"03f7b7094265be8aed3c29fa05617d21c8a8b608ecd6b96d9be741a98439f7d1"}
Oct 30 13:15:02.053260 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:02.052584    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx" event={"ID":"32e6a633-93f2-48be-87d6-5a9521132614","Type":"ContainerStarted","Data":"9a6db18cdffee3fdb971cba56cdf31950a4a4cc4ffe5198730673bce39319844"}
Oct 30 13:15:02.054552 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:02.054443    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq" event={"ID":"1ddc624d-f00d-4c01-b9e1-877259555186","Type":"ContainerStarted","Data":"c0457d223d5312188062516a156588d044edc5280f77f986b3a6b177c04ad7f6"}
Oct 30 13:15:02.054650 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:02.054640    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:15:02.056172 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:02.056126    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:15:02.066383 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:02.066364    4544 scope.go:122] "RemoveContainer" containerID="83dda0c79e593bd8a0ca00d17e15eaedd22ff3ac88e631143618e7c2d1dacfd5"
Oct 30 13:15:02.066648 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:02.066619    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:02.106600 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:02.106563    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn"]
Oct 30 13:15:02.111509 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:02.111477    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn"]
Oct 30 13:15:02.379412 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:02.379352    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx" podStartSLOduration=2.379337654 podStartE2EDuration="2.379337654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:02.331291976 +0000 UTC m=+1797.378425928" watchObservedRunningTime="2026-10-30 13:15:02.379337654 +0000 UTC m=+1797.426471603"
Oct 30 13:15:02.380562 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:02.380535    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc"]
Oct 30 13:15:02.386036 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:02.386007    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc"]
Oct 30 13:15:03.287715 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:03.286882    4544 generic.go:356] "Generic (PLEG): container finished" podID="32e6a633-93f2-48be-87d6-5a9521132614" containerID="03f7b7094265be8aed3c29fa05617d21c8a8b608ecd6b96d9be741a98439f7d1" exitCode=60
Oct 30 13:15:03.308002 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:03.307946    4544 scope.go:122] "RemoveContainer" containerID="03f7b7094265be8aed3c29fa05617d21c8a8b608ecd6b96d9be741a98439f7d1"
Oct 30 13:15:03.350550 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:03.350516    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caa19642-0691-40fe-aba4-33199f334f45" path="/var/lib/kubelet/pods/caa19642-0691-40fe-aba4-33199f334f45/volumes"
Oct 30 13:15:03.351687 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:03.351658    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f57ae3c0-2e03-4252-bc0a-608b3507e96a" path="/var/lib/kubelet/pods/f57ae3c0-2e03-4252-bc0a-608b3507e96a/volumes"
Oct 30 13:15:03.366974 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:03.366928    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx" event={"ID":"32e6a633-93f2-48be-87d6-5a9521132614","Type":"ContainerDied","Data":"03f7b7094265be8aed3c29fa05617d21c8a8b608ecd6b96d9be741a98439f7d1"}
Oct 30 13:15:03.383448 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:03.376551    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq" event={"ID":"1ddc624d-f00d-4c01-b9e1-877259555186","Type":"ContainerStarted","Data":"f4589ab2835bf9e7fb86bc6fed1da2f6128daeaaac307ad988e0b8e9bd920402"}
Oct 30 13:15:03.383448 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:03.376610    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq" event={"ID":"ac81161c-cd40-46f4-9fe0-2490f472393c","Type":"ContainerStarted","Data":"53bc712b1edbb69ef844ed2bbe24fbdd63b32fa426095873bcb647dcf39aaca2"}
Oct 30 13:15:03.383448 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:03.376639    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l" event={"ID":"03d3f1b3-4d1c-4aca-82d5-f106740125ad","Type":"ContainerStarted","Data":"3f35da18db4631aee7e81b02ef622d390de421844f0176faedd34615e98ad553"}
Oct 30 13:15:03.383448 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:03.376664    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq" event={"ID":"674c045d-1ba0-4075-8ce8-84e3b275dbd1","Type":"ContainerDied","Data":"5c3c2c196b81bc7dd431370b71f502ca129c27914a158d13d60cb89960a3ffc7"}
Oct 30 13:15:03.383448 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:03.370657    4544 generic.go:356] "Generic (PLEG): container finished" podID="674c045d-1ba0-4075-8ce8-84e3b275dbd1" containerID="5c3c2c196b81bc7dd431370b71f502ca129c27914a158d13d60cb89960a3ffc7" exitCode=60
Oct 30 13:15:03.383448 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:03.371112    4544 scope.go:122] "RemoveContainer" containerID="5c3c2c196b81bc7dd431370b71f502ca129c27914a158d13d60cb89960a3ffc7"
Oct 30 13:15:03.385918 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:03.385664    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7" event={"ID":"19f3a006-19e8-44af-9db0-366d400001f3","Type":"ContainerStarted","Data":"878c949d1b02e68c9166dba6d763bdaf49200d01bf77a7693f2c835f6a6c4f98"}
Oct 30 13:15:03.386166 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:03.386146    4544 scope.go:122] "RemoveContainer" containerID="878c949d1b02e68c9166dba6d763bdaf49200d01bf77a7693f2c835f6a6c4f98"
Oct 30 13:15:03.389381 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:03.388701    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f" event={"ID":"72837ba1-2075-4553-a926-3716a21270ee","Type":"ContainerStarted","Data":"88f01120e070ef0fd7e6970939587038b3a6d4e75bdc47a40bc5f43f030d972b"}
Oct 30 13:15:03.391607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:03.391578    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs" event={"ID":"d6ae22b0-e9c0-45bc-85e4-76c659da2808","Type":"ContainerStarted","Data":"b52c77893ea730ba8695b8e8139234f6fecb11bfe0ec95776a64ab7be51eb1c2"}
Oct 30 13:15:03.395729 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:03.395694    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj" event={"ID":"002634ca-d404-4827-b3e7-d007555a2792","Type":"ContainerStarted","Data":"b6854f1fb9842c1ee0aaf135458ec6cff2c579190a6d6cb186777778a41f57f6"}
Oct 30 13:15:03.396153 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:03.396131    4544 scope.go:122] "RemoveContainer" containerID="b6854f1fb9842c1ee0aaf135458ec6cff2c579190a6d6cb186777778a41f57f6"
Oct 30 13:15:03.444554 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:03.444499    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l" podStartSLOduration=3.444483084 podStartE2EDuration="3.444483084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:03.442618017 +0000 UTC m=+1798.489751970" watchObservedRunningTime="2026-10-30 13:15:03.444483084 +0000 UTC m=+1798.491617036"
Oct 30 13:15:03.545376 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:03.545321    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq" podStartSLOduration=3.545302592 podStartE2EDuration="3.545302592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:03.540945662 +0000 UTC m=+1798.588079614" watchObservedRunningTime="2026-10-30 13:15:03.545302592 +0000 UTC m=+1798.592436546"
Oct 30 13:15:03.601270 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:03.601061    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq" podStartSLOduration=3.601034933 podStartE2EDuration="3.601034933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:03.593709405 +0000 UTC m=+1798.640843359" watchObservedRunningTime="2026-10-30 13:15:03.601034933 +0000 UTC m=+1798.648168887"
Oct 30 13:15:03.752470 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:03.752385    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:03.806864 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:03.806751    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs" podStartSLOduration=3.806736003 podStartE2EDuration="3.806736003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:03.805848057 +0000 UTC m=+1798.852982009" watchObservedRunningTime="2026-10-30 13:15:03.806736003 +0000 UTC m=+1798.853869957"
Oct 30 13:15:03.868281 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:03.868200    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f" podStartSLOduration=3.868185366 podStartE2EDuration="3.868185366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:03.863619947 +0000 UTC m=+1798.910753899" watchObservedRunningTime="2026-10-30 13:15:03.868185366 +0000 UTC m=+1798.915319318"
Oct 30 13:15:03.955376 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:03.955341    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:04.429316 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:04.429279    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx" event={"ID":"32e6a633-93f2-48be-87d6-5a9521132614","Type":"ContainerStarted","Data":"850cde6b6cb3e239554e28f8bdc8557f080e3fc995ace6b29820d6c745f2be59"}
Oct 30 13:15:04.431128 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:04.431100    4544 generic.go:356] "Generic (PLEG): container finished" podID="1ddc624d-f00d-4c01-b9e1-877259555186" containerID="f4589ab2835bf9e7fb86bc6fed1da2f6128daeaaac307ad988e0b8e9bd920402" exitCode=60
Oct 30 13:15:04.431286 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:04.431179    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq" event={"ID":"1ddc624d-f00d-4c01-b9e1-877259555186","Type":"ContainerDied","Data":"f4589ab2835bf9e7fb86bc6fed1da2f6128daeaaac307ad988e0b8e9bd920402"}
Oct 30 13:15:04.431511 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:04.431496    4544 scope.go:122] "RemoveContainer" containerID="f4589ab2835bf9e7fb86bc6fed1da2f6128daeaaac307ad988e0b8e9bd920402"
Oct 30 13:15:04.433031 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:04.433006    4544 generic.go:356] "Generic (PLEG): container finished" podID="ac81161c-cd40-46f4-9fe0-2490f472393c" containerID="53bc712b1edbb69ef844ed2bbe24fbdd63b32fa426095873bcb647dcf39aaca2" exitCode=60
Oct 30 13:15:04.433138 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:04.433090    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq" event={"ID":"ac81161c-cd40-46f4-9fe0-2490f472393c","Type":"ContainerDied","Data":"53bc712b1edbb69ef844ed2bbe24fbdd63b32fa426095873bcb647dcf39aaca2"}
Oct 30 13:15:04.433358 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:04.433337    4544 scope.go:122] "RemoveContainer" containerID="53bc712b1edbb69ef844ed2bbe24fbdd63b32fa426095873bcb647dcf39aaca2"
Oct 30 13:15:04.435098 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:04.435062    4544 generic.go:356] "Generic (PLEG): container finished" podID="03d3f1b3-4d1c-4aca-82d5-f106740125ad" containerID="3f35da18db4631aee7e81b02ef622d390de421844f0176faedd34615e98ad553" exitCode=60
Oct 30 13:15:04.435163 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:04.435124    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l" event={"ID":"03d3f1b3-4d1c-4aca-82d5-f106740125ad","Type":"ContainerDied","Data":"3f35da18db4631aee7e81b02ef622d390de421844f0176faedd34615e98ad553"}
Oct 30 13:15:04.435488 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:04.435430    4544 scope.go:122] "RemoveContainer" containerID="3f35da18db4631aee7e81b02ef622d390de421844f0176faedd34615e98ad553"
Oct 30 13:15:04.437823 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:04.437794    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq" event={"ID":"674c045d-1ba0-4075-8ce8-84e3b275dbd1","Type":"ContainerStarted","Data":"1a0d6646183708363d9520088f95cd8816576e2f6a933c45eba30fd797b4ff15"}
Oct 30 13:15:04.439912 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:04.439888    4544 generic.go:356] "Generic (PLEG): container finished" podID="19f3a006-19e8-44af-9db0-366d400001f3" containerID="878c949d1b02e68c9166dba6d763bdaf49200d01bf77a7693f2c835f6a6c4f98" exitCode=60
Oct 30 13:15:04.440027 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:04.439969    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7" event={"ID":"19f3a006-19e8-44af-9db0-366d400001f3","Type":"ContainerDied","Data":"878c949d1b02e68c9166dba6d763bdaf49200d01bf77a7693f2c835f6a6c4f98"}
Oct 30 13:15:04.440027 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:04.440021    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7" event={"ID":"19f3a006-19e8-44af-9db0-366d400001f3","Type":"ContainerStarted","Data":"b0b76ba094471c5282b513605476500222e221bc386d4f51ab294539c657df99"}
Oct 30 13:15:04.441668 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:04.441644    4544 generic.go:356] "Generic (PLEG): container finished" podID="72837ba1-2075-4553-a926-3716a21270ee" containerID="88f01120e070ef0fd7e6970939587038b3a6d4e75bdc47a40bc5f43f030d972b" exitCode=60
Oct 30 13:15:04.441773 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:04.441719    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f" event={"ID":"72837ba1-2075-4553-a926-3716a21270ee","Type":"ContainerDied","Data":"88f01120e070ef0fd7e6970939587038b3a6d4e75bdc47a40bc5f43f030d972b"}
Oct 30 13:15:04.442006 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:04.441989    4544 scope.go:122] "RemoveContainer" containerID="88f01120e070ef0fd7e6970939587038b3a6d4e75bdc47a40bc5f43f030d972b"
Oct 30 13:15:04.443624 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:04.443601    4544 generic.go:356] "Generic (PLEG): container finished" podID="d6ae22b0-e9c0-45bc-85e4-76c659da2808" containerID="b52c77893ea730ba8695b8e8139234f6fecb11bfe0ec95776a64ab7be51eb1c2" exitCode=60
Oct 30 13:15:04.443721 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:04.443675    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs" event={"ID":"d6ae22b0-e9c0-45bc-85e4-76c659da2808","Type":"ContainerDied","Data":"b52c77893ea730ba8695b8e8139234f6fecb11bfe0ec95776a64ab7be51eb1c2"}
Oct 30 13:15:04.443941 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:04.443907    4544 scope.go:122] "RemoveContainer" containerID="b52c77893ea730ba8695b8e8139234f6fecb11bfe0ec95776a64ab7be51eb1c2"
Oct 30 13:15:04.447735 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:04.447707    4544 generic.go:356] "Generic (PLEG): container finished" podID="002634ca-d404-4827-b3e7-d007555a2792" containerID="b6854f1fb9842c1ee0aaf135458ec6cff2c579190a6d6cb186777778a41f57f6" exitCode=60
Oct 30 13:15:04.447821 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:04.447770    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj" event={"ID":"002634ca-d404-4827-b3e7-d007555a2792","Type":"ContainerDied","Data":"b6854f1fb9842c1ee0aaf135458ec6cff2c579190a6d6cb186777778a41f57f6"}
Oct 30 13:15:04.447821 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:04.447814    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj" event={"ID":"002634ca-d404-4827-b3e7-d007555a2792","Type":"ContainerStarted","Data":"ecdc9d99b6ced5d7d23f76b0419b875653a786d1ca2835aaf9291e677c6b2e1f"}
Oct 30 13:15:04.559898 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:04.559845    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7" podStartSLOduration=4.559830111 podStartE2EDuration="4.559830111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:04.559101832 +0000 UTC m=+1799.606235784" watchObservedRunningTime="2026-10-30 13:15:04.559830111 +0000 UTC m=+1799.606964065"
Oct 30 13:15:04.605697 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:04.605638    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq" podStartSLOduration=4.605622212 podStartE2EDuration="4.605622212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:04.601291007 +0000 UTC m=+1799.648424957" watchObservedRunningTime="2026-10-30 13:15:04.605622212 +0000 UTC m=+1799.652756166"
Oct 30 13:15:04.879566 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:04.879477    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj" podStartSLOduration=4.879454276 podStartE2EDuration="4.879454276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:04.861701904 +0000 UTC m=+1799.908835857" watchObservedRunningTime="2026-10-30 13:15:04.879454276 +0000 UTC m=+1799.926588230"
Oct 30 13:15:05.320080 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:05.319929    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:05.534349 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:05.534313    4544 generic.go:356] "Generic (PLEG): container finished" podID="32e6a633-93f2-48be-87d6-5a9521132614" containerID="850cde6b6cb3e239554e28f8bdc8557f080e3fc995ace6b29820d6c745f2be59" exitCode=60
Oct 30 13:15:05.534940 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:05.534918    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx" event={"ID":"32e6a633-93f2-48be-87d6-5a9521132614","Type":"ContainerDied","Data":"850cde6b6cb3e239554e28f8bdc8557f080e3fc995ace6b29820d6c745f2be59"}
Oct 30 13:15:05.535076 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:05.535065    4544 scope.go:122] "RemoveContainer" containerID="03f7b7094265be8aed3c29fa05617d21c8a8b608ecd6b96d9be741a98439f7d1"
Oct 30 13:15:05.563458 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:05.560074    4544 scope.go:122] "RemoveContainer" containerID="850cde6b6cb3e239554e28f8bdc8557f080e3fc995ace6b29820d6c745f2be59"
Oct 30 13:15:05.563458 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:15:05.560640    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx_odh-ai-gateway-infra(32e6a633-93f2-48be-87d6-5a9521132614)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx" podUID="32e6a633-93f2-48be-87d6-5a9521132614"
Oct 30 13:15:05.584480 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:05.583460    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq" event={"ID":"ac81161c-cd40-46f4-9fe0-2490f472393c","Type":"ContainerStarted","Data":"f417e02048a1b64e11a66bfa085bde462f9da5399aa78a0d6c134b809ff6a2fd"}
Oct 30 13:15:05.621748 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:05.615377    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l" event={"ID":"03d3f1b3-4d1c-4aca-82d5-f106740125ad","Type":"ContainerStarted","Data":"0c24808e7684626ed1060df0158e5112fbf9d28f24b93d5846672ba7fb4c294a"}
Oct 30 13:15:05.642403 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:05.633527    4544 generic.go:356] "Generic (PLEG): container finished" podID="674c045d-1ba0-4075-8ce8-84e3b275dbd1" containerID="1a0d6646183708363d9520088f95cd8816576e2f6a933c45eba30fd797b4ff15" exitCode=60
Oct 30 13:15:05.642795 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:05.642760    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq" event={"ID":"674c045d-1ba0-4075-8ce8-84e3b275dbd1","Type":"ContainerDied","Data":"1a0d6646183708363d9520088f95cd8816576e2f6a933c45eba30fd797b4ff15"}
Oct 30 13:15:05.643496 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:05.643470    4544 scope.go:122] "RemoveContainer" containerID="1a0d6646183708363d9520088f95cd8816576e2f6a933c45eba30fd797b4ff15"
Oct 30 13:15:05.671242 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:15:05.651602    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq_odh-ai-gateway-infra(674c045d-1ba0-4075-8ce8-84e3b275dbd1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq" podUID="674c045d-1ba0-4075-8ce8-84e3b275dbd1"
Oct 30 13:15:05.705541 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:05.703495    4544 generic.go:356] "Generic (PLEG): container finished" podID="19f3a006-19e8-44af-9db0-366d400001f3" containerID="b0b76ba094471c5282b513605476500222e221bc386d4f51ab294539c657df99" exitCode=60
Oct 30 13:15:05.705541 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:05.703960    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7" event={"ID":"19f3a006-19e8-44af-9db0-366d400001f3","Type":"ContainerDied","Data":"b0b76ba094471c5282b513605476500222e221bc386d4f51ab294539c657df99"}
Oct 30 13:15:05.705541 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:05.704084    4544 scope.go:122] "RemoveContainer" containerID="b0b76ba094471c5282b513605476500222e221bc386d4f51ab294539c657df99"
Oct 30 13:15:05.705541 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:15:05.704594    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7_odh-ai-gateway-infra(19f3a006-19e8-44af-9db0-366d400001f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7" podUID="19f3a006-19e8-44af-9db0-366d400001f3"
Oct 30 13:15:05.782248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:05.776124    4544 scope.go:122] "RemoveContainer" containerID="5c3c2c196b81bc7dd431370b71f502ca129c27914a158d13d60cb89960a3ffc7"
Oct 30 13:15:05.789797 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:05.786900    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f" event={"ID":"72837ba1-2075-4553-a926-3716a21270ee","Type":"ContainerStarted","Data":"487645a9e5d35daf01ff0e512c852081f40a5c343b5ab1033cee8829c5a8f1d3"}
Oct 30 13:15:05.813775 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:05.813682    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs" event={"ID":"d6ae22b0-e9c0-45bc-85e4-76c659da2808","Type":"ContainerStarted","Data":"d1261cd5c6e6d688918e962365b90bd3735dbebc25a7ff9f4cfe97c52e360b19"}
Oct 30 13:15:05.846447 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:05.842733    4544 generic.go:356] "Generic (PLEG): container finished" podID="002634ca-d404-4827-b3e7-d007555a2792" containerID="ecdc9d99b6ced5d7d23f76b0419b875653a786d1ca2835aaf9291e677c6b2e1f" exitCode=60
Oct 30 13:15:05.846447 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:05.842879    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj" event={"ID":"002634ca-d404-4827-b3e7-d007555a2792","Type":"ContainerDied","Data":"ecdc9d99b6ced5d7d23f76b0419b875653a786d1ca2835aaf9291e677c6b2e1f"}
Oct 30 13:15:05.846447 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:05.845424    4544 scope.go:122] "RemoveContainer" containerID="ecdc9d99b6ced5d7d23f76b0419b875653a786d1ca2835aaf9291e677c6b2e1f"
Oct 30 13:15:05.846447 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:15:05.845934    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj_odh-ai-gateway-infra(002634ca-d404-4827-b3e7-d007555a2792)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj" podUID="002634ca-d404-4827-b3e7-d007555a2792"
Oct 30 13:15:05.983111 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:05.981409    4544 scope.go:122] "RemoveContainer" containerID="878c949d1b02e68c9166dba6d763bdaf49200d01bf77a7693f2c835f6a6c4f98"
Oct 30 13:15:06.171554 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:06.171525    4544 scope.go:122] "RemoveContainer" containerID="b6854f1fb9842c1ee0aaf135458ec6cff2c579190a6d6cb186777778a41f57f6"
Oct 30 13:15:06.709256 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:06.691136    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:06.922115 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:06.922080    4544 generic.go:356] "Generic (PLEG): container finished" podID="1ddc624d-f00d-4c01-b9e1-877259555186" containerID="5b76ce30fb65a60949bada8da9068dc231ff0615b274eeea46932d84169b1009" exitCode=60
Oct 30 13:15:06.922533 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:06.922508    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq" event={"ID":"1ddc624d-f00d-4c01-b9e1-877259555186","Type":"ContainerDied","Data":"5b76ce30fb65a60949bada8da9068dc231ff0615b274eeea46932d84169b1009"}
Oct 30 13:15:06.926543 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:06.924618    4544 scope.go:122] "RemoveContainer" containerID="5b76ce30fb65a60949bada8da9068dc231ff0615b274eeea46932d84169b1009"
Oct 30 13:15:06.927214 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:15:06.927182    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq_odh-ai-gateway-infra(1ddc624d-f00d-4c01-b9e1-877259555186)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq" podUID="1ddc624d-f00d-4c01-b9e1-877259555186"
Oct 30 13:15:06.929944 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:06.929921    4544 scope.go:122] "RemoveContainer" containerID="f4589ab2835bf9e7fb86bc6fed1da2f6128daeaaac307ad988e0b8e9bd920402"
Oct 30 13:15:06.956110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:06.956070    4544 generic.go:356] "Generic (PLEG): container finished" podID="ac81161c-cd40-46f4-9fe0-2490f472393c" containerID="f417e02048a1b64e11a66bfa085bde462f9da5399aa78a0d6c134b809ff6a2fd" exitCode=60
Oct 30 13:15:06.956316 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:06.956206    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq" event={"ID":"ac81161c-cd40-46f4-9fe0-2490f472393c","Type":"ContainerDied","Data":"f417e02048a1b64e11a66bfa085bde462f9da5399aa78a0d6c134b809ff6a2fd"}
Oct 30 13:15:06.956805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:06.956784    4544 scope.go:122] "RemoveContainer" containerID="f417e02048a1b64e11a66bfa085bde462f9da5399aa78a0d6c134b809ff6a2fd"
Oct 30 13:15:06.957414 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:15:06.957311    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq_odh-ai-gateway-infra(ac81161c-cd40-46f4-9fe0-2490f472393c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq" podUID="ac81161c-cd40-46f4-9fe0-2490f472393c"
Oct 30 13:15:06.960548 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:06.960475    4544 generic.go:356] "Generic (PLEG): container finished" podID="03d3f1b3-4d1c-4aca-82d5-f106740125ad" containerID="0c24808e7684626ed1060df0158e5112fbf9d28f24b93d5846672ba7fb4c294a" exitCode=60
Oct 30 13:15:06.960667 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:06.960595    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l" event={"ID":"03d3f1b3-4d1c-4aca-82d5-f106740125ad","Type":"ContainerDied","Data":"0c24808e7684626ed1060df0158e5112fbf9d28f24b93d5846672ba7fb4c294a"}
Oct 30 13:15:06.961298 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:06.961276    4544 scope.go:122] "RemoveContainer" containerID="0c24808e7684626ed1060df0158e5112fbf9d28f24b93d5846672ba7fb4c294a"
Oct 30 13:15:06.961816 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:15:06.961789    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l_odh-ai-gateway-infra(03d3f1b3-4d1c-4aca-82d5-f106740125ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l" podUID="03d3f1b3-4d1c-4aca-82d5-f106740125ad"
Oct 30 13:15:06.964814 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:06.964789    4544 scope.go:122] "RemoveContainer" containerID="1a0d6646183708363d9520088f95cd8816576e2f6a933c45eba30fd797b4ff15"
Oct 30 13:15:06.965290 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:15:06.965263    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq_odh-ai-gateway-infra(674c045d-1ba0-4075-8ce8-84e3b275dbd1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq" podUID="674c045d-1ba0-4075-8ce8-84e3b275dbd1"
Oct 30 13:15:06.968903 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:06.968873    4544 scope.go:122] "RemoveContainer" containerID="b0b76ba094471c5282b513605476500222e221bc386d4f51ab294539c657df99"
Oct 30 13:15:06.969487 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:15:06.969352    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7_odh-ai-gateway-infra(19f3a006-19e8-44af-9db0-366d400001f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7" podUID="19f3a006-19e8-44af-9db0-366d400001f3"
Oct 30 13:15:06.971719 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:06.971675    4544 generic.go:356] "Generic (PLEG): container finished" podID="72837ba1-2075-4553-a926-3716a21270ee" containerID="487645a9e5d35daf01ff0e512c852081f40a5c343b5ab1033cee8829c5a8f1d3" exitCode=60
Oct 30 13:15:06.984748 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:06.983969    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f" event={"ID":"72837ba1-2075-4553-a926-3716a21270ee","Type":"ContainerDied","Data":"487645a9e5d35daf01ff0e512c852081f40a5c343b5ab1033cee8829c5a8f1d3"}
Oct 30 13:15:06.984748 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:06.984350    4544 scope.go:122] "RemoveContainer" containerID="487645a9e5d35daf01ff0e512c852081f40a5c343b5ab1033cee8829c5a8f1d3"
Oct 30 13:15:06.985014 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:15:06.984930    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f_odh-ai-gateway-infra(72837ba1-2075-4553-a926-3716a21270ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f" podUID="72837ba1-2075-4553-a926-3716a21270ee"
Oct 30 13:15:06.987582 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:06.987400    4544 scope.go:122] "RemoveContainer" containerID="53bc712b1edbb69ef844ed2bbe24fbdd63b32fa426095873bcb647dcf39aaca2"
Oct 30 13:15:06.990638 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:06.990607    4544 generic.go:356] "Generic (PLEG): container finished" podID="d6ae22b0-e9c0-45bc-85e4-76c659da2808" containerID="d1261cd5c6e6d688918e962365b90bd3735dbebc25a7ff9f4cfe97c52e360b19" exitCode=60
Oct 30 13:15:06.990761 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:06.990719    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs" event={"ID":"d6ae22b0-e9c0-45bc-85e4-76c659da2808","Type":"ContainerDied","Data":"d1261cd5c6e6d688918e962365b90bd3735dbebc25a7ff9f4cfe97c52e360b19"}
Oct 30 13:15:06.991278 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:06.991253    4544 scope.go:122] "RemoveContainer" containerID="d1261cd5c6e6d688918e962365b90bd3735dbebc25a7ff9f4cfe97c52e360b19"
Oct 30 13:15:06.991794 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:15:06.991755    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs_odh-ai-gateway-infra(d6ae22b0-e9c0-45bc-85e4-76c659da2808)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs" podUID="d6ae22b0-e9c0-45bc-85e4-76c659da2808"
Oct 30 13:15:06.998282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:06.997666    4544 scope.go:122] "RemoveContainer" containerID="ecdc9d99b6ced5d7d23f76b0419b875653a786d1ca2835aaf9291e677c6b2e1f"
Oct 30 13:15:06.998282 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:15:06.998162    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj_odh-ai-gateway-infra(002634ca-d404-4827-b3e7-d007555a2792)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj" podUID="002634ca-d404-4827-b3e7-d007555a2792"
Oct 30 13:15:07.004689 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:07.001491    4544 scope.go:122] "RemoveContainer" containerID="850cde6b6cb3e239554e28f8bdc8557f080e3fc995ace6b29820d6c745f2be59"
Oct 30 13:15:07.004689 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:15:07.001957    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx_odh-ai-gateway-infra(32e6a633-93f2-48be-87d6-5a9521132614)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx" podUID="32e6a633-93f2-48be-87d6-5a9521132614"
Oct 30 13:15:07.014790 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:07.014750    4544 scope.go:122] "RemoveContainer" containerID="3f35da18db4631aee7e81b02ef622d390de421844f0176faedd34615e98ad553"
Oct 30 13:15:07.039881 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:07.039844    4544 scope.go:122] "RemoveContainer" containerID="88f01120e070ef0fd7e6970939587038b3a6d4e75bdc47a40bc5f43f030d972b"
Oct 30 13:15:07.064590 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:07.064327    4544 scope.go:122] "RemoveContainer" containerID="b52c77893ea730ba8695b8e8139234f6fecb11bfe0ec95776a64ab7be51eb1c2"
Oct 30 13:15:07.401250 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:07.399019    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:07.597723 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:07.597680    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:07.994251 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:07.974626    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:07.994251 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:07.976152    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"]
Oct 30 13:15:07.994251 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:07.976461    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr" podUID="4bbc446d-8807-4613-9d86-b87b0733d2cf" containerName="istio-proxy" containerID="cri-o://63c5590fda351734414b3bea73c33d5731093bf2b3f65a1d9873980323188810" gracePeriod=30
Oct 30 13:15:08.307523 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:08.307428    4544 scope.go:122] "RemoveContainer" containerID="5b76ce30fb65a60949bada8da9068dc231ff0615b274eeea46932d84169b1009"
Oct 30 13:15:08.333277 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:08.317635    4544 scope.go:122] "RemoveContainer" containerID="f417e02048a1b64e11a66bfa085bde462f9da5399aa78a0d6c134b809ff6a2fd"
Oct 30 13:15:08.333277 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:15:08.318117    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq_odh-ai-gateway-infra(ac81161c-cd40-46f4-9fe0-2490f472393c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq" podUID="ac81161c-cd40-46f4-9fe0-2490f472393c"
Oct 30 13:15:08.333277 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:08.321326    4544 scope.go:122] "RemoveContainer" containerID="0c24808e7684626ed1060df0158e5112fbf9d28f24b93d5846672ba7fb4c294a"
Oct 30 13:15:08.333277 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:15:08.321788    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l_odh-ai-gateway-infra(03d3f1b3-4d1c-4aca-82d5-f106740125ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l" podUID="03d3f1b3-4d1c-4aca-82d5-f106740125ad"
Oct 30 13:15:08.334357 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:15:08.334323    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq_odh-ai-gateway-infra(1ddc624d-f00d-4c01-b9e1-877259555186)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq" podUID="1ddc624d-f00d-4c01-b9e1-877259555186"
Oct 30 13:15:08.460235 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:08.460184    4544 scope.go:122] "RemoveContainer" containerID="487645a9e5d35daf01ff0e512c852081f40a5c343b5ab1033cee8829c5a8f1d3"
Oct 30 13:15:08.481982 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:15:08.481934    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f_odh-ai-gateway-infra(72837ba1-2075-4553-a926-3716a21270ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f" podUID="72837ba1-2075-4553-a926-3716a21270ee"
Oct 30 13:15:08.521615 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:08.521582    4544 scope.go:122] "RemoveContainer" containerID="d1261cd5c6e6d688918e962365b90bd3735dbebc25a7ff9f4cfe97c52e360b19"
Oct 30 13:15:08.544693 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:15:08.544652    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs_odh-ai-gateway-infra(d6ae22b0-e9c0-45bc-85e4-76c659da2808)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs" podUID="d6ae22b0-e9c0-45bc-85e4-76c659da2808"
Oct 30 13:15:08.759250 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:08.759184    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:15:08.794350 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:08.794312    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-716b4c-qqfdl"]
Oct 30 13:15:08.796025 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:08.795992    4544 cpu_manager.go:414] "containerMap: removing container" podUID="f57ae3c0-2e03-4252-bc0a-608b3507e96a" containerName="payload-processing"
Oct 30 13:15:08.873368 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:08.873320    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f57ae3c0-2e03-4252-bc0a-608b3507e96a" containerName="payload-processing"
Oct 30 13:15:08.886409 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:08.886375    4544 cpu_manager.go:414] "containerMap: removing container" podUID="caa19642-0691-40fe-aba4-33199f334f45" containerName="payload-pre-processing"
Oct 30 13:15:08.886712 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:08.886689    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caa19642-0691-40fe-aba4-33199f334f45" containerName="payload-pre-processing"
Oct 30 13:15:08.887159 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:08.887137    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="f57ae3c0-2e03-4252-bc0a-608b3507e96a" containerName="payload-processing"
Oct 30 13:15:08.921293 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:08.917335    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="caa19642-0691-40fe-aba4-33199f334f45" containerName="payload-pre-processing"
Oct 30 13:15:08.945204 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:08.945161    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:15:09.003111 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:09.003068    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:09.004156 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:09.004132    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-716b4c-qqfdl"]
Oct 30 13:15:09.023104 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:09.020316    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:15:09.023668 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:09.020503    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-716b4c-qqfdl"
Oct 30 13:15:09.069246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:09.060407    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:15:09.124828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:09.124777    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75ec2a60-7de0-4cff-9961-989ee7c2961d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-716b4c-qqfdl\" (UID: \"75ec2a60-7de0-4cff-9961-989ee7c2961d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-716b4c-qqfdl"
Oct 30 13:15:09.125044 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:09.124881    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75ec2a60-7de0-4cff-9961-989ee7c2961d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-716b4c-qqfdl\" (UID: \"75ec2a60-7de0-4cff-9961-989ee7c2961d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-716b4c-qqfdl"
Oct 30 13:15:09.125044 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:09.124950    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chprr\" (UniqueName: \"kubernetes.io/projected/75ec2a60-7de0-4cff-9961-989ee7c2961d-kube-api-access-chprr\") pod \"maas-api-revoke-keys-e2e-api-b-716b4c-qqfdl\" (UID: \"75ec2a60-7de0-4cff-9961-989ee7c2961d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-716b4c-qqfdl"
Oct 30 13:15:09.270246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:09.269751    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75ec2a60-7de0-4cff-9961-989ee7c2961d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-716b4c-qqfdl\" (UID: \"75ec2a60-7de0-4cff-9961-989ee7c2961d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-716b4c-qqfdl"
Oct 30 13:15:09.270246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:09.269875    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75ec2a60-7de0-4cff-9961-989ee7c2961d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-716b4c-qqfdl\" (UID: \"75ec2a60-7de0-4cff-9961-989ee7c2961d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-716b4c-qqfdl"
Oct 30 13:15:09.270246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:09.269952    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chprr\" (UniqueName: \"kubernetes.io/projected/75ec2a60-7de0-4cff-9961-989ee7c2961d-kube-api-access-chprr\") pod \"maas-api-revoke-keys-e2e-api-b-716b4c-qqfdl\" (UID: \"75ec2a60-7de0-4cff-9961-989ee7c2961d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-716b4c-qqfdl"
Oct 30 13:15:09.358274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:09.337983    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75ec2a60-7de0-4cff-9961-989ee7c2961d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-716b4c-qqfdl\" (UID: \"75ec2a60-7de0-4cff-9961-989ee7c2961d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-716b4c-qqfdl"
Oct 30 13:15:09.397746 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:09.397673    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75ec2a60-7de0-4cff-9961-989ee7c2961d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-716b4c-qqfdl\" (UID: \"75ec2a60-7de0-4cff-9961-989ee7c2961d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-716b4c-qqfdl"
Oct 30 13:15:09.418244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:09.414156    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chprr\" (UniqueName: \"kubernetes.io/projected/75ec2a60-7de0-4cff-9961-989ee7c2961d-kube-api-access-chprr\") pod \"maas-api-revoke-keys-e2e-api-b-716b4c-qqfdl\" (UID: \"75ec2a60-7de0-4cff-9961-989ee7c2961d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-716b4c-qqfdl"
Oct 30 13:15:09.517246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:09.515292    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:15:09.642298 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:09.641603    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-716b4c-qqfdl"
Oct 30 13:15:09.870628 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:09.870582    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 13:15:09.872635 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:09.872595    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:15:09.872966 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:09.872945    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:15:09.918298 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:09.917739    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:15:10.530577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:10.530545    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-716b4c-qqfdl"]
Oct 30 13:15:10.531067 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:15:10.531040    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75ec2a60_7de0_4cff_9961_989ee7c2961d.slice/crio-fb2606a165de2e438cbb0df82e994567e9805061af9a899a8aab9d89061f8610 WatchSource:0}: Error finding container fb2606a165de2e438cbb0df82e994567e9805061af9a899a8aab9d89061f8610: Status 404 returned error can't find the container with id fb2606a165de2e438cbb0df82e994567e9805061af9a899a8aab9d89061f8610
Oct 30 13:15:10.601750 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:10.601717    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-716b4c-qqfdl" event={"ID":"75ec2a60-7de0-4cff-9961-989ee7c2961d","Type":"ContainerStarted","Data":"fb2606a165de2e438cbb0df82e994567e9805061af9a899a8aab9d89061f8610"}
Oct 30 13:15:11.613757 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:11.613717    4544 generic.go:356] "Generic (PLEG): container finished" podID="75ec2a60-7de0-4cff-9961-989ee7c2961d" containerID="73c3209fcc1f2274c34495595d4d68935c81b1a81c120141307ffe01b22b49f0" exitCode=0
Oct 30 13:15:11.614149 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:11.613914    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-716b4c-qqfdl" event={"ID":"75ec2a60-7de0-4cff-9961-989ee7c2961d","Type":"ContainerDied","Data":"73c3209fcc1f2274c34495595d4d68935c81b1a81c120141307ffe01b22b49f0"}
Oct 30 13:15:11.631295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:11.631254    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:12.809092 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:12.809050    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-716b4c-qqfdl"
Oct 30 13:15:12.923571 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:12.923474    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75ec2a60-7de0-4cff-9961-989ee7c2961d-kube-api-access-chprr\" (UniqueName: \"kubernetes.io/projected/75ec2a60-7de0-4cff-9961-989ee7c2961d-kube-api-access-chprr\") pod \"75ec2a60-7de0-4cff-9961-989ee7c2961d\" (UID: \"75ec2a60-7de0-4cff-9961-989ee7c2961d\") "
Oct 30 13:15:12.925452 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:12.923574    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75ec2a60-7de0-4cff-9961-989ee7c2961d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75ec2a60-7de0-4cff-9961-989ee7c2961d-maas-api-xks-ca\") pod \"75ec2a60-7de0-4cff-9961-989ee7c2961d\" (UID: \"75ec2a60-7de0-4cff-9961-989ee7c2961d\") "
Oct 30 13:15:12.925452 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:12.923736    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75ec2a60-7de0-4cff-9961-989ee7c2961d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75ec2a60-7de0-4cff-9961-989ee7c2961d-maas-api-openshift-ca\") pod \"75ec2a60-7de0-4cff-9961-989ee7c2961d\" (UID: \"75ec2a60-7de0-4cff-9961-989ee7c2961d\") "
Oct 30 13:15:12.925452 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:12.924365    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75ec2a60-7de0-4cff-9961-989ee7c2961d-maas-api-openshift-ca" pod "75ec2a60-7de0-4cff-9961-989ee7c2961d" (UID: "75ec2a60-7de0-4cff-9961-989ee7c2961d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:12.925452 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:12.924605    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75ec2a60-7de0-4cff-9961-989ee7c2961d-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:12.935167 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:12.926627    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75ec2a60-7de0-4cff-9961-989ee7c2961d-maas-api-xks-ca" pod "75ec2a60-7de0-4cff-9961-989ee7c2961d" (UID: "75ec2a60-7de0-4cff-9961-989ee7c2961d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:15:12.936508 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:12.936475    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ec2a60-7de0-4cff-9961-989ee7c2961d-kube-api-access-chprr" pod "75ec2a60-7de0-4cff-9961-989ee7c2961d" (UID: "75ec2a60-7de0-4cff-9961-989ee7c2961d"). InnerVolumeSpecName "kube-api-access-chprr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:13.026631 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.026598    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chprr\" (UniqueName: \"kubernetes.io/projected/75ec2a60-7de0-4cff-9961-989ee7c2961d-kube-api-access-chprr\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:13.026631 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.026628    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75ec2a60-7de0-4cff-9961-989ee7c2961d-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:13.423967 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.423939    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:15:13.534788 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.534684    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-workload-socket\") pod \"4bbc446d-8807-4613-9d86-b87b0733d2cf\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") "
Oct 30 13:15:13.534962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.534877    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bbc446d-8807-4613-9d86-b87b0733d2cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bbc446d-8807-4613-9d86-b87b0733d2cf-istiod-ca-cert\") pod \"4bbc446d-8807-4613-9d86-b87b0733d2cf\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") "
Oct 30 13:15:13.534962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.534931    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-credential-socket\") pod \"4bbc446d-8807-4613-9d86-b87b0733d2cf\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") "
Oct 30 13:15:13.535094 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.534973    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-workload-socket" pod "4bbc446d-8807-4613-9d86-b87b0733d2cf" (UID: "4bbc446d-8807-4613-9d86-b87b0733d2cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:13.535094 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.535003    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bbc446d-8807-4613-9d86-b87b0733d2cf-istio-token\" (UniqueName: \"kubernetes.io/projected/4bbc446d-8807-4613-9d86-b87b0733d2cf-istio-token\") pod \"4bbc446d-8807-4613-9d86-b87b0733d2cf\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") "
Oct 30 13:15:13.535094 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.535057    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-workload-certs\") pod \"4bbc446d-8807-4613-9d86-b87b0733d2cf\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") "
Oct 30 13:15:13.535094 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.535078    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-istio-data\") pod \"4bbc446d-8807-4613-9d86-b87b0733d2cf\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") "
Oct 30 13:15:13.535331 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.535121    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-istio-envoy\") pod \"4bbc446d-8807-4613-9d86-b87b0733d2cf\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") "
Oct 30 13:15:13.535331 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.535190    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4bbc446d-8807-4613-9d86-b87b0733d2cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bbc446d-8807-4613-9d86-b87b0733d2cf-istio-podinfo\") pod \"4bbc446d-8807-4613-9d86-b87b0733d2cf\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") "
Oct 30 13:15:13.535331 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.535214    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-credential-socket" pod "4bbc446d-8807-4613-9d86-b87b0733d2cf" (UID: "4bbc446d-8807-4613-9d86-b87b0733d2cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:13.535331 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.535291    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bbc446d-8807-4613-9d86-b87b0733d2cf-kube-api-access-2pltd\" (UniqueName: \"kubernetes.io/projected/4bbc446d-8807-4613-9d86-b87b0733d2cf-kube-api-access-2pltd\") pod \"4bbc446d-8807-4613-9d86-b87b0733d2cf\" (UID: \"4bbc446d-8807-4613-9d86-b87b0733d2cf\") "
Oct 30 13:15:13.535331 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.535295    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bbc446d-8807-4613-9d86-b87b0733d2cf-istiod-ca-cert" pod "4bbc446d-8807-4613-9d86-b87b0733d2cf" (UID: "4bbc446d-8807-4613-9d86-b87b0733d2cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:13.535598 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.535429    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-istio-data" pod "4bbc446d-8807-4613-9d86-b87b0733d2cf" (UID: "4bbc446d-8807-4613-9d86-b87b0733d2cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:13.535662 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.535621    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-workload-certs" pod "4bbc446d-8807-4613-9d86-b87b0733d2cf" (UID: "4bbc446d-8807-4613-9d86-b87b0733d2cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:13.536297 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.536267    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bbc446d-8807-4613-9d86-b87b0733d2cf-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:13.536407 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.536299    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:13.536407 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.536319    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:13.536407 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.536336    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:13.536407 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.536350    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:13.537879 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.537854    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bbc446d-8807-4613-9d86-b87b0733d2cf-istio-podinfo" pod "4bbc446d-8807-4613-9d86-b87b0733d2cf" (UID: "4bbc446d-8807-4613-9d86-b87b0733d2cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:15:13.538056 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.538027    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-istio-envoy" pod "4bbc446d-8807-4613-9d86-b87b0733d2cf" (UID: "4bbc446d-8807-4613-9d86-b87b0733d2cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:13.538197 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.538178    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bbc446d-8807-4613-9d86-b87b0733d2cf-istio-token" pod "4bbc446d-8807-4613-9d86-b87b0733d2cf" (UID: "4bbc446d-8807-4613-9d86-b87b0733d2cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:13.538861 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.538832    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bbc446d-8807-4613-9d86-b87b0733d2cf-kube-api-access-2pltd" pod "4bbc446d-8807-4613-9d86-b87b0733d2cf" (UID: "4bbc446d-8807-4613-9d86-b87b0733d2cf"). InnerVolumeSpecName "kube-api-access-2pltd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:13.595388 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.595329    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:13.680474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.677659    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bbc446d-8807-4613-9d86-b87b0733d2cf-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:13.684784 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.684712    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bbc446d-8807-4613-9d86-b87b0733d2cf-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:13.684784 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.684789    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bbc446d-8807-4613-9d86-b87b0733d2cf-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:13.685013 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.684839    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pltd\" (UniqueName: \"kubernetes.io/projected/4bbc446d-8807-4613-9d86-b87b0733d2cf-kube-api-access-2pltd\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:13.700102 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.700058    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-716b4c-qqfdl" event={"ID":"75ec2a60-7de0-4cff-9961-989ee7c2961d","Type":"ContainerDied","Data":"fb2606a165de2e438cbb0df82e994567e9805061af9a899a8aab9d89061f8610"}
Oct 30 13:15:13.700306 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.700163    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb2606a165de2e438cbb0df82e994567e9805061af9a899a8aab9d89061f8610"
Oct 30 13:15:13.700306 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.700271    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-716b4c-qqfdl"
Oct 30 13:15:13.703431 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.703397    4544 generic.go:356] "Generic (PLEG): container finished" podID="4bbc446d-8807-4613-9d86-b87b0733d2cf" containerID="63c5590fda351734414b3bea73c33d5731093bf2b3f65a1d9873980323188810" exitCode=0
Oct 30 13:15:13.703571 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.703432    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr" event={"ID":"4bbc446d-8807-4613-9d86-b87b0733d2cf","Type":"ContainerDied","Data":"63c5590fda351734414b3bea73c33d5731093bf2b3f65a1d9873980323188810"}
Oct 30 13:15:13.703571 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.703465    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"
Oct 30 13:15:13.703571 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.703501    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr" event={"ID":"4bbc446d-8807-4613-9d86-b87b0733d2cf","Type":"ContainerDied","Data":"f4e098b4abd7e9df6b3be4ff723f2262afe279966bc2d3e2276e8ebf858bd34f"}
Oct 30 13:15:13.703571 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.703533    4544 scope.go:122] "RemoveContainer" containerID="63c5590fda351734414b3bea73c33d5731093bf2b3f65a1d9873980323188810"
Oct 30 13:15:13.719100 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.719068    4544 scope.go:122] "RemoveContainer" containerID="13fd4d84b6bf12c6d6f835bd82bf43d5eb7acd38a82180c57f66e497c0fea67d"
Oct 30 13:15:13.732676 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.732638    4544 scope.go:122] "RemoveContainer" containerID="63c5590fda351734414b3bea73c33d5731093bf2b3f65a1d9873980323188810"
Oct 30 13:15:13.732970 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:15:13.732935    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63c5590fda351734414b3bea73c33d5731093bf2b3f65a1d9873980323188810\": container with ID starting with 63c5590fda351734414b3bea73c33d5731093bf2b3f65a1d9873980323188810 not found: ID does not exist" containerID="63c5590fda351734414b3bea73c33d5731093bf2b3f65a1d9873980323188810"
Oct 30 13:15:13.733122 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.732982    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63c5590fda351734414b3bea73c33d5731093bf2b3f65a1d9873980323188810"} err="failed to get container status \"63c5590fda351734414b3bea73c33d5731093bf2b3f65a1d9873980323188810\": rpc error: code = NotFound desc = could not find container \"63c5590fda351734414b3bea73c33d5731093bf2b3f65a1d9873980323188810\": container with ID starting with 63c5590fda351734414b3bea73c33d5731093bf2b3f65a1d9873980323188810 not found: ID does not exist"
Oct 30 13:15:13.733122 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.733003    4544 scope.go:122] "RemoveContainer" containerID="13fd4d84b6bf12c6d6f835bd82bf43d5eb7acd38a82180c57f66e497c0fea67d"
Oct 30 13:15:13.733280 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:15:13.733260    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13fd4d84b6bf12c6d6f835bd82bf43d5eb7acd38a82180c57f66e497c0fea67d\": container with ID starting with 13fd4d84b6bf12c6d6f835bd82bf43d5eb7acd38a82180c57f66e497c0fea67d not found: ID does not exist" containerID="13fd4d84b6bf12c6d6f835bd82bf43d5eb7acd38a82180c57f66e497c0fea67d"
Oct 30 13:15:13.733340 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.733292    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13fd4d84b6bf12c6d6f835bd82bf43d5eb7acd38a82180c57f66e497c0fea67d"} err="failed to get container status \"13fd4d84b6bf12c6d6f835bd82bf43d5eb7acd38a82180c57f66e497c0fea67d\": rpc error: code = NotFound desc = could not find container \"13fd4d84b6bf12c6d6f835bd82bf43d5eb7acd38a82180c57f66e497c0fea67d\": container with ID starting with 13fd4d84b6bf12c6d6f835bd82bf43d5eb7acd38a82180c57f66e497c0fea67d not found: ID does not exist"
Oct 30 13:15:13.768601 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.768563    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"]
Oct 30 13:15:13.782465 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:13.782435    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr"]
Oct 30 13:15:15.098864 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:15.098814    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bbc446d-8807-4613-9d86-b87b0733d2cf" path="/var/lib/kubelet/pods/4bbc446d-8807-4613-9d86-b87b0733d2cf/volumes"
Oct 30 13:15:15.579529 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:15.579480    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:16.078961 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:16.078929    4544 scope.go:122] "RemoveContainer" containerID="850cde6b6cb3e239554e28f8bdc8557f080e3fc995ace6b29820d6c745f2be59"
Oct 30 13:15:16.490144 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:16.490056    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:16.623238 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:16.622789    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:16.733935 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:16.733897    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx" event={"ID":"32e6a633-93f2-48be-87d6-5a9521132614","Type":"ContainerStarted","Data":"d384bf20476e640210260162816ce13a4ca5531418e5335502d2aa08f91fc5da"}
Oct 30 13:15:17.083576 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:17.081843    4544 scope.go:122] "RemoveContainer" containerID="1a0d6646183708363d9520088f95cd8816576e2f6a933c45eba30fd797b4ff15"
Oct 30 13:15:17.083576 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:17.083003    4544 scope.go:122] "RemoveContainer" containerID="0c24808e7684626ed1060df0158e5112fbf9d28f24b93d5846672ba7fb4c294a"
Oct 30 13:15:17.100253 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:17.097857    4544 scope.go:122] "RemoveContainer" containerID="b0b76ba094471c5282b513605476500222e221bc386d4f51ab294539c657df99"
Oct 30 13:15:17.607625 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:17.607518    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:17.745152 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:17.745105    4544 generic.go:356] "Generic (PLEG): container finished" podID="32e6a633-93f2-48be-87d6-5a9521132614" containerID="d384bf20476e640210260162816ce13a4ca5531418e5335502d2aa08f91fc5da" exitCode=60
Oct 30 13:15:17.745416 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:17.745195    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx" event={"ID":"32e6a633-93f2-48be-87d6-5a9521132614","Type":"ContainerDied","Data":"d384bf20476e640210260162816ce13a4ca5531418e5335502d2aa08f91fc5da"}
Oct 30 13:15:17.745416 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:17.745350    4544 scope.go:122] "RemoveContainer" containerID="850cde6b6cb3e239554e28f8bdc8557f080e3fc995ace6b29820d6c745f2be59"
Oct 30 13:15:17.745819 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:17.745796    4544 scope.go:122] "RemoveContainer" containerID="d384bf20476e640210260162816ce13a4ca5531418e5335502d2aa08f91fc5da"
Oct 30 13:15:17.746287 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:15:17.746258    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx_odh-ai-gateway-infra(32e6a633-93f2-48be-87d6-5a9521132614)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx" podUID="32e6a633-93f2-48be-87d6-5a9521132614"
Oct 30 13:15:17.788081 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:17.788047    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx"]
Oct 30 13:15:18.078589 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:18.078558    4544 scope.go:122] "RemoveContainer" containerID="487645a9e5d35daf01ff0e512c852081f40a5c343b5ab1033cee8829c5a8f1d3"
Oct 30 13:15:18.079192 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:18.079167    4544 scope.go:122] "RemoveContainer" containerID="ecdc9d99b6ced5d7d23f76b0419b875653a786d1ca2835aaf9291e677c6b2e1f"
Oct 30 13:15:18.079519 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:18.079499    4544 scope.go:122] "RemoveContainer" containerID="d1261cd5c6e6d688918e962365b90bd3735dbebc25a7ff9f4cfe97c52e360b19"
Oct 30 13:15:18.079657 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:18.079598    4544 scope.go:122] "RemoveContainer" containerID="f417e02048a1b64e11a66bfa085bde462f9da5399aa78a0d6c134b809ff6a2fd"
Oct 30 13:15:18.757996 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:18.757963    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq" event={"ID":"ac81161c-cd40-46f4-9fe0-2490f472393c","Type":"ContainerStarted","Data":"15b180018e5d0fc3431881a5cd166b67b7fb43fc0fa86f90e514ebf68b1c1ebb"}
Oct 30 13:15:18.760042 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:18.760018    4544 generic.go:356] "Generic (PLEG): container finished" podID="03d3f1b3-4d1c-4aca-82d5-f106740125ad" containerID="0a6fd14dca6ec2205d77e8ecd4924715fd6bbcc5bd1f9bf503249a052ddcd233" exitCode=60
Oct 30 13:15:18.760156 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:18.760087    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l" event={"ID":"03d3f1b3-4d1c-4aca-82d5-f106740125ad","Type":"ContainerDied","Data":"0a6fd14dca6ec2205d77e8ecd4924715fd6bbcc5bd1f9bf503249a052ddcd233"}
Oct 30 13:15:18.760156 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:18.760141    4544 scope.go:122] "RemoveContainer" containerID="0c24808e7684626ed1060df0158e5112fbf9d28f24b93d5846672ba7fb4c294a"
Oct 30 13:15:18.760576 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:18.760552    4544 scope.go:122] "RemoveContainer" containerID="0a6fd14dca6ec2205d77e8ecd4924715fd6bbcc5bd1f9bf503249a052ddcd233"
Oct 30 13:15:18.760980 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:15:18.760960    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l_odh-ai-gateway-infra(03d3f1b3-4d1c-4aca-82d5-f106740125ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l" podUID="03d3f1b3-4d1c-4aca-82d5-f106740125ad"
Oct 30 13:15:18.762292 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:18.762268    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq" event={"ID":"674c045d-1ba0-4075-8ce8-84e3b275dbd1","Type":"ContainerStarted","Data":"84e2d1f9407202388d173b0981dc1ae2860ce715664e60b41e30e9549cd723e3"}
Oct 30 13:15:18.764247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:18.764211    4544 generic.go:356] "Generic (PLEG): container finished" podID="19f3a006-19e8-44af-9db0-366d400001f3" containerID="750784d15324dd5739d521034a4ca702b082637b2377aadb54f0d8b45ffdc223" exitCode=60
Oct 30 13:15:18.764351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:18.764244    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7" event={"ID":"19f3a006-19e8-44af-9db0-366d400001f3","Type":"ContainerDied","Data":"750784d15324dd5739d521034a4ca702b082637b2377aadb54f0d8b45ffdc223"}
Oct 30 13:15:18.764582 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:18.764564    4544 scope.go:122] "RemoveContainer" containerID="750784d15324dd5739d521034a4ca702b082637b2377aadb54f0d8b45ffdc223"
Oct 30 13:15:18.764921 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:15:18.764901    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7_odh-ai-gateway-infra(19f3a006-19e8-44af-9db0-366d400001f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7" podUID="19f3a006-19e8-44af-9db0-366d400001f3"
Oct 30 13:15:18.766568 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:18.766544    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f" event={"ID":"72837ba1-2075-4553-a926-3716a21270ee","Type":"ContainerStarted","Data":"d5a942f04c9150ba9b0f2d46e8c9b3d23c95c05a15e074ab000f5892bb4dde16"}
Oct 30 13:15:18.768900 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:18.768882    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs" event={"ID":"d6ae22b0-e9c0-45bc-85e4-76c659da2808","Type":"ContainerStarted","Data":"8421e2c40ee87975835c05f6aef096ce8e49f1a7dce40c7264fe3618ea60cf36"}
Oct 30 13:15:18.775800 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:18.775779    4544 scope.go:122] "RemoveContainer" containerID="b0b76ba094471c5282b513605476500222e221bc386d4f51ab294539c657df99"
Oct 30 13:15:18.946410 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:18.946384    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx"
Oct 30 13:15:19.093261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:19.093232    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32e6a633-93f2-48be-87d6-5a9521132614-kube-api-access-9qqkz\" (UniqueName: \"kubernetes.io/projected/32e6a633-93f2-48be-87d6-5a9521132614-kube-api-access-9qqkz\") pod \"32e6a633-93f2-48be-87d6-5a9521132614\" (UID: \"32e6a633-93f2-48be-87d6-5a9521132614\") "
Oct 30 13:15:19.093460 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:19.093341    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32e6a633-93f2-48be-87d6-5a9521132614-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e6a633-93f2-48be-87d6-5a9521132614-maas-api-service-ca\") pod \"32e6a633-93f2-48be-87d6-5a9521132614\" (UID: \"32e6a633-93f2-48be-87d6-5a9521132614\") "
Oct 30 13:15:19.093707 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:19.093678    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32e6a633-93f2-48be-87d6-5a9521132614-maas-api-service-ca" pod "32e6a633-93f2-48be-87d6-5a9521132614" (UID: "32e6a633-93f2-48be-87d6-5a9521132614"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:19.094063 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:19.094042    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e6a633-93f2-48be-87d6-5a9521132614-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:19.095394 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:19.095375    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32e6a633-93f2-48be-87d6-5a9521132614-kube-api-access-9qqkz" pod "32e6a633-93f2-48be-87d6-5a9521132614" (UID: "32e6a633-93f2-48be-87d6-5a9521132614"). InnerVolumeSpecName "kube-api-access-9qqkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:19.119615 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:19.119579    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh"]
Oct 30 13:15:19.196351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:19.196309    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qqkz\" (UniqueName: \"kubernetes.io/projected/32e6a633-93f2-48be-87d6-5a9521132614-kube-api-access-9qqkz\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:19.230319 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:19.230280    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:19.360663 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:19.360362    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:19.506321 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:19.506284    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:19.921439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:19.921405    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq"]
Oct 30 13:15:19.949804 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:19.949753    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx" event={"ID":"32e6a633-93f2-48be-87d6-5a9521132614","Type":"ContainerDied","Data":"9a6db18cdffee3fdb971cba56cdf31950a4a4cc4ffe5198730673bce39319844"}
Oct 30 13:15:19.967239 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:19.960176    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq" event={"ID":"ac81161c-cd40-46f4-9fe0-2490f472393c","Type":"ContainerDied","Data":"15b180018e5d0fc3431881a5cd166b67b7fb43fc0fa86f90e514ebf68b1c1ebb"}
Oct 30 13:15:19.968030 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:19.957746    4544 generic.go:356] "Generic (PLEG): container finished" podID="ac81161c-cd40-46f4-9fe0-2490f472393c" containerID="15b180018e5d0fc3431881a5cd166b67b7fb43fc0fa86f90e514ebf68b1c1ebb" exitCode=60
Oct 30 13:15:19.968491 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:19.950613    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx"
Oct 30 13:15:19.983491 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:19.967931    4544 scope.go:122] "RemoveContainer" containerID="d384bf20476e640210260162816ce13a4ca5531418e5335502d2aa08f91fc5da"
Oct 30 13:15:20.005512 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.004407    4544 generic.go:356] "Generic (PLEG): container finished" podID="674c045d-1ba0-4075-8ce8-84e3b275dbd1" containerID="84e2d1f9407202388d173b0981dc1ae2860ce715664e60b41e30e9549cd723e3" exitCode=60
Oct 30 13:15:20.005512 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.004519    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq" event={"ID":"674c045d-1ba0-4075-8ce8-84e3b275dbd1","Type":"ContainerDied","Data":"84e2d1f9407202388d173b0981dc1ae2860ce715664e60b41e30e9549cd723e3"}
Oct 30 13:15:20.005512 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.005054    4544 scope.go:122] "RemoveContainer" containerID="84e2d1f9407202388d173b0981dc1ae2860ce715664e60b41e30e9549cd723e3"
Oct 30 13:15:20.007114 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:15:20.006718    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq_odh-ai-gateway-infra(674c045d-1ba0-4075-8ce8-84e3b275dbd1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq" podUID="674c045d-1ba0-4075-8ce8-84e3b275dbd1"
Oct 30 13:15:20.020360 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.020338    4544 scope.go:122] "RemoveContainer" containerID="f417e02048a1b64e11a66bfa085bde462f9da5399aa78a0d6c134b809ff6a2fd"
Oct 30 13:15:20.022697 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.022667    4544 generic.go:356] "Generic (PLEG): container finished" podID="72837ba1-2075-4553-a926-3716a21270ee" containerID="d5a942f04c9150ba9b0f2d46e8c9b3d23c95c05a15e074ab000f5892bb4dde16" exitCode=60
Oct 30 13:15:20.022942 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.022922    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f" event={"ID":"72837ba1-2075-4553-a926-3716a21270ee","Type":"ContainerDied","Data":"d5a942f04c9150ba9b0f2d46e8c9b3d23c95c05a15e074ab000f5892bb4dde16"}
Oct 30 13:15:20.023460 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.023439    4544 scope.go:122] "RemoveContainer" containerID="d5a942f04c9150ba9b0f2d46e8c9b3d23c95c05a15e074ab000f5892bb4dde16"
Oct 30 13:15:20.029243 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:15:20.023917    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f_odh-ai-gateway-infra(72837ba1-2075-4553-a926-3716a21270ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f" podUID="72837ba1-2075-4553-a926-3716a21270ee"
Oct 30 13:15:20.032871 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.032611    4544 generic.go:356] "Generic (PLEG): container finished" podID="d6ae22b0-e9c0-45bc-85e4-76c659da2808" containerID="8421e2c40ee87975835c05f6aef096ce8e49f1a7dce40c7264fe3618ea60cf36" exitCode=60
Oct 30 13:15:20.033330 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.032751    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs" event={"ID":"d6ae22b0-e9c0-45bc-85e4-76c659da2808","Type":"ContainerDied","Data":"8421e2c40ee87975835c05f6aef096ce8e49f1a7dce40c7264fe3618ea60cf36"}
Oct 30 13:15:20.037406 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.035160    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7"]
Oct 30 13:15:20.041832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.041808    4544 scope.go:122] "RemoveContainer" containerID="8421e2c40ee87975835c05f6aef096ce8e49f1a7dce40c7264fe3618ea60cf36"
Oct 30 13:15:20.043406 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.041988    4544 generic.go:356] "Generic (PLEG): container finished" podID="002634ca-d404-4827-b3e7-d007555a2792" containerID="d4b511f471a1d2a5c254a68040e568b1072364afff91b4e447d151cdde0501b9" exitCode=60
Oct 30 13:15:20.043523 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.042014    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj" event={"ID":"002634ca-d404-4827-b3e7-d007555a2792","Type":"ContainerDied","Data":"d4b511f471a1d2a5c254a68040e568b1072364afff91b4e447d151cdde0501b9"}
Oct 30 13:15:20.044029 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.044007    4544 scope.go:122] "RemoveContainer" containerID="d4b511f471a1d2a5c254a68040e568b1072364afff91b4e447d151cdde0501b9"
Oct 30 13:15:20.044531 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:15:20.044500    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj_odh-ai-gateway-infra(002634ca-d404-4827-b3e7-d007555a2792)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj" podUID="002634ca-d404-4827-b3e7-d007555a2792"
Oct 30 13:15:20.046126 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:15:20.046066    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs_odh-ai-gateway-infra(d6ae22b0-e9c0-45bc-85e4-76c659da2808)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs" podUID="d6ae22b0-e9c0-45bc-85e4-76c659da2808"
Oct 30 13:15:20.049346 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.049277    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq"]
Oct 30 13:15:20.054585 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.054562    4544 scope.go:122] "RemoveContainer" containerID="1a0d6646183708363d9520088f95cd8816576e2f6a933c45eba30fd797b4ff15"
Oct 30 13:15:20.064990 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.064856    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs"]
Oct 30 13:15:20.081814 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.078417    4544 scope.go:122] "RemoveContainer" containerID="5b76ce30fb65a60949bada8da9068dc231ff0615b274eeea46932d84169b1009"
Oct 30 13:15:20.087244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.086667    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l"]
Oct 30 13:15:20.098492 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.098466    4544 scope.go:122] "RemoveContainer" containerID="487645a9e5d35daf01ff0e512c852081f40a5c343b5ab1033cee8829c5a8f1d3"
Oct 30 13:15:20.108298 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.108267    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f"]
Oct 30 13:15:20.116476 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.116445    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx"]
Oct 30 13:15:20.127867 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.127831    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7a3603-29889435-8rrhx"]
Oct 30 13:15:20.165894 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.165868    4544 scope.go:122] "RemoveContainer" containerID="d1261cd5c6e6d688918e962365b90bd3735dbebc25a7ff9f4cfe97c52e360b19"
Oct 30 13:15:20.197949 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.197903    4544 scope.go:122] "RemoveContainer" containerID="ecdc9d99b6ced5d7d23f76b0419b875653a786d1ca2835aaf9291e677c6b2e1f"
Oct 30 13:15:20.380755 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.380725    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq"
Oct 30 13:15:20.383772 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.383741    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7"
Oct 30 13:15:20.387938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.387913    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l"
Oct 30 13:15:20.436005 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.435976    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac81161c-cd40-46f4-9fe0-2490f472393c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac81161c-cd40-46f4-9fe0-2490f472393c-maas-api-service-ca\") pod \"ac81161c-cd40-46f4-9fe0-2490f472393c\" (UID: \"ac81161c-cd40-46f4-9fe0-2490f472393c\") "
Oct 30 13:15:20.436143 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.436053    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19f3a006-19e8-44af-9db0-366d400001f3-kube-api-access-8w287\" (UniqueName: \"kubernetes.io/projected/19f3a006-19e8-44af-9db0-366d400001f3-kube-api-access-8w287\") pod \"19f3a006-19e8-44af-9db0-366d400001f3\" (UID: \"19f3a006-19e8-44af-9db0-366d400001f3\") "
Oct 30 13:15:20.436215 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.436168    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19f3a006-19e8-44af-9db0-366d400001f3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19f3a006-19e8-44af-9db0-366d400001f3-maas-api-service-ca\") pod \"19f3a006-19e8-44af-9db0-366d400001f3\" (UID: \"19f3a006-19e8-44af-9db0-366d400001f3\") "
Oct 30 13:15:20.436215 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.436188    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03d3f1b3-4d1c-4aca-82d5-f106740125ad-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03d3f1b3-4d1c-4aca-82d5-f106740125ad-maas-api-service-ca\") pod \"03d3f1b3-4d1c-4aca-82d5-f106740125ad\" (UID: \"03d3f1b3-4d1c-4aca-82d5-f106740125ad\") "
Oct 30 13:15:20.436364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.436295    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac81161c-cd40-46f4-9fe0-2490f472393c-kube-api-access-74f6b\" (UniqueName: \"kubernetes.io/projected/ac81161c-cd40-46f4-9fe0-2490f472393c-kube-api-access-74f6b\") pod \"ac81161c-cd40-46f4-9fe0-2490f472393c\" (UID: \"ac81161c-cd40-46f4-9fe0-2490f472393c\") "
Oct 30 13:15:20.436364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.436345    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03d3f1b3-4d1c-4aca-82d5-f106740125ad-kube-api-access-rxv5l\" (UniqueName: \"kubernetes.io/projected/03d3f1b3-4d1c-4aca-82d5-f106740125ad-kube-api-access-rxv5l\") pod \"03d3f1b3-4d1c-4aca-82d5-f106740125ad\" (UID: \"03d3f1b3-4d1c-4aca-82d5-f106740125ad\") "
Oct 30 13:15:20.436477 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.436457    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac81161c-cd40-46f4-9fe0-2490f472393c-maas-api-service-ca" pod "ac81161c-cd40-46f4-9fe0-2490f472393c" (UID: "ac81161c-cd40-46f4-9fe0-2490f472393c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:20.436573 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.436553    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19f3a006-19e8-44af-9db0-366d400001f3-maas-api-service-ca" pod "19f3a006-19e8-44af-9db0-366d400001f3" (UID: "19f3a006-19e8-44af-9db0-366d400001f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:20.436674 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.436652    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03d3f1b3-4d1c-4aca-82d5-f106740125ad-maas-api-service-ca" pod "03d3f1b3-4d1c-4aca-82d5-f106740125ad" (UID: "03d3f1b3-4d1c-4aca-82d5-f106740125ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:20.437185 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.437158    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac81161c-cd40-46f4-9fe0-2490f472393c-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:20.437185 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.437187    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19f3a006-19e8-44af-9db0-366d400001f3-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:20.437418 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.437200    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03d3f1b3-4d1c-4aca-82d5-f106740125ad-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:20.438982 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.438918    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac81161c-cd40-46f4-9fe0-2490f472393c-kube-api-access-74f6b" pod "ac81161c-cd40-46f4-9fe0-2490f472393c" (UID: "ac81161c-cd40-46f4-9fe0-2490f472393c"). InnerVolumeSpecName "kube-api-access-74f6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:20.438982 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.438960    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03d3f1b3-4d1c-4aca-82d5-f106740125ad-kube-api-access-rxv5l" pod "03d3f1b3-4d1c-4aca-82d5-f106740125ad" (UID: "03d3f1b3-4d1c-4aca-82d5-f106740125ad"). InnerVolumeSpecName "kube-api-access-rxv5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:20.439101 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.438988    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19f3a006-19e8-44af-9db0-366d400001f3-kube-api-access-8w287" pod "19f3a006-19e8-44af-9db0-366d400001f3" (UID: "19f3a006-19e8-44af-9db0-366d400001f3"). InnerVolumeSpecName "kube-api-access-8w287". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:20.539032 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.538992    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w287\" (UniqueName: \"kubernetes.io/projected/19f3a006-19e8-44af-9db0-366d400001f3-kube-api-access-8w287\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:20.539032 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.539027    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74f6b\" (UniqueName: \"kubernetes.io/projected/ac81161c-cd40-46f4-9fe0-2490f472393c-kube-api-access-74f6b\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:20.539283 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:20.539043    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxv5l\" (UniqueName: \"kubernetes.io/projected/03d3f1b3-4d1c-4aca-82d5-f106740125ad-kube-api-access-rxv5l\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:21.055556 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.055522    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq"
Oct 30 13:15:21.055556 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.055539    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq" event={"ID":"ac81161c-cd40-46f4-9fe0-2490f472393c","Type":"ContainerDied","Data":"2cd327a23285d691e02d46ea566355d4c09110ee0cabd55fb9289cd2b03a80be"}
Oct 30 13:15:21.056060 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.055600    4544 scope.go:122] "RemoveContainer" containerID="15b180018e5d0fc3431881a5cd166b67b7fb43fc0fa86f90e514ebf68b1c1ebb"
Oct 30 13:15:21.059268 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.057470    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l"
Oct 30 13:15:21.059268 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.057490    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l" event={"ID":"03d3f1b3-4d1c-4aca-82d5-f106740125ad","Type":"ContainerDied","Data":"c45482331ad8004146dc46894b9b8556ce1ba7eae995f7af5c9d02817f51df75"}
Oct 30 13:15:21.073091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.071697    4544 scope.go:122] "RemoveContainer" containerID="0a6fd14dca6ec2205d77e8ecd4924715fd6bbcc5bd1f9bf503249a052ddcd233"
Oct 30 13:15:21.074523 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.074500    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7"
Oct 30 13:15:21.074706 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.074674    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7" event={"ID":"19f3a006-19e8-44af-9db0-366d400001f3","Type":"ContainerDied","Data":"92fe53390561d3191b07716cb5e22d3b98c419bb1c7451800a4c7a99cb8f6445"}
Oct 30 13:15:21.106445 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.106281    4544 scope.go:122] "RemoveContainer" containerID="750784d15324dd5739d521034a4ca702b082637b2377aadb54f0d8b45ffdc223"
Oct 30 13:15:21.106621 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.106560    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32e6a633-93f2-48be-87d6-5a9521132614" path="/var/lib/kubelet/pods/32e6a633-93f2-48be-87d6-5a9521132614/volumes"
Oct 30 13:15:21.108320 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.107392    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq" event={"ID":"1ddc624d-f00d-4c01-b9e1-877259555186","Type":"ContainerStarted","Data":"80305e7a6156f092364068b02294b3c776ac44a2ee07f3c09d6d0dff70a11aac"}
Oct 30 13:15:21.190232 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.190160    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7"]
Oct 30 13:15:21.205548 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.205443    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889435-w6bm7"]
Oct 30 13:15:21.283257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.283211    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f"
Oct 30 13:15:21.354670 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.354391    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj"]
Oct 30 13:15:21.354670 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.354645    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72837ba1-2075-4553-a926-3716a21270ee-kube-api-access-w7dgl\" (UniqueName: \"kubernetes.io/projected/72837ba1-2075-4553-a926-3716a21270ee-kube-api-access-w7dgl\") pod \"72837ba1-2075-4553-a926-3716a21270ee\" (UID: \"72837ba1-2075-4553-a926-3716a21270ee\") "
Oct 30 13:15:21.355102 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.355072    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72837ba1-2075-4553-a926-3716a21270ee-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72837ba1-2075-4553-a926-3716a21270ee-maas-api-service-ca\") pod \"72837ba1-2075-4553-a926-3716a21270ee\" (UID: \"72837ba1-2075-4553-a926-3716a21270ee\") "
Oct 30 13:15:21.361498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.361469    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l"]
Oct 30 13:15:21.369730 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.369683    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889435-8zv6l"]
Oct 30 13:15:21.376928 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.376881    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72837ba1-2075-4553-a926-3716a21270ee-maas-api-service-ca" pod "72837ba1-2075-4553-a926-3716a21270ee" (UID: "72837ba1-2075-4553-a926-3716a21270ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:21.383097 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.383005    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72837ba1-2075-4553-a926-3716a21270ee-kube-api-access-w7dgl" pod "72837ba1-2075-4553-a926-3716a21270ee" (UID: "72837ba1-2075-4553-a926-3716a21270ee"). InnerVolumeSpecName "kube-api-access-w7dgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:21.407605 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.407573    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq"
Oct 30 13:15:21.456847 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.456680    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/674c045d-1ba0-4075-8ce8-84e3b275dbd1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/674c045d-1ba0-4075-8ce8-84e3b275dbd1-maas-api-service-ca\") pod \"674c045d-1ba0-4075-8ce8-84e3b275dbd1\" (UID: \"674c045d-1ba0-4075-8ce8-84e3b275dbd1\") "
Oct 30 13:15:21.456847 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.456802    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/674c045d-1ba0-4075-8ce8-84e3b275dbd1-kube-api-access-p9t4g\" (UniqueName: \"kubernetes.io/projected/674c045d-1ba0-4075-8ce8-84e3b275dbd1-kube-api-access-p9t4g\") pod \"674c045d-1ba0-4075-8ce8-84e3b275dbd1\" (UID: \"674c045d-1ba0-4075-8ce8-84e3b275dbd1\") "
Oct 30 13:15:21.457396 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.457376    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7dgl\" (UniqueName: \"kubernetes.io/projected/72837ba1-2075-4553-a926-3716a21270ee-kube-api-access-w7dgl\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:21.457541 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.457399    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72837ba1-2075-4553-a926-3716a21270ee-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:21.458372 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.458150    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/674c045d-1ba0-4075-8ce8-84e3b275dbd1-maas-api-service-ca" pod "674c045d-1ba0-4075-8ce8-84e3b275dbd1" (UID: "674c045d-1ba0-4075-8ce8-84e3b275dbd1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:21.465263 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.460673    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs"
Oct 30 13:15:21.469049 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.467849    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/674c045d-1ba0-4075-8ce8-84e3b275dbd1-kube-api-access-p9t4g" pod "674c045d-1ba0-4075-8ce8-84e3b275dbd1" (UID: "674c045d-1ba0-4075-8ce8-84e3b275dbd1"). InnerVolumeSpecName "kube-api-access-p9t4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:21.502540 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.502505    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq"]
Oct 30 13:15:21.514985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.514951    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-7a71ca-29889435-rf5sq"]
Oct 30 13:15:21.558691 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.558660    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6ae22b0-e9c0-45bc-85e4-76c659da2808-kube-api-access-wpgck\" (UniqueName: \"kubernetes.io/projected/d6ae22b0-e9c0-45bc-85e4-76c659da2808-kube-api-access-wpgck\") pod \"d6ae22b0-e9c0-45bc-85e4-76c659da2808\" (UID: \"d6ae22b0-e9c0-45bc-85e4-76c659da2808\") "
Oct 30 13:15:21.558861 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.558700    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6ae22b0-e9c0-45bc-85e4-76c659da2808-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6ae22b0-e9c0-45bc-85e4-76c659da2808-maas-api-service-ca\") pod \"d6ae22b0-e9c0-45bc-85e4-76c659da2808\" (UID: \"d6ae22b0-e9c0-45bc-85e4-76c659da2808\") "
Oct 30 13:15:21.559168 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.559136    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6ae22b0-e9c0-45bc-85e4-76c659da2808-maas-api-service-ca" pod "d6ae22b0-e9c0-45bc-85e4-76c659da2808" (UID: "d6ae22b0-e9c0-45bc-85e4-76c659da2808"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:21.559537 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.559512    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/674c045d-1ba0-4075-8ce8-84e3b275dbd1-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:21.559635 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.559541    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9t4g\" (UniqueName: \"kubernetes.io/projected/674c045d-1ba0-4075-8ce8-84e3b275dbd1-kube-api-access-p9t4g\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:21.559635 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.559558    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6ae22b0-e9c0-45bc-85e4-76c659da2808-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:21.561435 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.561410    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6ae22b0-e9c0-45bc-85e4-76c659da2808-kube-api-access-wpgck" pod "d6ae22b0-e9c0-45bc-85e4-76c659da2808" (UID: "d6ae22b0-e9c0-45bc-85e4-76c659da2808"). InnerVolumeSpecName "kube-api-access-wpgck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:21.569371 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.569334    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj"
Oct 30 13:15:21.668130 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.661881    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/002634ca-d404-4827-b3e7-d007555a2792-kube-api-access-5z4kn\" (UniqueName: \"kubernetes.io/projected/002634ca-d404-4827-b3e7-d007555a2792-kube-api-access-5z4kn\") pod \"002634ca-d404-4827-b3e7-d007555a2792\" (UID: \"002634ca-d404-4827-b3e7-d007555a2792\") "
Oct 30 13:15:21.668130 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.662088    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/002634ca-d404-4827-b3e7-d007555a2792-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/002634ca-d404-4827-b3e7-d007555a2792-maas-api-service-ca\") pod \"002634ca-d404-4827-b3e7-d007555a2792\" (UID: \"002634ca-d404-4827-b3e7-d007555a2792\") "
Oct 30 13:15:21.668130 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.663344    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpgck\" (UniqueName: \"kubernetes.io/projected/d6ae22b0-e9c0-45bc-85e4-76c659da2808-kube-api-access-wpgck\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:21.668130 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.663754    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/002634ca-d404-4827-b3e7-d007555a2792-maas-api-service-ca" pod "002634ca-d404-4827-b3e7-d007555a2792" (UID: "002634ca-d404-4827-b3e7-d007555a2792"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:21.668130 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.666887    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/002634ca-d404-4827-b3e7-d007555a2792-kube-api-access-5z4kn" pod "002634ca-d404-4827-b3e7-d007555a2792" (UID: "002634ca-d404-4827-b3e7-d007555a2792"). InnerVolumeSpecName "kube-api-access-5z4kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:21.769981 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.769590    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z4kn\" (UniqueName: \"kubernetes.io/projected/002634ca-d404-4827-b3e7-d007555a2792-kube-api-access-5z4kn\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:21.769981 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:21.769623    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/002634ca-d404-4827-b3e7-d007555a2792-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:22.124281 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:22.124242    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:22.149410 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:22.136631    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f" event={"ID":"72837ba1-2075-4553-a926-3716a21270ee","Type":"ContainerDied","Data":"b406e7a806749140e7345c7db66c898b3baa3af7bf18e7c5f20ad8f1c5dd7d36"}
Oct 30 13:15:22.149723 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:22.149688    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs" event={"ID":"d6ae22b0-e9c0-45bc-85e4-76c659da2808","Type":"ContainerDied","Data":"94af8df1e5ade542f2bfa2b03eff14ce889fa3f59e2b101d0536918282b06a6d"}
Oct 30 13:15:22.151598 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:22.151316    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj" event={"ID":"002634ca-d404-4827-b3e7-d007555a2792","Type":"ContainerDied","Data":"1b5769b5fd03f661c96e72bfee01173548827dbb60899c53cd026e6f7a31e07e"}
Oct 30 13:15:22.151598 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:22.151360    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq" event={"ID":"1ddc624d-f00d-4c01-b9e1-877259555186","Type":"ContainerDied","Data":"80305e7a6156f092364068b02294b3c776ac44a2ee07f3c09d6d0dff70a11aac"}
Oct 30 13:15:22.151768 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:22.142882    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj"
Oct 30 13:15:22.152352 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:22.149890    4544 scope.go:122] "RemoveContainer" containerID="d5a942f04c9150ba9b0f2d46e8c9b3d23c95c05a15e074ab000f5892bb4dde16"
Oct 30 13:15:22.152464 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:22.136749    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f"
Oct 30 13:15:22.152530 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:22.152481    4544 scope.go:122] "RemoveContainer" containerID="80305e7a6156f092364068b02294b3c776ac44a2ee07f3c09d6d0dff70a11aac"
Oct 30 13:15:22.152593 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:22.138744    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs"
Oct 30 13:15:22.153031 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:15:22.152870    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq_odh-ai-gateway-infra(1ddc624d-f00d-4c01-b9e1-877259555186)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq" podUID="1ddc624d-f00d-4c01-b9e1-877259555186"
Oct 30 13:15:22.153031 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:22.145470    4544 generic.go:356] "Generic (PLEG): container finished" podID="1ddc624d-f00d-4c01-b9e1-877259555186" containerID="80305e7a6156f092364068b02294b3c776ac44a2ee07f3c09d6d0dff70a11aac" exitCode=60
Oct 30 13:15:22.167449 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:22.166924    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq" event={"ID":"674c045d-1ba0-4075-8ce8-84e3b275dbd1","Type":"ContainerDied","Data":"4ee7e89e7b2ad00316904f06e428d8da4a355f66258f5a836c6fe3fbac035646"}
Oct 30 13:15:22.167449 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:22.166986    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq"
Oct 30 13:15:22.179294 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:22.179269    4544 scope.go:122] "RemoveContainer" containerID="8421e2c40ee87975835c05f6aef096ce8e49f1a7dce40c7264fe3618ea60cf36"
Oct 30 13:15:22.191116 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:22.191099    4544 scope.go:122] "RemoveContainer" containerID="d4b511f471a1d2a5c254a68040e568b1072364afff91b4e447d151cdde0501b9"
Oct 30 13:15:22.201712 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:22.201688    4544 scope.go:122] "RemoveContainer" containerID="5b76ce30fb65a60949bada8da9068dc231ff0615b274eeea46932d84169b1009"
Oct 30 13:15:22.216117 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:22.216090    4544 scope.go:122] "RemoveContainer" containerID="84e2d1f9407202388d173b0981dc1ae2860ce715664e60b41e30e9549cd723e3"
Oct 30 13:15:22.255508 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:22.255478    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq"]
Oct 30 13:15:22.262505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:22.262477    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-11cfa6-29889435-cshrq"]
Oct 30 13:15:22.299383 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:22.299351    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs"]
Oct 30 13:15:22.326851 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:22.326825    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889435-x5krs"]
Oct 30 13:15:22.379137 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:22.379060    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f"]
Oct 30 13:15:22.390061 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:22.390031    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889435-tgg6f"]
Oct 30 13:15:22.411128 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:22.411100    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq"]
Oct 30 13:15:22.435888 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:22.435863    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj"]
Oct 30 13:15:22.451438 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:22.451416    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889435-br5qj"]
Oct 30 13:15:23.101380 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:23.101340    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="002634ca-d404-4827-b3e7-d007555a2792" path="/var/lib/kubelet/pods/002634ca-d404-4827-b3e7-d007555a2792/volumes"
Oct 30 13:15:23.102387 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:23.102360    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03d3f1b3-4d1c-4aca-82d5-f106740125ad" path="/var/lib/kubelet/pods/03d3f1b3-4d1c-4aca-82d5-f106740125ad/volumes"
Oct 30 13:15:23.103280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:23.103255    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19f3a006-19e8-44af-9db0-366d400001f3" path="/var/lib/kubelet/pods/19f3a006-19e8-44af-9db0-366d400001f3/volumes"
Oct 30 13:15:23.103858 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:23.103839    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="674c045d-1ba0-4075-8ce8-84e3b275dbd1" path="/var/lib/kubelet/pods/674c045d-1ba0-4075-8ce8-84e3b275dbd1/volumes"
Oct 30 13:15:23.104407 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:23.104390    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72837ba1-2075-4553-a926-3716a21270ee" path="/var/lib/kubelet/pods/72837ba1-2075-4553-a926-3716a21270ee/volumes"
Oct 30 13:15:23.105393 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:23.105370    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac81161c-cd40-46f4-9fe0-2490f472393c" path="/var/lib/kubelet/pods/ac81161c-cd40-46f4-9fe0-2490f472393c/volumes"
Oct 30 13:15:23.105932 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:23.105915    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6ae22b0-e9c0-45bc-85e4-76c659da2808" path="/var/lib/kubelet/pods/d6ae22b0-e9c0-45bc-85e4-76c659da2808/volumes"
Oct 30 13:15:23.351965 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:23.351908    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq"
Oct 30 13:15:23.418620 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:23.418586    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ddc624d-f00d-4c01-b9e1-877259555186-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ddc624d-f00d-4c01-b9e1-877259555186-maas-api-service-ca\") pod \"1ddc624d-f00d-4c01-b9e1-877259555186\" (UID: \"1ddc624d-f00d-4c01-b9e1-877259555186\") "
Oct 30 13:15:23.418971 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:23.418944    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ddc624d-f00d-4c01-b9e1-877259555186-maas-api-service-ca" pod "1ddc624d-f00d-4c01-b9e1-877259555186" (UID: "1ddc624d-f00d-4c01-b9e1-877259555186"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:23.419022 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:23.419002    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ddc624d-f00d-4c01-b9e1-877259555186-kube-api-access-s6xn9\" (UniqueName: \"kubernetes.io/projected/1ddc624d-f00d-4c01-b9e1-877259555186-kube-api-access-s6xn9\") pod \"1ddc624d-f00d-4c01-b9e1-877259555186\" (UID: \"1ddc624d-f00d-4c01-b9e1-877259555186\") "
Oct 30 13:15:23.419720 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:23.419694    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ddc624d-f00d-4c01-b9e1-877259555186-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:23.421038 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:23.421013    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ddc624d-f00d-4c01-b9e1-877259555186-kube-api-access-s6xn9" pod "1ddc624d-f00d-4c01-b9e1-877259555186" (UID: "1ddc624d-f00d-4c01-b9e1-877259555186"). InnerVolumeSpecName "kube-api-access-s6xn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:23.521478 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:23.521443    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6xn9\" (UniqueName: \"kubernetes.io/projected/1ddc624d-f00d-4c01-b9e1-877259555186-kube-api-access-s6xn9\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:24.120502 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:24.120440    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh" podUID="b4c5666f-6f23-4001-a5d4-c305bb5cd950" containerName="payload-pre-processing" containerID="cri-o://a667e13629e69fe46bbdbe3cbeec458b169f69e48118cbb3fb5da833fd017c09" gracePeriod=25
Oct 30 13:15:24.169474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:24.169432    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:24.190888 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:24.190863    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq"
Oct 30 13:15:24.191013 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:24.190889    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq" event={"ID":"1ddc624d-f00d-4c01-b9e1-877259555186","Type":"ContainerDied","Data":"c0457d223d5312188062516a156588d044edc5280f77f986b3a6b177c04ad7f6"}
Oct 30 13:15:24.191013 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:24.190947    4544 scope.go:122] "RemoveContainer" containerID="80305e7a6156f092364068b02294b3c776ac44a2ee07f3c09d6d0dff70a11aac"
Oct 30 13:15:24.248893 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:24.248871    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq"]
Oct 30 13:15:24.260929 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:24.260901    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-716b4c-29889435-xqhlq"]
Oct 30 13:15:24.399954 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:24.340186    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:24.472875 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:24.472840    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:24.532752 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:24.532730    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh"
Oct 30 13:15:24.606684 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:24.606646    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4c5666f-6f23-4001-a5d4-c305bb5cd950-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4c5666f-6f23-4001-a5d4-c305bb5cd950-plugins-config-volume\") pod \"b4c5666f-6f23-4001-a5d4-c305bb5cd950\" (UID: \"b4c5666f-6f23-4001-a5d4-c305bb5cd950\") "
Oct 30 13:15:24.606930 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:24.606914    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4c5666f-6f23-4001-a5d4-c305bb5cd950-kube-api-access-2gdwp\" (UniqueName: \"kubernetes.io/projected/b4c5666f-6f23-4001-a5d4-c305bb5cd950-kube-api-access-2gdwp\") pod \"b4c5666f-6f23-4001-a5d4-c305bb5cd950\" (UID: \"b4c5666f-6f23-4001-a5d4-c305bb5cd950\") "
Oct 30 13:15:24.607068 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:24.607040    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4c5666f-6f23-4001-a5d4-c305bb5cd950-plugins-config-volume" pod "b4c5666f-6f23-4001-a5d4-c305bb5cd950" (UID: "b4c5666f-6f23-4001-a5d4-c305bb5cd950"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:24.607545 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:24.607523    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4c5666f-6f23-4001-a5d4-c305bb5cd950-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:24.609081 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:24.609045    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c5666f-6f23-4001-a5d4-c305bb5cd950-kube-api-access-2gdwp" pod "b4c5666f-6f23-4001-a5d4-c305bb5cd950" (UID: "b4c5666f-6f23-4001-a5d4-c305bb5cd950"). InnerVolumeSpecName "kube-api-access-2gdwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:24.708881 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:24.708806    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gdwp\" (UniqueName: \"kubernetes.io/projected/b4c5666f-6f23-4001-a5d4-c305bb5cd950-kube-api-access-2gdwp\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:25.123003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:25.122595    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ddc624d-f00d-4c01-b9e1-877259555186" path="/var/lib/kubelet/pods/1ddc624d-f00d-4c01-b9e1-877259555186/volumes"
Oct 30 13:15:25.202187 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:25.202158    4544 generic.go:356] "Generic (PLEG): container finished" podID="b4c5666f-6f23-4001-a5d4-c305bb5cd950" containerID="a667e13629e69fe46bbdbe3cbeec458b169f69e48118cbb3fb5da833fd017c09" exitCode=0
Oct 30 13:15:25.202385 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:25.202195    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh" event={"ID":"b4c5666f-6f23-4001-a5d4-c305bb5cd950","Type":"ContainerDied","Data":"a667e13629e69fe46bbdbe3cbeec458b169f69e48118cbb3fb5da833fd017c09"}
Oct 30 13:15:25.202385 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:25.202201    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh"
Oct 30 13:15:25.202385 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:25.202276    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh" event={"ID":"b4c5666f-6f23-4001-a5d4-c305bb5cd950","Type":"ContainerDied","Data":"f0b935340c80d700a7af211a71af7e36a9972a3a498dcb7b403b31fe421d76a5"}
Oct 30 13:15:25.202385 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:25.202300    4544 scope.go:122] "RemoveContainer" containerID="a667e13629e69fe46bbdbe3cbeec458b169f69e48118cbb3fb5da833fd017c09"
Oct 30 13:15:25.213176 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:25.213148    4544 scope.go:122] "RemoveContainer" containerID="a667e13629e69fe46bbdbe3cbeec458b169f69e48118cbb3fb5da833fd017c09"
Oct 30 13:15:25.213523 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:15:25.213497    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a667e13629e69fe46bbdbe3cbeec458b169f69e48118cbb3fb5da833fd017c09\": container with ID starting with a667e13629e69fe46bbdbe3cbeec458b169f69e48118cbb3fb5da833fd017c09 not found: ID does not exist" containerID="a667e13629e69fe46bbdbe3cbeec458b169f69e48118cbb3fb5da833fd017c09"
Oct 30 13:15:25.213612 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:25.213532    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a667e13629e69fe46bbdbe3cbeec458b169f69e48118cbb3fb5da833fd017c09"} err="failed to get container status \"a667e13629e69fe46bbdbe3cbeec458b169f69e48118cbb3fb5da833fd017c09\": rpc error: code = NotFound desc = could not find container \"a667e13629e69fe46bbdbe3cbeec458b169f69e48118cbb3fb5da833fd017c09\": container with ID starting with a667e13629e69fe46bbdbe3cbeec458b169f69e48118cbb3fb5da833fd017c09 not found: ID does not exist"
Oct 30 13:15:25.261504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:25.261468    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh"]
Oct 30 13:15:25.272372 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:25.272344    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh"]
Oct 30 13:15:26.718467 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:26.710589    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:27.008139 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:27.008047    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"]
Oct 30 13:15:27.008348 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:27.008324    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q" podUID="297febba-5574-48a5-9e02-71dc942697a9" containerName="istio-proxy" containerID="cri-o://b7356a1f4371755d5f71eac9e32dd702dd243387fbb9e1363a8f98a1cee05233" gracePeriod=30
Oct 30 13:15:27.089139 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:27.089105    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4c5666f-6f23-4001-a5d4-c305bb5cd950" path="/var/lib/kubelet/pods/b4c5666f-6f23-4001-a5d4-c305bb5cd950/volumes"
Oct 30 13:15:28.092817 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:28.092781    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"]
Oct 30 13:15:28.093263 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:28.092990    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x" podUID="ab79901f-7e31-4e95-a033-6eb1235af1fa" containerName="istio-proxy" containerID="cri-o://cbd936f4a22fdec1ef4f436357f28313e0368980388cb7d4e7b9115c2ea239ac" gracePeriod=30
Oct 30 13:15:28.666511 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:28.666454    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:28.811185 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:28.811096    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:29.007408 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:15:29.007325    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:15:29.920103 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:29.920071    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:30.069022 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:30.068985    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:30.929982 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:30.929915    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:32.267156 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:32.267116    4544 generic.go:356] "Generic (PLEG): container finished" podID="297febba-5574-48a5-9e02-71dc942697a9" containerID="b7356a1f4371755d5f71eac9e32dd702dd243387fbb9e1363a8f98a1cee05233" exitCode=0
Oct 30 13:15:32.267608 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:32.267194    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q" event={"ID":"297febba-5574-48a5-9e02-71dc942697a9","Type":"ContainerDied","Data":"b7356a1f4371755d5f71eac9e32dd702dd243387fbb9e1363a8f98a1cee05233"}
Oct 30 13:15:32.312629 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:32.312599    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:15:32.356897 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:32.356864    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-istio-data\") pod \"297febba-5574-48a5-9e02-71dc942697a9\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") "
Oct 30 13:15:32.357106 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:32.356988    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-credential-socket\") pod \"297febba-5574-48a5-9e02-71dc942697a9\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") "
Oct 30 13:15:32.357106 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:32.357053    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-workload-certs\") pod \"297febba-5574-48a5-9e02-71dc942697a9\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") "
Oct 30 13:15:32.357253 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:32.357148    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/297febba-5574-48a5-9e02-71dc942697a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/297febba-5574-48a5-9e02-71dc942697a9-istiod-ca-cert\") pod \"297febba-5574-48a5-9e02-71dc942697a9\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") "
Oct 30 13:15:32.357325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:32.357205    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-istio-data" pod "297febba-5574-48a5-9e02-71dc942697a9" (UID: "297febba-5574-48a5-9e02-71dc942697a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:32.357325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:32.357253    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/297febba-5574-48a5-9e02-71dc942697a9-kube-api-access-v4jp8\" (UniqueName: \"kubernetes.io/projected/297febba-5574-48a5-9e02-71dc942697a9-kube-api-access-v4jp8\") pod \"297febba-5574-48a5-9e02-71dc942697a9\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") "
Oct 30 13:15:32.357325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:32.357315    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/297febba-5574-48a5-9e02-71dc942697a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/297febba-5574-48a5-9e02-71dc942697a9-istio-podinfo\") pod \"297febba-5574-48a5-9e02-71dc942697a9\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") "
Oct 30 13:15:32.357490 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:32.357329    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-credential-socket" pod "297febba-5574-48a5-9e02-71dc942697a9" (UID: "297febba-5574-48a5-9e02-71dc942697a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:32.357490 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:32.357412    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-workload-certs" pod "297febba-5574-48a5-9e02-71dc942697a9" (UID: "297febba-5574-48a5-9e02-71dc942697a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:32.357490 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:32.357456    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-workload-socket\") pod \"297febba-5574-48a5-9e02-71dc942697a9\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") "
Oct 30 13:15:32.357713 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:32.357501    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-istio-envoy\") pod \"297febba-5574-48a5-9e02-71dc942697a9\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") "
Oct 30 13:15:32.357713 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:32.357540    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/297febba-5574-48a5-9e02-71dc942697a9-istio-token\" (UniqueName: \"kubernetes.io/projected/297febba-5574-48a5-9e02-71dc942697a9-istio-token\") pod \"297febba-5574-48a5-9e02-71dc942697a9\" (UID: \"297febba-5574-48a5-9e02-71dc942697a9\") "
Oct 30 13:15:32.357713 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:32.357662    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-workload-socket" pod "297febba-5574-48a5-9e02-71dc942697a9" (UID: "297febba-5574-48a5-9e02-71dc942697a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:32.357904 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:32.357835    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/297febba-5574-48a5-9e02-71dc942697a9-istiod-ca-cert" pod "297febba-5574-48a5-9e02-71dc942697a9" (UID: "297febba-5574-48a5-9e02-71dc942697a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:32.358447 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:32.358423    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:32.358447 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:32.358450    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/297febba-5574-48a5-9e02-71dc942697a9-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:32.358642 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:32.358463    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:32.358642 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:32.358476    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:32.358642 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:32.358488    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:32.359811 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:32.359786    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/297febba-5574-48a5-9e02-71dc942697a9-kube-api-access-v4jp8" pod "297febba-5574-48a5-9e02-71dc942697a9" (UID: "297febba-5574-48a5-9e02-71dc942697a9"). InnerVolumeSpecName "kube-api-access-v4jp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:32.360026 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:32.360006    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/297febba-5574-48a5-9e02-71dc942697a9-istio-podinfo" pod "297febba-5574-48a5-9e02-71dc942697a9" (UID: "297febba-5574-48a5-9e02-71dc942697a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:15:32.360157 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:32.360124    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/297febba-5574-48a5-9e02-71dc942697a9-istio-token" pod "297febba-5574-48a5-9e02-71dc942697a9" (UID: "297febba-5574-48a5-9e02-71dc942697a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:32.360303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:32.360241    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-istio-envoy" pod "297febba-5574-48a5-9e02-71dc942697a9" (UID: "297febba-5574-48a5-9e02-71dc942697a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:32.460180 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:32.460144    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/297febba-5574-48a5-9e02-71dc942697a9-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:32.460180 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:32.460175    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/297febba-5574-48a5-9e02-71dc942697a9-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:32.460180 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:32.460186    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4jp8\" (UniqueName: \"kubernetes.io/projected/297febba-5574-48a5-9e02-71dc942697a9-kube-api-access-v4jp8\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:32.460482 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:32.460201    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/297febba-5574-48a5-9e02-71dc942697a9-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:32.795603 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:32.795566    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:33.283854 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.283824    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"
Oct 30 13:15:33.283854 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.283834    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q" event={"ID":"297febba-5574-48a5-9e02-71dc942697a9","Type":"ContainerDied","Data":"5c3e85e772eee168f07a05d8645f436dd47470fa92cf773f4ac2578ca5fe8e9c"}
Oct 30 13:15:33.284391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.283897    4544 scope.go:122] "RemoveContainer" containerID="b7356a1f4371755d5f71eac9e32dd702dd243387fbb9e1363a8f98a1cee05233"
Oct 30 13:15:33.287853 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.287786    4544 generic.go:356] "Generic (PLEG): container finished" podID="ab79901f-7e31-4e95-a033-6eb1235af1fa" containerID="cbd936f4a22fdec1ef4f436357f28313e0368980388cb7d4e7b9115c2ea239ac" exitCode=0
Oct 30 13:15:33.288143 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.288119    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x" event={"ID":"ab79901f-7e31-4e95-a033-6eb1235af1fa","Type":"ContainerDied","Data":"cbd936f4a22fdec1ef4f436357f28313e0368980388cb7d4e7b9115c2ea239ac"}
Oct 30 13:15:33.308708 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.308669    4544 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:15:33.308862 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.308747    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q" podUID="297febba-5574-48a5-9e02-71dc942697a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:15:33.353154 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.353078    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"]
Oct 30 13:15:33.364895 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.364853    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q"]
Oct 30 13:15:33.381843 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.381816    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:15:33.476739 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.476705    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab79901f-7e31-4e95-a033-6eb1235af1fa-istio-token\" (UniqueName: \"kubernetes.io/projected/ab79901f-7e31-4e95-a033-6eb1235af1fa-istio-token\") pod \"ab79901f-7e31-4e95-a033-6eb1235af1fa\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") "
Oct 30 13:15:33.476940 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.476762    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab79901f-7e31-4e95-a033-6eb1235af1fa-kube-api-access-mtlr7\" (UniqueName: \"kubernetes.io/projected/ab79901f-7e31-4e95-a033-6eb1235af1fa-kube-api-access-mtlr7\") pod \"ab79901f-7e31-4e95-a033-6eb1235af1fa\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") "
Oct 30 13:15:33.477014 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.476961    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab79901f-7e31-4e95-a033-6eb1235af1fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab79901f-7e31-4e95-a033-6eb1235af1fa-istiod-ca-cert\") pod \"ab79901f-7e31-4e95-a033-6eb1235af1fa\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") "
Oct 30 13:15:33.477014 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.477001    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab79901f-7e31-4e95-a033-6eb1235af1fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab79901f-7e31-4e95-a033-6eb1235af1fa-istio-podinfo\") pod \"ab79901f-7e31-4e95-a033-6eb1235af1fa\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") "
Oct 30 13:15:33.477129 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.477047    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-workload-certs\") pod \"ab79901f-7e31-4e95-a033-6eb1235af1fa\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") "
Oct 30 13:15:33.477195 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.477133    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-istio-envoy\") pod \"ab79901f-7e31-4e95-a033-6eb1235af1fa\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") "
Oct 30 13:15:33.477412 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.477374    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-workload-certs" pod "ab79901f-7e31-4e95-a033-6eb1235af1fa" (UID: "ab79901f-7e31-4e95-a033-6eb1235af1fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:33.477634 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.477429    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab79901f-7e31-4e95-a033-6eb1235af1fa-istiod-ca-cert" pod "ab79901f-7e31-4e95-a033-6eb1235af1fa" (UID: "ab79901f-7e31-4e95-a033-6eb1235af1fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:33.477852 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.477797    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-istio-data\") pod \"ab79901f-7e31-4e95-a033-6eb1235af1fa\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") "
Oct 30 13:15:33.478083 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.478054    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-istio-data" pod "ab79901f-7e31-4e95-a033-6eb1235af1fa" (UID: "ab79901f-7e31-4e95-a033-6eb1235af1fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:33.478322 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.478298    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-workload-socket\") pod \"ab79901f-7e31-4e95-a033-6eb1235af1fa\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") "
Oct 30 13:15:33.478420 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.478356    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-credential-socket\") pod \"ab79901f-7e31-4e95-a033-6eb1235af1fa\" (UID: \"ab79901f-7e31-4e95-a033-6eb1235af1fa\") "
Oct 30 13:15:33.479027 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.479003    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab79901f-7e31-4e95-a033-6eb1235af1fa-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:33.479134 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.479029    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:33.479134 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.479044    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:33.479343 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.479323    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-credential-socket" pod "ab79901f-7e31-4e95-a033-6eb1235af1fa" (UID: "ab79901f-7e31-4e95-a033-6eb1235af1fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:33.479522 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.479502    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-workload-socket" pod "ab79901f-7e31-4e95-a033-6eb1235af1fa" (UID: "ab79901f-7e31-4e95-a033-6eb1235af1fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:33.484745 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.484717    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab79901f-7e31-4e95-a033-6eb1235af1fa-kube-api-access-mtlr7" pod "ab79901f-7e31-4e95-a033-6eb1235af1fa" (UID: "ab79901f-7e31-4e95-a033-6eb1235af1fa"). InnerVolumeSpecName "kube-api-access-mtlr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:33.484915 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.484874    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab79901f-7e31-4e95-a033-6eb1235af1fa-istio-podinfo" pod "ab79901f-7e31-4e95-a033-6eb1235af1fa" (UID: "ab79901f-7e31-4e95-a033-6eb1235af1fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:15:33.485067 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.485043    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab79901f-7e31-4e95-a033-6eb1235af1fa-istio-token" pod "ab79901f-7e31-4e95-a033-6eb1235af1fa" (UID: "ab79901f-7e31-4e95-a033-6eb1235af1fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:33.485283 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.485209    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-istio-envoy" pod "ab79901f-7e31-4e95-a033-6eb1235af1fa" (UID: "ab79901f-7e31-4e95-a033-6eb1235af1fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:33.580763 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.580727    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:33.580763 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.580757    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:33.580763 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.580768    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab79901f-7e31-4e95-a033-6eb1235af1fa-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:33.581011 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.580777    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtlr7\" (UniqueName: \"kubernetes.io/projected/ab79901f-7e31-4e95-a033-6eb1235af1fa-kube-api-access-mtlr7\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:33.581011 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.580787    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab79901f-7e31-4e95-a033-6eb1235af1fa-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:33.581011 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:33.580796    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab79901f-7e31-4e95-a033-6eb1235af1fa-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:34.277474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:34.277440    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:34.298968 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:34.298930    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x" event={"ID":"ab79901f-7e31-4e95-a033-6eb1235af1fa","Type":"ContainerDied","Data":"6ca0df542599bc3615c75f3d40d051ff4c112be70240688e2a519f92f9713ff2"}
Oct 30 13:15:34.298968 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:34.298962    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"
Oct 30 13:15:34.299523 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:34.298990    4544 scope.go:122] "RemoveContainer" containerID="cbd936f4a22fdec1ef4f436357f28313e0368980388cb7d4e7b9115c2ea239ac"
Oct 30 13:15:34.355400 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:34.355357    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"]
Oct 30 13:15:34.367065 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:34.367031    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x"]
Oct 30 13:15:34.971423 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:34.971386    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:35.088807 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:35.088765    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="297febba-5574-48a5-9e02-71dc942697a9" path="/var/lib/kubelet/pods/297febba-5574-48a5-9e02-71dc942697a9/volumes"
Oct 30 13:15:35.089553 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:35.089521    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab79901f-7e31-4e95-a033-6eb1235af1fa" path="/var/lib/kubelet/pods/ab79901f-7e31-4e95-a033-6eb1235af1fa/volumes"
Oct 30 13:15:35.115488 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:35.115444    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:35.239060 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:35.238905    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:35.653912 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:35.653879    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:37.737935 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:37.737878    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:37.905042 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:37.905008    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:38.436243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:38.435431    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:39.554262 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.554207    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"]
Oct 30 13:15:39.556093 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556063    4544 cpu_manager.go:414] "containerMap: removing container" podUID="d6ae22b0-e9c0-45bc-85e4-76c659da2808" containerName="cleanup"
Oct 30 13:15:39.556093 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556091    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6ae22b0-e9c0-45bc-85e4-76c659da2808" containerName="cleanup"
Oct 30 13:15:39.556351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556123    4544 cpu_manager.go:414] "containerMap: removing container" podUID="72837ba1-2075-4553-a926-3716a21270ee" containerName="cleanup"
Oct 30 13:15:39.556351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556132    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72837ba1-2075-4553-a926-3716a21270ee" containerName="cleanup"
Oct 30 13:15:39.556351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556151    4544 cpu_manager.go:414] "containerMap: removing container" podUID="19f3a006-19e8-44af-9db0-366d400001f3" containerName="cleanup"
Oct 30 13:15:39.556351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556159    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19f3a006-19e8-44af-9db0-366d400001f3" containerName="cleanup"
Oct 30 13:15:39.556351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556174    4544 cpu_manager.go:414] "containerMap: removing container" podUID="32e6a633-93f2-48be-87d6-5a9521132614" containerName="cleanup"
Oct 30 13:15:39.556351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556181    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32e6a633-93f2-48be-87d6-5a9521132614" containerName="cleanup"
Oct 30 13:15:39.556351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556198    4544 cpu_manager.go:414] "containerMap: removing container" podUID="d6ae22b0-e9c0-45bc-85e4-76c659da2808" containerName="cleanup"
Oct 30 13:15:39.556351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556205    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6ae22b0-e9c0-45bc-85e4-76c659da2808" containerName="cleanup"
Oct 30 13:15:39.556351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556240    4544 cpu_manager.go:414] "containerMap: removing container" podUID="002634ca-d404-4827-b3e7-d007555a2792" containerName="cleanup"
Oct 30 13:15:39.556351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556248    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="002634ca-d404-4827-b3e7-d007555a2792" containerName="cleanup"
Oct 30 13:15:39.556351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556263    4544 cpu_manager.go:414] "containerMap: removing container" podUID="4bbc446d-8807-4613-9d86-b87b0733d2cf" containerName="istio-proxy"
Oct 30 13:15:39.556351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556270    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bbc446d-8807-4613-9d86-b87b0733d2cf" containerName="istio-proxy"
Oct 30 13:15:39.556351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556293    4544 cpu_manager.go:414] "containerMap: removing container" podUID="03d3f1b3-4d1c-4aca-82d5-f106740125ad" containerName="cleanup"
Oct 30 13:15:39.556351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556300    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03d3f1b3-4d1c-4aca-82d5-f106740125ad" containerName="cleanup"
Oct 30 13:15:39.556351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556316    4544 cpu_manager.go:414] "containerMap: removing container" podUID="32e6a633-93f2-48be-87d6-5a9521132614" containerName="cleanup"
Oct 30 13:15:39.556351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556323    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32e6a633-93f2-48be-87d6-5a9521132614" containerName="cleanup"
Oct 30 13:15:39.556351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556348    4544 cpu_manager.go:414] "containerMap: removing container" podUID="1ddc624d-f00d-4c01-b9e1-877259555186" containerName="cleanup"
Oct 30 13:15:39.556351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556356    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ddc624d-f00d-4c01-b9e1-877259555186" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556365    4544 cpu_manager.go:414] "containerMap: removing container" podUID="002634ca-d404-4827-b3e7-d007555a2792" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556373    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="002634ca-d404-4827-b3e7-d007555a2792" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556392    4544 cpu_manager.go:414] "containerMap: removing container" podUID="ac81161c-cd40-46f4-9fe0-2490f472393c" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556399    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac81161c-cd40-46f4-9fe0-2490f472393c" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556413    4544 cpu_manager.go:414] "containerMap: removing container" podUID="03d3f1b3-4d1c-4aca-82d5-f106740125ad" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556422    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03d3f1b3-4d1c-4aca-82d5-f106740125ad" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556443    4544 cpu_manager.go:414] "containerMap: removing container" podUID="ac81161c-cd40-46f4-9fe0-2490f472393c" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556450    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac81161c-cd40-46f4-9fe0-2490f472393c" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556466    4544 cpu_manager.go:414] "containerMap: removing container" podUID="d6ae22b0-e9c0-45bc-85e4-76c659da2808" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556474    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6ae22b0-e9c0-45bc-85e4-76c659da2808" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556485    4544 cpu_manager.go:414] "containerMap: removing container" podUID="ac81161c-cd40-46f4-9fe0-2490f472393c" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556492    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac81161c-cd40-46f4-9fe0-2490f472393c" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556527    4544 cpu_manager.go:414] "containerMap: removing container" podUID="674c045d-1ba0-4075-8ce8-84e3b275dbd1" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556535    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="674c045d-1ba0-4075-8ce8-84e3b275dbd1" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556552    4544 cpu_manager.go:414] "containerMap: removing container" podUID="4bbc446d-8807-4613-9d86-b87b0733d2cf" containerName="istio-proxy"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556559    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bbc446d-8807-4613-9d86-b87b0733d2cf" containerName="istio-proxy"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556580    4544 cpu_manager.go:414] "containerMap: removing container" podUID="1ddc624d-f00d-4c01-b9e1-877259555186" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556587    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ddc624d-f00d-4c01-b9e1-877259555186" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556610    4544 cpu_manager.go:414] "containerMap: removing container" podUID="03d3f1b3-4d1c-4aca-82d5-f106740125ad" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556617    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03d3f1b3-4d1c-4aca-82d5-f106740125ad" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556638    4544 cpu_manager.go:414] "containerMap: removing container" podUID="ab79901f-7e31-4e95-a033-6eb1235af1fa" containerName="istio-proxy"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556645    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab79901f-7e31-4e95-a033-6eb1235af1fa" containerName="istio-proxy"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556685    4544 cpu_manager.go:414] "containerMap: removing container" podUID="b4c5666f-6f23-4001-a5d4-c305bb5cd950" containerName="payload-pre-processing"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556692    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4c5666f-6f23-4001-a5d4-c305bb5cd950" containerName="payload-pre-processing"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556709    4544 cpu_manager.go:414] "containerMap: removing container" podUID="19f3a006-19e8-44af-9db0-366d400001f3" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556716    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19f3a006-19e8-44af-9db0-366d400001f3" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556735    4544 cpu_manager.go:414] "containerMap: removing container" podUID="72837ba1-2075-4553-a926-3716a21270ee" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556742    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72837ba1-2075-4553-a926-3716a21270ee" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556766    4544 cpu_manager.go:414] "containerMap: removing container" podUID="72837ba1-2075-4553-a926-3716a21270ee" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556773    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72837ba1-2075-4553-a926-3716a21270ee" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556785    4544 cpu_manager.go:414] "containerMap: removing container" podUID="002634ca-d404-4827-b3e7-d007555a2792" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556792    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="002634ca-d404-4827-b3e7-d007555a2792" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556820    4544 cpu_manager.go:414] "containerMap: removing container" podUID="674c045d-1ba0-4075-8ce8-84e3b275dbd1" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556827    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="674c045d-1ba0-4075-8ce8-84e3b275dbd1" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556843    4544 cpu_manager.go:414] "containerMap: removing container" podUID="75ec2a60-7de0-4cff-9961-989ee7c2961d" containerName="revoke-keys"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556850    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75ec2a60-7de0-4cff-9961-989ee7c2961d" containerName="revoke-keys"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556878    4544 cpu_manager.go:414] "containerMap: removing container" podUID="297febba-5574-48a5-9e02-71dc942697a9" containerName="istio-proxy"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.556885    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="297febba-5574-48a5-9e02-71dc942697a9" containerName="istio-proxy"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.557204    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="75ec2a60-7de0-4cff-9961-989ee7c2961d" containerName="revoke-keys"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.557234    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="297febba-5574-48a5-9e02-71dc942697a9" containerName="istio-proxy"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.557257    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6ae22b0-e9c0-45bc-85e4-76c659da2808" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.557274    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="72837ba1-2075-4553-a926-3716a21270ee" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.557292    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="19f3a006-19e8-44af-9db0-366d400001f3" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.557302    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="32e6a633-93f2-48be-87d6-5a9521132614" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.557309    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6ae22b0-e9c0-45bc-85e4-76c659da2808" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.557325    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="002634ca-d404-4827-b3e7-d007555a2792" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.557335    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bbc446d-8807-4613-9d86-b87b0733d2cf" containerName="istio-proxy"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.557344    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="03d3f1b3-4d1c-4aca-82d5-f106740125ad" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.557361    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="32e6a633-93f2-48be-87d6-5a9521132614" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.557372    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ddc624d-f00d-4c01-b9e1-877259555186" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.557389    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="002634ca-d404-4827-b3e7-d007555a2792" containerName="cleanup"
Oct 30 13:15:39.557369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.557404    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac81161c-cd40-46f4-9fe0-2490f472393c" containerName="cleanup"
Oct 30 13:15:39.560091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.557412    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="03d3f1b3-4d1c-4aca-82d5-f106740125ad" containerName="cleanup"
Oct 30 13:15:39.560091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.557421    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac81161c-cd40-46f4-9fe0-2490f472393c" containerName="cleanup"
Oct 30 13:15:39.560091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.557438    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac81161c-cd40-46f4-9fe0-2490f472393c" containerName="cleanup"
Oct 30 13:15:39.560091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.557457    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ddc624d-f00d-4c01-b9e1-877259555186" containerName="cleanup"
Oct 30 13:15:39.560091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.557465    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="674c045d-1ba0-4075-8ce8-84e3b275dbd1" containerName="cleanup"
Oct 30 13:15:39.560091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.557482    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ddc624d-f00d-4c01-b9e1-877259555186" containerName="cleanup"
Oct 30 13:15:39.560091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.557492    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab79901f-7e31-4e95-a033-6eb1235af1fa" containerName="istio-proxy"
Oct 30 13:15:39.560091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.557501    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="674c045d-1ba0-4075-8ce8-84e3b275dbd1" containerName="cleanup"
Oct 30 13:15:39.560091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.557518    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4c5666f-6f23-4001-a5d4-c305bb5cd950" containerName="payload-pre-processing"
Oct 30 13:15:39.560091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.557536    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="19f3a006-19e8-44af-9db0-366d400001f3" containerName="cleanup"
Oct 30 13:15:39.560091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.557545    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="72837ba1-2075-4553-a926-3716a21270ee" containerName="cleanup"
Oct 30 13:15:39.563488 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.563454    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:39.565980 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.565948    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"]
Oct 30 13:15:39.566133 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.566090    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:15:39.652049 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.652011    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6f9071-openshift-default-dockercfg-vvqfz\""
Oct 30 13:15:39.657870 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.657832    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-workload-certs\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:15:39.658036 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.657898    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istio-envoy\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:15:39.658138 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.658122    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-credential-socket\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:15:39.658243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.658211    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-workload-socket\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:15:39.658333 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.658305    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istio-token\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:15:39.658386 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.658332    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istiod-ca-cert\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:15:39.658444 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.658397    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmjzb\" (UniqueName: \"kubernetes.io/projected/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-kube-api-access-rmjzb\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:15:39.658684 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.658665    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istio-podinfo\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:15:39.658762 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.658703    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istio-data\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:15:39.760598 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.760553    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istio-podinfo\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:15:39.760784 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.760605    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istio-data\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:15:39.760784 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.760703    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-workload-certs\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:15:39.760784 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.760750    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istio-envoy\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:15:39.760953 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.760811    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-credential-socket\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:15:39.760953 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.760880    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-workload-socket\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:15:39.760953 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.760945    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istio-token\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:15:39.761169 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.760970    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istiod-ca-cert\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:15:39.761169 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.761007    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmjzb\" (UniqueName: \"kubernetes.io/projected/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-kube-api-access-rmjzb\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:15:39.761303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.761174    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istio-data\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:15:39.761303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.761236    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-workload-certs\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:15:39.761735 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.761701    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-credential-socket\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:15:39.761913 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.761890    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-workload-socket\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:15:39.762079 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.762039    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istiod-ca-cert\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:15:39.763584 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.763551    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istio-envoy\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:15:39.763795 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.763775    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istio-podinfo\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:15:39.802987 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.802953    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istio-token\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:15:39.803968 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.803936    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmjzb\" (UniqueName: \"kubernetes.io/projected/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-kube-api-access-rmjzb\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:15:39.856245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.853324    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:39.890581 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:39.890540    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:15:40.281912 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:40.281876    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"]
Oct 30 13:15:40.282111 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:15:40.282088    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1253c4f1_9b9a_4a12_b1d2_fc61d9e219ff.slice/crio-92858d1aa2f8579ae45405cf9ab499ae79e94188744076ec1390f74675b71668 WatchSource:0}: Error finding container 92858d1aa2f8579ae45405cf9ab499ae79e94188744076ec1390f74675b71668: Status 404 returned error can't find the container with id 92858d1aa2f8579ae45405cf9ab499ae79e94188744076ec1390f74675b71668
Oct 30 13:15:40.285378 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:40.285338    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:15:40.285506 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:40.285427    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:15:40.285506 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:40.285476    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:15:40.428830 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:40.428563    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:40.597477 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:40.597435    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj" event={"ID":"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff","Type":"ContainerStarted","Data":"0e84581c5e263c206940181471dcfde276bbb741244437eeab53f8d17892d386"}
Oct 30 13:15:40.597886 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:40.597503    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj" event={"ID":"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff","Type":"ContainerStarted","Data":"92858d1aa2f8579ae45405cf9ab499ae79e94188744076ec1390f74675b71668"}
Oct 30 13:15:40.664239 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:40.664179    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj" podStartSLOduration=1.664162562 podStartE2EDuration="1.664162562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:40.659562298 +0000 UTC m=+1835.706696251" watchObservedRunningTime="2026-10-30 13:15:40.664162562 +0000 UTC m=+1835.711296516"
Oct 30 13:15:40.892500 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:40.892406    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:15:41.897724 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:41.897693    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:15:42.638279 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:42.638049    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:15:42.639368 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:42.639345    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:15:42.782253 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:42.782201    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:43.063134 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:43.063047    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:43.289034 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:43.288994    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:43.443083 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:43.443049    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"]
Oct 30 13:15:43.443361 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:43.443334    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz" podUID="613b391b-f5e4-4c93-84ba-70d00d691d27" containerName="istio-proxy" containerID="cri-o://dc2ffadd7fb978b8452e516b301d48013661ce31747e3d9fcd1006d39d7bdd1f" gracePeriod=30
Oct 30 13:15:43.557631 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:43.557595    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:44.503663 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:44.503621    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:45.231688 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.218898    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"]
Oct 30 13:15:45.231688 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.220187    4544 cpu_manager.go:414] "containerMap: removing container" podUID="19f3a006-19e8-44af-9db0-366d400001f3" containerName="cleanup"
Oct 30 13:15:45.231688 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.220209    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19f3a006-19e8-44af-9db0-366d400001f3" containerName="cleanup"
Oct 30 13:15:45.231688 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.220254    4544 cpu_manager.go:414] "containerMap: removing container" podUID="32e6a633-93f2-48be-87d6-5a9521132614" containerName="cleanup"
Oct 30 13:15:45.231688 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.220263    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32e6a633-93f2-48be-87d6-5a9521132614" containerName="cleanup"
Oct 30 13:15:45.231688 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.220281    4544 cpu_manager.go:414] "containerMap: removing container" podUID="1ddc624d-f00d-4c01-b9e1-877259555186" containerName="cleanup"
Oct 30 13:15:45.231688 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.220288    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ddc624d-f00d-4c01-b9e1-877259555186" containerName="cleanup"
Oct 30 13:15:45.231688 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.220321    4544 cpu_manager.go:414] "containerMap: removing container" podUID="674c045d-1ba0-4075-8ce8-84e3b275dbd1" containerName="cleanup"
Oct 30 13:15:45.231688 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.220330    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="674c045d-1ba0-4075-8ce8-84e3b275dbd1" containerName="cleanup"
Oct 30 13:15:45.231688 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.220553    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="72837ba1-2075-4553-a926-3716a21270ee" containerName="cleanup"
Oct 30 13:15:45.231688 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.220568    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6ae22b0-e9c0-45bc-85e4-76c659da2808" containerName="cleanup"
Oct 30 13:15:45.231688 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.220579    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bbc446d-8807-4613-9d86-b87b0733d2cf" containerName="istio-proxy"
Oct 30 13:15:45.231688 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.220591    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="32e6a633-93f2-48be-87d6-5a9521132614" containerName="cleanup"
Oct 30 13:15:45.231688 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.220601    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="002634ca-d404-4827-b3e7-d007555a2792" containerName="cleanup"
Oct 30 13:15:45.231688 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.220611    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="03d3f1b3-4d1c-4aca-82d5-f106740125ad" containerName="cleanup"
Oct 30 13:15:45.231688 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.220630    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="674c045d-1ba0-4075-8ce8-84e3b275dbd1" containerName="cleanup"
Oct 30 13:15:45.231688 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.220639    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="19f3a006-19e8-44af-9db0-366d400001f3" containerName="cleanup"
Oct 30 13:15:45.259241 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.253600    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:15:45.283514 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.283478    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-workload-socket\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:15:45.284261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.284235    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-workload-certs\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:15:45.284495 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.284473    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xmkm\" (UniqueName: \"kubernetes.io/projected/d685dcc9-430c-4f84-aa00-7506306061d1-kube-api-access-7xmkm\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:15:45.286755 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.286727    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d685dcc9-430c-4f84-aa00-7506306061d1-istio-token\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:15:45.287151 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.287131    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d685dcc9-430c-4f84-aa00-7506306061d1-istiod-ca-cert\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:15:45.295267 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.292387    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-credential-socket\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:15:45.295267 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.292431    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-istio-envoy\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:15:45.295267 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.292468    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d685dcc9-430c-4f84-aa00-7506306061d1-istio-podinfo\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:15:45.295267 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.292521    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-istio-data\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:15:45.342742 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.331321    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-80fbd9-openshift-default-dockercfg-c2r4s\""
Oct 30 13:15:45.342742 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.336844    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"]
Oct 30 13:15:45.347709 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.347576    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:45.453244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.420964    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d685dcc9-430c-4f84-aa00-7506306061d1-istiod-ca-cert\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:15:45.474252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.453710    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-credential-socket\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:15:45.482403 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.474521    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-istio-envoy\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:15:45.628460 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.454743    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-credential-socket\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:15:45.628460 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.628450    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d685dcc9-430c-4f84-aa00-7506306061d1-istio-podinfo\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:15:45.628978 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.561638    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:45.628978 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.628557    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-istio-data\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:15:45.628978 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.628877    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-workload-socket\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:15:45.628978 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.628914    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-workload-certs\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:15:45.628978 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.628944    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xmkm\" (UniqueName: \"kubernetes.io/projected/d685dcc9-430c-4f84-aa00-7506306061d1-kube-api-access-7xmkm\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:15:45.628978 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.628975    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d685dcc9-430c-4f84-aa00-7506306061d1-istio-token\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:15:45.629426 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.424172    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d685dcc9-430c-4f84-aa00-7506306061d1-istiod-ca-cert\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:15:45.629808 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.629782    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-istio-data\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:15:45.630006 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.590845    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk"]
Oct 30 13:15:45.631664 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.631633    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-workload-certs\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:15:45.660244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.630062    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-workload-socket\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:15:45.660244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.650899    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d685dcc9-430c-4f84-aa00-7506306061d1-istio-podinfo\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:15:45.660244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.650999    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-istio-envoy\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:15:45.697501 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.697466    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk"]
Oct 30 13:15:45.700533 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.700502    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk"
Oct 30 13:15:45.704857 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.704829    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xmkm\" (UniqueName: \"kubernetes.io/projected/d685dcc9-430c-4f84-aa00-7506306061d1-kube-api-access-7xmkm\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:15:45.723263 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.722542    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-6f9071\""
Oct 30 13:15:45.723263 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.722730    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-6f9071-dockercfg-llhxd\""
Oct 30 13:15:45.812033 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.796804    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d685dcc9-430c-4f84-aa00-7506306061d1-istio-token\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:15:45.856617 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.856557    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs2pv\" (UniqueName: \"kubernetes.io/projected/9564d09c-f00e-48e9-bd57-a96f0bb6efc3-kube-api-access-gs2pv\") pod \"payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk\" (UID: \"9564d09c-f00e-48e9-bd57-a96f0bb6efc3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk"
Oct 30 13:15:45.856945 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.856915    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9564d09c-f00e-48e9-bd57-a96f0bb6efc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk\" (UID: \"9564d09c-f00e-48e9-bd57-a96f0bb6efc3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk"
Oct 30 13:15:45.977137 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.975679    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs2pv\" (UniqueName: \"kubernetes.io/projected/9564d09c-f00e-48e9-bd57-a96f0bb6efc3-kube-api-access-gs2pv\") pod \"payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk\" (UID: \"9564d09c-f00e-48e9-bd57-a96f0bb6efc3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk"
Oct 30 13:15:45.977137 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.975780    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9564d09c-f00e-48e9-bd57-a96f0bb6efc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk\" (UID: \"9564d09c-f00e-48e9-bd57-a96f0bb6efc3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk"
Oct 30 13:15:45.977137 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:45.976855    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9564d09c-f00e-48e9-bd57-a96f0bb6efc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk\" (UID: \"9564d09c-f00e-48e9-bd57-a96f0bb6efc3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk"
Oct 30 13:15:46.060039 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:46.059856    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:15:46.061891 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:46.061859    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs2pv\" (UniqueName: \"kubernetes.io/projected/9564d09c-f00e-48e9-bd57-a96f0bb6efc3-kube-api-access-gs2pv\") pod \"payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk\" (UID: \"9564d09c-f00e-48e9-bd57-a96f0bb6efc3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk"
Oct 30 13:15:46.227665 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:46.227612    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk"
Oct 30 13:15:46.426773 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:46.426736    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:46.484168 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:46.484134    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"]
Oct 30 13:15:46.485426 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:15:46.485393    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd685dcc9_430c_4f84_aa00_7506306061d1.slice/crio-f2a62dd5274fb81f34792a24f26519f9b17c9e4cdd4ca06473b4ed116e424f5c WatchSource:0}: Error finding container f2a62dd5274fb81f34792a24f26519f9b17c9e4cdd4ca06473b4ed116e424f5c: Status 404 returned error can't find the container with id f2a62dd5274fb81f34792a24f26519f9b17c9e4cdd4ca06473b4ed116e424f5c
Oct 30 13:15:46.488514 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:46.488476    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:15:46.488614 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:46.488554    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:15:46.488614 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:46.488588    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:15:46.582636 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:46.580182    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk"]
Oct 30 13:15:46.593265 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:46.588139    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:46.730450 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:46.730335    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f" event={"ID":"d685dcc9-430c-4f84-aa00-7506306061d1","Type":"ContainerStarted","Data":"b61ba5361033a17783ae4e7598056be0f5835e19e3ede4aae9c71d67d1bac2a0"}
Oct 30 13:15:46.730450 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:46.730413    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f" event={"ID":"d685dcc9-430c-4f84-aa00-7506306061d1","Type":"ContainerStarted","Data":"f2a62dd5274fb81f34792a24f26519f9b17c9e4cdd4ca06473b4ed116e424f5c"}
Oct 30 13:15:46.732381 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:46.732348    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk" event={"ID":"9564d09c-f00e-48e9-bd57-a96f0bb6efc3","Type":"ContainerStarted","Data":"862454dff47b6538f309151fec2880ed877a54dd8dcd8447bdd0d2319caa0211"}
Oct 30 13:15:46.792836 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:46.791709    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f" podStartSLOduration=1.791686237 podStartE2EDuration="1.791686237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:46.780336475 +0000 UTC m=+1841.827470485" watchObservedRunningTime="2026-10-30 13:15:46.791686237 +0000 UTC m=+1841.838820193"
Oct 30 13:15:47.035673 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:47.035552    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:15:47.037209 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:47.037173    4544 patch_prober.go:29] interesting pod/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 13:15:47.037361 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:47.037254    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f" podUID="d685dcc9-430c-4f84-aa00-7506306061d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 13:15:47.744971 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:47.744920    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk" event={"ID":"9564d09c-f00e-48e9-bd57-a96f0bb6efc3","Type":"ContainerStarted","Data":"789c1d6bc39f0376e06f2d93f69ad888743d546b10bdf7d662998b8a626fc798"}
Oct 30 13:15:47.745464 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:47.745024    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk"
Oct 30 13:15:47.795645 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:47.795588    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk" podStartSLOduration=2.795572317 podStartE2EDuration="2.795572317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:47.790665842 +0000 UTC m=+1842.837799795" watchObservedRunningTime="2026-10-30 13:15:47.795572317 +0000 UTC m=+1842.842706269"
Oct 30 13:15:48.041028 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:48.040943    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:15:48.622751 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:48.622619    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:48.765384 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:48.765291    4544 generic.go:356] "Generic (PLEG): container finished" podID="613b391b-f5e4-4c93-84ba-70d00d691d27" containerID="dc2ffadd7fb978b8452e516b301d48013661ce31747e3d9fcd1006d39d7bdd1f" exitCode=0
Oct 30 13:15:48.765384 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:48.765366    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz" event={"ID":"613b391b-f5e4-4c93-84ba-70d00d691d27","Type":"ContainerDied","Data":"dc2ffadd7fb978b8452e516b301d48013661ce31747e3d9fcd1006d39d7bdd1f"}
Oct 30 13:15:48.765931 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:48.765447    4544 scope.go:122] "RemoveContainer" containerID="fdd8a7f89d9f07b8ba011c6087fbbadce69c965e14cb3f316853059a30a8b02a"
Oct 30 13:15:48.766088 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:48.766033    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:15:48.768878 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:48.768860    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:15:48.830720 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:48.830693    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:15:48.945650 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:48.943775    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/613b391b-f5e4-4c93-84ba-70d00d691d27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/613b391b-f5e4-4c93-84ba-70d00d691d27-istiod-ca-cert\") pod \"613b391b-f5e4-4c93-84ba-70d00d691d27\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") "
Oct 30 13:15:48.945650 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:48.943836    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-istio-envoy\") pod \"613b391b-f5e4-4c93-84ba-70d00d691d27\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") "
Oct 30 13:15:48.945650 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:48.943883    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/613b391b-f5e4-4c93-84ba-70d00d691d27-istio-token\" (UniqueName: \"kubernetes.io/projected/613b391b-f5e4-4c93-84ba-70d00d691d27-istio-token\") pod \"613b391b-f5e4-4c93-84ba-70d00d691d27\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") "
Oct 30 13:15:48.945650 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:48.943979    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-credential-socket\") pod \"613b391b-f5e4-4c93-84ba-70d00d691d27\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") "
Oct 30 13:15:48.945650 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:48.944008    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-workload-socket\") pod \"613b391b-f5e4-4c93-84ba-70d00d691d27\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") "
Oct 30 13:15:48.945650 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:48.944200    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/613b391b-f5e4-4c93-84ba-70d00d691d27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/613b391b-f5e4-4c93-84ba-70d00d691d27-istio-podinfo\") pod \"613b391b-f5e4-4c93-84ba-70d00d691d27\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") "
Oct 30 13:15:48.945650 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:48.944322    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/613b391b-f5e4-4c93-84ba-70d00d691d27-kube-api-access-gqcsq\" (UniqueName: \"kubernetes.io/projected/613b391b-f5e4-4c93-84ba-70d00d691d27-kube-api-access-gqcsq\") pod \"613b391b-f5e4-4c93-84ba-70d00d691d27\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") "
Oct 30 13:15:48.945650 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:48.944359    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-workload-certs\") pod \"613b391b-f5e4-4c93-84ba-70d00d691d27\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") "
Oct 30 13:15:48.945650 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:48.944463    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-istio-data\") pod \"613b391b-f5e4-4c93-84ba-70d00d691d27\" (UID: \"613b391b-f5e4-4c93-84ba-70d00d691d27\") "
Oct 30 13:15:48.947314 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:48.947106    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613b391b-f5e4-4c93-84ba-70d00d691d27-istio-token" pod "613b391b-f5e4-4c93-84ba-70d00d691d27" (UID: "613b391b-f5e4-4c93-84ba-70d00d691d27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:48.947314 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:48.947139    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/613b391b-f5e4-4c93-84ba-70d00d691d27-istiod-ca-cert" pod "613b391b-f5e4-4c93-84ba-70d00d691d27" (UID: "613b391b-f5e4-4c93-84ba-70d00d691d27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:48.947836 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:48.947741    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-istio-data" pod "613b391b-f5e4-4c93-84ba-70d00d691d27" (UID: "613b391b-f5e4-4c93-84ba-70d00d691d27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:48.948500 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:48.948183    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-credential-socket" pod "613b391b-f5e4-4c93-84ba-70d00d691d27" (UID: "613b391b-f5e4-4c93-84ba-70d00d691d27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:48.949180 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:48.948950    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-workload-socket" pod "613b391b-f5e4-4c93-84ba-70d00d691d27" (UID: "613b391b-f5e4-4c93-84ba-70d00d691d27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:48.949694 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:48.949674    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-workload-certs" pod "613b391b-f5e4-4c93-84ba-70d00d691d27" (UID: "613b391b-f5e4-4c93-84ba-70d00d691d27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:48.952653 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:48.952623    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-istio-envoy" pod "613b391b-f5e4-4c93-84ba-70d00d691d27" (UID: "613b391b-f5e4-4c93-84ba-70d00d691d27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:48.953633 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:48.953603    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/613b391b-f5e4-4c93-84ba-70d00d691d27-istio-podinfo" pod "613b391b-f5e4-4c93-84ba-70d00d691d27" (UID: "613b391b-f5e4-4c93-84ba-70d00d691d27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:15:48.957954 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:48.957917    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613b391b-f5e4-4c93-84ba-70d00d691d27-kube-api-access-gqcsq" pod "613b391b-f5e4-4c93-84ba-70d00d691d27" (UID: "613b391b-f5e4-4c93-84ba-70d00d691d27"). InnerVolumeSpecName "kube-api-access-gqcsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:49.058733 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:49.049402    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqcsq\" (UniqueName: \"kubernetes.io/projected/613b391b-f5e4-4c93-84ba-70d00d691d27-kube-api-access-gqcsq\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:49.058733 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:49.049441    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:49.058733 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:49.049459    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:49.058733 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:49.049479    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/613b391b-f5e4-4c93-84ba-70d00d691d27-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:49.058733 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:49.049495    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:49.058733 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:49.049507    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/613b391b-f5e4-4c93-84ba-70d00d691d27-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:49.058733 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:49.049521    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:49.058733 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:49.049539    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/613b391b-f5e4-4c93-84ba-70d00d691d27-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:49.058733 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:49.049552    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/613b391b-f5e4-4c93-84ba-70d00d691d27-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:15:49.385089 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:49.385024    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:49.488375 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:49.488345    4544 patch_prober.go:29] interesting pod/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 30 13:15:49.488526 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:49.488405    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v" podUID="4fc60c82-a658-4c5d-9dd4-a4eae3af4846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 30 13:15:49.573198 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:49.573160    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:49.775139 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:49.775022    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"
Oct 30 13:15:49.775139 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:49.775069    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz" event={"ID":"613b391b-f5e4-4c93-84ba-70d00d691d27","Type":"ContainerDied","Data":"01af02814413c9477fc2a80bf558a94da0bbf93388d32aaa3d02d75e44b07ab7"}
Oct 30 13:15:49.775743 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:49.775151    4544 scope.go:122] "RemoveContainer" containerID="dc2ffadd7fb978b8452e516b301d48013661ce31747e3d9fcd1006d39d7bdd1f"
Oct 30 13:15:49.777084 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:49.777057    4544 generic.go:356] "Generic (PLEG): container finished" podID="4fc60c82-a658-4c5d-9dd4-a4eae3af4846" containerID="8eb07ad10148fc577f31de8b2c4715f2668e48fbb32527af0083cbf48d637553" exitCode=0
Oct 30 13:15:49.777212 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:49.777124    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v" event={"ID":"4fc60c82-a658-4c5d-9dd4-a4eae3af4846","Type":"ContainerDied","Data":"8eb07ad10148fc577f31de8b2c4715f2668e48fbb32527af0083cbf48d637553"}
Oct 30 13:15:49.777717 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:49.777694    4544 scope.go:122] "RemoveContainer" containerID="8eb07ad10148fc577f31de8b2c4715f2668e48fbb32527af0083cbf48d637553"
Oct 30 13:15:49.780039 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:49.780001    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:15:49.780139 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:49.780101    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:15:49.780188 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:49.780151    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:15:49.846585 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:49.846556    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"]
Oct 30 13:15:49.851645 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:49.851621    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz"]
Oct 30 13:15:50.791632 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:50.791599    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v" event={"ID":"4fc60c82-a658-4c5d-9dd4-a4eae3af4846","Type":"ContainerStarted","Data":"a41fea898496fdd20f3ebf27e37c1b9fea9c32967ed3ae16660249892a6a1f82"}
Oct 30 13:15:50.965619 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:50.965586    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:15:50.965883 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:50.965786    4544 patch_prober.go:29] interesting pod/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 30 13:15:50.965883 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:50.965840    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v" podUID="4fc60c82-a658-4c5d-9dd4-a4eae3af4846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 30 13:15:51.091440 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:51.091384    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="613b391b-f5e4-4c93-84ba-70d00d691d27" path="/var/lib/kubelet/pods/613b391b-f5e4-4c93-84ba-70d00d691d27/volumes"
Oct 30 13:15:51.228245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:51.222713    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:51.394112 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:51.393539    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:51.536248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:51.533849    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:51.567712 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:51.567679    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4"]
Oct 30 13:15:51.568513 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:51.568497    4544 cpu_manager.go:414] "containerMap: removing container" podUID="613b391b-f5e4-4c93-84ba-70d00d691d27" containerName="istio-proxy"
Oct 30 13:15:51.568592 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:51.568515    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="613b391b-f5e4-4c93-84ba-70d00d691d27" containerName="istio-proxy"
Oct 30 13:15:51.568592 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:51.568554    4544 cpu_manager.go:414] "containerMap: removing container" podUID="613b391b-f5e4-4c93-84ba-70d00d691d27" containerName="istio-proxy"
Oct 30 13:15:51.568592 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:51.568560    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="613b391b-f5e4-4c93-84ba-70d00d691d27" containerName="istio-proxy"
Oct 30 13:15:51.568702 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:51.568682    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="613b391b-f5e4-4c93-84ba-70d00d691d27" containerName="istio-proxy"
Oct 30 13:15:51.568702 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:51.568694    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="613b391b-f5e4-4c93-84ba-70d00d691d27" containerName="istio-proxy"
Oct 30 13:15:51.572743 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:51.572722    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4"
Oct 30 13:15:51.589723 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:51.589701    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-80fbd9-dockercfg-flrnv\""
Oct 30 13:15:51.590009 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:51.589799    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-80fbd9\""
Oct 30 13:15:51.644407 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:51.632664    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4"]
Oct 30 13:15:51.777327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:51.763698    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swq8w\" (UniqueName: \"kubernetes.io/projected/6abef82a-570c-494b-afe4-83bf81a4330c-kube-api-access-swq8w\") pod \"payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4\" (UID: \"6abef82a-570c-494b-afe4-83bf81a4330c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4"
Oct 30 13:15:51.796246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:51.778500    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6abef82a-570c-494b-afe4-83bf81a4330c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4\" (UID: \"6abef82a-570c-494b-afe4-83bf81a4330c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4"
Oct 30 13:15:51.816754 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:51.779781    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4"]
Oct 30 13:15:52.071775 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:52.071657    4544 patch_prober.go:29] interesting pod/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 30 13:15:52.071775 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:52.071752    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v" podUID="4fc60c82-a658-4c5d-9dd4-a4eae3af4846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 30 13:15:52.082655 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:52.082611    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6abef82a-570c-494b-afe4-83bf81a4330c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4\" (UID: \"6abef82a-570c-494b-afe4-83bf81a4330c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4"
Oct 30 13:15:52.082925 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:52.082908    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swq8w\" (UniqueName: \"kubernetes.io/projected/6abef82a-570c-494b-afe4-83bf81a4330c-kube-api-access-swq8w\") pod \"payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4\" (UID: \"6abef82a-570c-494b-afe4-83bf81a4330c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4"
Oct 30 13:15:52.084810 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:52.084773    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6abef82a-570c-494b-afe4-83bf81a4330c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4\" (UID: \"6abef82a-570c-494b-afe4-83bf81a4330c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4"
Oct 30 13:15:52.094199 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:52.090824    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4"]
Oct 30 13:15:52.094199 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:52.090958    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4"
Oct 30 13:15:52.149245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:52.148265    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swq8w\" (UniqueName: \"kubernetes.io/projected/6abef82a-570c-494b-afe4-83bf81a4330c-kube-api-access-swq8w\") pod \"payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4\" (UID: \"6abef82a-570c-494b-afe4-83bf81a4330c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4"
Oct 30 13:15:52.188495 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:52.188441    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4"
Oct 30 13:15:52.189030 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:52.189002    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/382ed63d-b520-4d49-ba0b-5fc982f309df-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4\" (UID: \"382ed63d-b520-4d49-ba0b-5fc982f309df\") " pod="openshift-ingress/payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4"
Oct 30 13:15:52.189437 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:52.189409    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gg9b\" (UniqueName: \"kubernetes.io/projected/382ed63d-b520-4d49-ba0b-5fc982f309df-kube-api-access-2gg9b\") pod \"payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4\" (UID: \"382ed63d-b520-4d49-ba0b-5fc982f309df\") " pod="openshift-ingress/payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4"
Oct 30 13:15:52.293958 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:52.293656    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/382ed63d-b520-4d49-ba0b-5fc982f309df-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4\" (UID: \"382ed63d-b520-4d49-ba0b-5fc982f309df\") " pod="openshift-ingress/payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4"
Oct 30 13:15:52.295449 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:52.295293    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/382ed63d-b520-4d49-ba0b-5fc982f309df-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4\" (UID: \"382ed63d-b520-4d49-ba0b-5fc982f309df\") " pod="openshift-ingress/payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4"
Oct 30 13:15:52.295449 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:52.295346    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gg9b\" (UniqueName: \"kubernetes.io/projected/382ed63d-b520-4d49-ba0b-5fc982f309df-kube-api-access-2gg9b\") pod \"payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4\" (UID: \"382ed63d-b520-4d49-ba0b-5fc982f309df\") " pod="openshift-ingress/payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4"
Oct 30 13:15:52.341356 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:52.340994    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gg9b\" (UniqueName: \"kubernetes.io/projected/382ed63d-b520-4d49-ba0b-5fc982f309df-kube-api-access-2gg9b\") pod \"payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4\" (UID: \"382ed63d-b520-4d49-ba0b-5fc982f309df\") " pod="openshift-ingress/payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4"
Oct 30 13:15:52.437580 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:52.437522    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4"
Oct 30 13:15:52.578235 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:52.578104    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4"]
Oct 30 13:15:52.775918 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:52.775886    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4"]
Oct 30 13:15:52.777072 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:15:52.777044    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod382ed63d_b520_4d49_ba0b_5fc982f309df.slice/crio-072273e70a0a2732ed43b1975f98f02fe0ce84590faf1db3ae7afb6384816cdf WatchSource:0}: Error finding container 072273e70a0a2732ed43b1975f98f02fe0ce84590faf1db3ae7afb6384816cdf: Status 404 returned error can't find the container with id 072273e70a0a2732ed43b1975f98f02fe0ce84590faf1db3ae7afb6384816cdf
Oct 30 13:15:52.966849 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:52.966817    4544 patch_prober.go:29] interesting pod/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 30 13:15:52.967387 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:52.966871    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v" podUID="4fc60c82-a658-4c5d-9dd4-a4eae3af4846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 30 13:15:53.094354 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:53.094311    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4" event={"ID":"6abef82a-570c-494b-afe4-83bf81a4330c","Type":"ContainerStarted","Data":"480a0800895fd605124602bced6e4eaca04b4fb29d86ee75a55468e9d0b2004f"}
Oct 30 13:15:53.094570 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:53.094377    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4" event={"ID":"6abef82a-570c-494b-afe4-83bf81a4330c","Type":"ContainerStarted","Data":"e7208a35170642ec818a681096b28ea065210cb8241c69cb575274aa1ed836b6"}
Oct 30 13:15:53.094570 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:53.094551    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4"
Oct 30 13:15:53.096732 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:53.096698    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4" event={"ID":"382ed63d-b520-4d49-ba0b-5fc982f309df","Type":"ContainerStarted","Data":"f5cc933dd0777184a3067958f2af3c778e158ded5c5e7cfaa03a2b9c3363fd26"}
Oct 30 13:15:53.096876 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:53.096753    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4" event={"ID":"382ed63d-b520-4d49-ba0b-5fc982f309df","Type":"ContainerStarted","Data":"072273e70a0a2732ed43b1975f98f02fe0ce84590faf1db3ae7afb6384816cdf"}
Oct 30 13:15:53.096876 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:53.096800    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4"
Oct 30 13:15:53.149485 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:53.149377    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4" podStartSLOduration=2.149362298 podStartE2EDuration="2.149362298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:53.148020888 +0000 UTC m=+1848.195154842" watchObservedRunningTime="2026-10-30 13:15:53.149362298 +0000 UTC m=+1848.196496251"
Oct 30 13:15:53.198110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:53.198040    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4" podStartSLOduration=2.198022192 podStartE2EDuration="2.198022192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:53.194889307 +0000 UTC m=+1848.242023262" watchObservedRunningTime="2026-10-30 13:15:53.198022192 +0000 UTC m=+1848.245156148"
Oct 30 13:15:53.764844 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:53.764808    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:53.966132 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:53.966092    4544 patch_prober.go:29] interesting pod/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 30 13:15:53.966361 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:53.966155    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v" podUID="4fc60c82-a658-4c5d-9dd4-a4eae3af4846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 30 13:15:54.965969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:54.965936    4544 patch_prober.go:29] interesting pod/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 30 13:15:54.966553 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:54.966006    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v" podUID="4fc60c82-a658-4c5d-9dd4-a4eae3af4846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 30 13:15:55.198674 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:55.198635    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:55.934874 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:55.934831    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:55.976247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:55.970422    4544 patch_prober.go:29] interesting pod/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 30 13:15:55.976247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:55.970502    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v" podUID="4fc60c82-a658-4c5d-9dd4-a4eae3af4846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 30 13:15:56.971608 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:56.971573    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:15:56.972067 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:56.972047    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:15:56.972728 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:56.972710    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:15:57.900981 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:57.900930    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:58.046123 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:58.046082    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:15:58.770022 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:58.769992    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk"
Oct 30 13:15:59.008583 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:15:59.008542    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:15:59.986329 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:15:59.986287    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:01.938746 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:01.938704    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:02.680675 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:02.680638    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:02.838503 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:02.838466    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:03.942551 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:03.942445    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:04.109031 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:04.109002    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4"
Oct 30 13:16:04.109213 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:04.109079    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4"
Oct 30 13:16:05.955557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:05.955501    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:07.725583 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:07.725543    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:09.548875 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:09.548834    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:11.271521 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:11.271480    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:12.871185 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:12.871157    4544 scope.go:122] "RemoveContainer" containerID="9f90330de2f8b30896bb0655a6865d51c6a3f94e5dd491ca35a7111112eddf6b"
Oct 30 13:16:12.882661 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:12.882638    4544 scope.go:122] "RemoveContainer" containerID="d1517540eed9dbb7bccd957e276759e40137460f0660895410b3e5d34ea97de1"
Oct 30 13:16:13.462244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:13.461983    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:15.463895 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:15.463849    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:17.017078 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:17.017042    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:17.410702 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:17.410666    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:18.288291 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:18.288216    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:19.413244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:19.407829    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:19.856837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:19.856801    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:20.875556 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:20.875503    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7"]
Oct 30 13:16:20.876063 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:20.875773    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7" podUID="faeaf797-0836-49db-a548-6b7dc6faa766" containerName="maas-api" containerID="cri-o://2daff1df0dfa287b042d265ae5830aeb645fb67c961e79964519711fdab9e532" gracePeriod=30
Oct 30 13:16:20.948135 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:20.948085    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:21.359925 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:21.359895    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7"
Oct 30 13:16:21.537246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:21.529295    4544 generic.go:356] "Generic (PLEG): container finished" podID="faeaf797-0836-49db-a548-6b7dc6faa766" containerID="2daff1df0dfa287b042d265ae5830aeb645fb67c961e79964519711fdab9e532" exitCode=0
Oct 30 13:16:21.537246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:21.529670    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7" event={"ID":"faeaf797-0836-49db-a548-6b7dc6faa766","Type":"ContainerDied","Data":"2daff1df0dfa287b042d265ae5830aeb645fb67c961e79964519711fdab9e532"}
Oct 30 13:16:21.537246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:21.529737    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7" event={"ID":"faeaf797-0836-49db-a548-6b7dc6faa766","Type":"ContainerDied","Data":"eae57e37223c3ef5f1633d1eda3d6f523c0378ff99ad9606671fbc6c176ccdbe"}
Oct 30 13:16:21.537246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:21.529780    4544 scope.go:122] "RemoveContainer" containerID="2daff1df0dfa287b042d265ae5830aeb645fb67c961e79964519711fdab9e532"
Oct 30 13:16:21.537246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:21.530042    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7"
Oct 30 13:16:21.552308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:21.549289    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faeaf797-0836-49db-a548-6b7dc6faa766-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/faeaf797-0836-49db-a548-6b7dc6faa766-openshift-service-ca\") pod \"faeaf797-0836-49db-a548-6b7dc6faa766\" (UID: \"faeaf797-0836-49db-a548-6b7dc6faa766\") "
Oct 30 13:16:21.552308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:21.549470    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/faeaf797-0836-49db-a548-6b7dc6faa766-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/faeaf797-0836-49db-a548-6b7dc6faa766-maas-api-tls\") pod \"faeaf797-0836-49db-a548-6b7dc6faa766\" (UID: \"faeaf797-0836-49db-a548-6b7dc6faa766\") "
Oct 30 13:16:21.552308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:21.549804    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/faeaf797-0836-49db-a548-6b7dc6faa766-metrics-certs\" (UniqueName: \"kubernetes.io/secret/faeaf797-0836-49db-a548-6b7dc6faa766-metrics-certs\") pod \"faeaf797-0836-49db-a548-6b7dc6faa766\" (UID: \"faeaf797-0836-49db-a548-6b7dc6faa766\") "
Oct 30 13:16:21.552308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:21.550053    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faeaf797-0836-49db-a548-6b7dc6faa766-kube-api-access-zktcv\" (UniqueName: \"kubernetes.io/projected/faeaf797-0836-49db-a548-6b7dc6faa766-kube-api-access-zktcv\") pod \"faeaf797-0836-49db-a548-6b7dc6faa766\" (UID: \"faeaf797-0836-49db-a548-6b7dc6faa766\") "
Oct 30 13:16:21.552308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:21.550802    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faeaf797-0836-49db-a548-6b7dc6faa766-openshift-service-ca" pod "faeaf797-0836-49db-a548-6b7dc6faa766" (UID: "faeaf797-0836-49db-a548-6b7dc6faa766"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:16:21.566297 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:21.554208    4544 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/faeaf797-0836-49db-a548-6b7dc6faa766-openshift-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:16:21.596602 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:21.573386    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faeaf797-0836-49db-a548-6b7dc6faa766-metrics-certs" pod "faeaf797-0836-49db-a548-6b7dc6faa766" (UID: "faeaf797-0836-49db-a548-6b7dc6faa766"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:16:21.596602 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:21.573426    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faeaf797-0836-49db-a548-6b7dc6faa766-kube-api-access-zktcv" pod "faeaf797-0836-49db-a548-6b7dc6faa766" (UID: "faeaf797-0836-49db-a548-6b7dc6faa766"). InnerVolumeSpecName "kube-api-access-zktcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:16:21.596602 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:21.592376    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faeaf797-0836-49db-a548-6b7dc6faa766-maas-api-tls" pod "faeaf797-0836-49db-a548-6b7dc6faa766" (UID: "faeaf797-0836-49db-a548-6b7dc6faa766"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:16:21.596602 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:21.592795    4544 scope.go:122] "RemoveContainer" containerID="2daff1df0dfa287b042d265ae5830aeb645fb67c961e79964519711fdab9e532"
Oct 30 13:16:21.628245 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:16:21.601676    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2daff1df0dfa287b042d265ae5830aeb645fb67c961e79964519711fdab9e532\": container with ID starting with 2daff1df0dfa287b042d265ae5830aeb645fb67c961e79964519711fdab9e532 not found: ID does not exist" containerID="2daff1df0dfa287b042d265ae5830aeb645fb67c961e79964519711fdab9e532"
Oct 30 13:16:21.628245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:21.601730    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2daff1df0dfa287b042d265ae5830aeb645fb67c961e79964519711fdab9e532"} err="failed to get container status \"2daff1df0dfa287b042d265ae5830aeb645fb67c961e79964519711fdab9e532\": rpc error: code = NotFound desc = could not find container \"2daff1df0dfa287b042d265ae5830aeb645fb67c961e79964519711fdab9e532\": container with ID starting with 2daff1df0dfa287b042d265ae5830aeb645fb67c961e79964519711fdab9e532 not found: ID does not exist"
Oct 30 13:16:21.657208 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:21.655715    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/faeaf797-0836-49db-a548-6b7dc6faa766-maas-api-tls\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:16:21.657208 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:21.655751    4544 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/faeaf797-0836-49db-a548-6b7dc6faa766-metrics-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:16:21.657208 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:21.655765    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zktcv\" (UniqueName: \"kubernetes.io/projected/faeaf797-0836-49db-a548-6b7dc6faa766-kube-api-access-zktcv\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:16:21.822898 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:21.822872    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:21.880437 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:21.880342    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7"]
Oct 30 13:16:21.887452 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:21.887421    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5b9e8e-76d9c4b95d-wj6b7"]
Oct 30 13:16:21.976818 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:21.976773    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:22.105025 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:22.103082    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:22.612110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:22.612073    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:23.089434 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:23.089399    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faeaf797-0836-49db-a548-6b7dc6faa766" path="/var/lib/kubelet/pods/faeaf797-0836-49db-a548-6b7dc6faa766/volumes"
Oct 30 13:16:24.086104 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:24.086063    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:25.848736 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:25.848686    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:27.985429 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:27.985393    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:29.041491 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:16:29.041452    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:16:29.945967 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:29.945929    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6"]
Oct 30 13:16:29.946792 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:29.946775    4544 cpu_manager.go:414] "containerMap: removing container" podUID="faeaf797-0836-49db-a548-6b7dc6faa766" containerName="maas-api"
Oct 30 13:16:29.946907 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:29.946792    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faeaf797-0836-49db-a548-6b7dc6faa766" containerName="maas-api"
Oct 30 13:16:29.946950 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:29.946941    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="faeaf797-0836-49db-a548-6b7dc6faa766" containerName="maas-api"
Oct 30 13:16:29.950926 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:29.950906    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6"
Oct 30 13:16:30.004488 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:30.004461    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-6f9071-serving-cert\""
Oct 30 13:16:30.040974 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:30.040786    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40c596cb-37ac-468d-828a-65708ed129f1-maas-api-tls\") pod \"maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6\" (UID: \"40c596cb-37ac-468d-828a-65708ed129f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6"
Oct 30 13:16:30.040974 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:30.040891    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40c596cb-37ac-468d-828a-65708ed129f1-metrics-certs\") pod \"maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6\" (UID: \"40c596cb-37ac-468d-828a-65708ed129f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6"
Oct 30 13:16:30.041254 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:30.041138    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40c596cb-37ac-468d-828a-65708ed129f1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6\" (UID: \"40c596cb-37ac-468d-828a-65708ed129f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6"
Oct 30 13:16:30.041355 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:30.041331    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x9hp\" (UniqueName: \"kubernetes.io/projected/40c596cb-37ac-468d-828a-65708ed129f1-kube-api-access-9x9hp\") pod \"maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6\" (UID: \"40c596cb-37ac-468d-828a-65708ed129f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6"
Oct 30 13:16:30.045128 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:30.045106    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6"]
Oct 30 13:16:30.143468 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:30.143427    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x9hp\" (UniqueName: \"kubernetes.io/projected/40c596cb-37ac-468d-828a-65708ed129f1-kube-api-access-9x9hp\") pod \"maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6\" (UID: \"40c596cb-37ac-468d-828a-65708ed129f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6"
Oct 30 13:16:30.143843 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:30.143817    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40c596cb-37ac-468d-828a-65708ed129f1-maas-api-tls\") pod \"maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6\" (UID: \"40c596cb-37ac-468d-828a-65708ed129f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6"
Oct 30 13:16:30.143980 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:30.143893    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40c596cb-37ac-468d-828a-65708ed129f1-metrics-certs\") pod \"maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6\" (UID: \"40c596cb-37ac-468d-828a-65708ed129f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6"
Oct 30 13:16:30.144054 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:30.144010    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40c596cb-37ac-468d-828a-65708ed129f1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6\" (UID: \"40c596cb-37ac-468d-828a-65708ed129f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6"
Oct 30 13:16:30.144348 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:16:30.144325    4544 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-6f9071-serving-cert: secret "maas-api-e2e-inf-b-6f9071-serving-cert" not found
Oct 30 13:16:30.144558 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:16:30.144538    4544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40c596cb-37ac-468d-828a-65708ed129f1-maas-api-tls podName:40c596cb-37ac-468d-828a-65708ed129f1 nodeName:}" failed. No retries permitted until 2026-10-30 13:16:30.64451313 +0000 UTC m=+1885.691647080 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/40c596cb-37ac-468d-828a-65708ed129f1-maas-api-tls") pod "maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6" (UID: "40c596cb-37ac-468d-828a-65708ed129f1") : secret "maas-api-e2e-inf-b-6f9071-serving-cert" not found
Oct 30 13:16:30.144866 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:30.144847    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40c596cb-37ac-468d-828a-65708ed129f1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6\" (UID: \"40c596cb-37ac-468d-828a-65708ed129f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6"
Oct 30 13:16:30.146934 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:30.146909    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40c596cb-37ac-468d-828a-65708ed129f1-metrics-certs\") pod \"maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6\" (UID: \"40c596cb-37ac-468d-828a-65708ed129f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6"
Oct 30 13:16:30.177848 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:30.177819    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x9hp\" (UniqueName: \"kubernetes.io/projected/40c596cb-37ac-468d-828a-65708ed129f1-kube-api-access-9x9hp\") pod \"maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6\" (UID: \"40c596cb-37ac-468d-828a-65708ed129f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6"
Oct 30 13:16:30.236304 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:30.231890    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:30.445256 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:30.441115    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:30.652156 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:30.652121    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40c596cb-37ac-468d-828a-65708ed129f1-maas-api-tls\") pod \"maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6\" (UID: \"40c596cb-37ac-468d-828a-65708ed129f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6"
Oct 30 13:16:30.654842 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:30.654812    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40c596cb-37ac-468d-828a-65708ed129f1-maas-api-tls\") pod \"maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6\" (UID: \"40c596cb-37ac-468d-828a-65708ed129f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6"
Oct 30 13:16:30.865808 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:30.865773    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6"
Oct 30 13:16:31.041610 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:31.041534    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:31.262371 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:31.252829    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6"]
Oct 30 13:16:31.276430 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:16:31.276397    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c596cb_37ac_468d_828a_65708ed129f1.slice/crio-b1289922a499577c54d7ad6a841e56049f59c06b11e370a9fcdc756f6a75736a WatchSource:0}: Error finding container b1289922a499577c54d7ad6a841e56049f59c06b11e370a9fcdc756f6a75736a: Status 404 returned error can't find the container with id b1289922a499577c54d7ad6a841e56049f59c06b11e370a9fcdc756f6a75736a
Oct 30 13:16:31.277907 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:31.277888    4544 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:16:31.336954 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:31.336906    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:31.502358 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:31.502253    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"]
Oct 30 13:16:31.502563 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:31.502494    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l" podUID="eaa6149f-3c75-4666-8607-5d2686a65b6d" containerName="istio-proxy" containerID="cri-o://2386d332bfb4479225eef9bbe398aea56a0926ee89a1875c8ab7148cb8c8c474" gracePeriod=30
Oct 30 13:16:31.861266 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:31.854185    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:32.018967 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:32.014560    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6" event={"ID":"40c596cb-37ac-468d-828a-65708ed129f1","Type":"ContainerStarted","Data":"b1289922a499577c54d7ad6a841e56049f59c06b11e370a9fcdc756f6a75736a"}
Oct 30 13:16:32.653138 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:32.653100    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:33.076260 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:33.076206    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6" event={"ID":"40c596cb-37ac-468d-828a-65708ed129f1","Type":"ContainerStarted","Data":"41f3b93a49e85a993611ba1dcea23d74022356084a07c5ab2e5a97f4925e3dcd"}
Oct 30 13:16:33.076734 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:33.076708    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6"
Oct 30 13:16:33.202241 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:33.172915    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6" podStartSLOduration=3.448226623 podStartE2EDuration="4.172896973s" totalImagesPullingTime="724.67035ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:33.1648452 +0000 UTC m=+1888.211979156" watchObservedRunningTime="2026-10-30 13:16:33.172896973 +0000 UTC m=+1888.220030929"
Oct 30 13:16:33.898092 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:33.898057    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:34.140285 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:34.129798    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:34.439261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:34.439164    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:34.739018 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:34.738918    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:36.907486 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:36.907387    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:16:36.973417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:36.973381    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-workload-certs\") pod \"eaa6149f-3c75-4666-8607-5d2686a65b6d\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") "
Oct 30 13:16:36.973607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:36.973432    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaa6149f-3c75-4666-8607-5d2686a65b6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa6149f-3c75-4666-8607-5d2686a65b6d-istiod-ca-cert\") pod \"eaa6149f-3c75-4666-8607-5d2686a65b6d\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") "
Oct 30 13:16:36.973607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:36.973473    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-credential-socket\") pod \"eaa6149f-3c75-4666-8607-5d2686a65b6d\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") "
Oct 30 13:16:36.973607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:36.973517    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaa6149f-3c75-4666-8607-5d2686a65b6d-istio-token\" (UniqueName: \"kubernetes.io/projected/eaa6149f-3c75-4666-8607-5d2686a65b6d-istio-token\") pod \"eaa6149f-3c75-4666-8607-5d2686a65b6d\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") "
Oct 30 13:16:36.973607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:36.973538    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-istio-data\") pod \"eaa6149f-3c75-4666-8607-5d2686a65b6d\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") "
Oct 30 13:16:36.973841 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:36.973668    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eaa6149f-3c75-4666-8607-5d2686a65b6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa6149f-3c75-4666-8607-5d2686a65b6d-istio-podinfo\") pod \"eaa6149f-3c75-4666-8607-5d2686a65b6d\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") "
Oct 30 13:16:36.973841 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:36.973727    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-workload-socket\") pod \"eaa6149f-3c75-4666-8607-5d2686a65b6d\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") "
Oct 30 13:16:36.973841 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:36.973754    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-workload-certs" pod "eaa6149f-3c75-4666-8607-5d2686a65b6d" (UID: "eaa6149f-3c75-4666-8607-5d2686a65b6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:16:36.973841 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:36.973786    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaa6149f-3c75-4666-8607-5d2686a65b6d-kube-api-access-57mbt\" (UniqueName: \"kubernetes.io/projected/eaa6149f-3c75-4666-8607-5d2686a65b6d-kube-api-access-57mbt\") pod \"eaa6149f-3c75-4666-8607-5d2686a65b6d\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") "
Oct 30 13:16:36.974063 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:36.973857    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-istio-envoy\") pod \"eaa6149f-3c75-4666-8607-5d2686a65b6d\" (UID: \"eaa6149f-3c75-4666-8607-5d2686a65b6d\") "
Oct 30 13:16:36.974063 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:36.973899    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaa6149f-3c75-4666-8607-5d2686a65b6d-istiod-ca-cert" pod "eaa6149f-3c75-4666-8607-5d2686a65b6d" (UID: "eaa6149f-3c75-4666-8607-5d2686a65b6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:16:36.974190 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:36.974062    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-workload-socket" pod "eaa6149f-3c75-4666-8607-5d2686a65b6d" (UID: "eaa6149f-3c75-4666-8607-5d2686a65b6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:16:36.974335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:36.973894    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-credential-socket" pod "eaa6149f-3c75-4666-8607-5d2686a65b6d" (UID: "eaa6149f-3c75-4666-8607-5d2686a65b6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:16:36.974474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:36.974302    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-istio-data" pod "eaa6149f-3c75-4666-8607-5d2686a65b6d" (UID: "eaa6149f-3c75-4666-8607-5d2686a65b6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:16:36.974703 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:36.974681    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:16:36.974772 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:36.974710    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa6149f-3c75-4666-8607-5d2686a65b6d-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:16:36.974772 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:36.974725    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:16:36.974772 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:36.974741    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:16:36.974772 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:36.974756    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:16:36.976350 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:36.976318    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa6149f-3c75-4666-8607-5d2686a65b6d-istio-token" pod "eaa6149f-3c75-4666-8607-5d2686a65b6d" (UID: "eaa6149f-3c75-4666-8607-5d2686a65b6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:16:36.976350 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:36.976322    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-istio-envoy" pod "eaa6149f-3c75-4666-8607-5d2686a65b6d" (UID: "eaa6149f-3c75-4666-8607-5d2686a65b6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:16:36.976522 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:36.976399    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eaa6149f-3c75-4666-8607-5d2686a65b6d-istio-podinfo" pod "eaa6149f-3c75-4666-8607-5d2686a65b6d" (UID: "eaa6149f-3c75-4666-8607-5d2686a65b6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:16:36.976522 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:36.976429    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa6149f-3c75-4666-8607-5d2686a65b6d-kube-api-access-57mbt" pod "eaa6149f-3c75-4666-8607-5d2686a65b6d" (UID: "eaa6149f-3c75-4666-8607-5d2686a65b6d"). InnerVolumeSpecName "kube-api-access-57mbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:16:37.014881 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:37.014846    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:37.076532 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:37.076503    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaa6149f-3c75-4666-8607-5d2686a65b6d-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:16:37.076532 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:37.076536    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa6149f-3c75-4666-8607-5d2686a65b6d-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:16:37.076735 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:37.076552    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57mbt\" (UniqueName: \"kubernetes.io/projected/eaa6149f-3c75-4666-8607-5d2686a65b6d-kube-api-access-57mbt\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:16:37.076735 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:37.076568    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa6149f-3c75-4666-8607-5d2686a65b6d-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:16:37.130534 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:37.130493    4544 generic.go:356] "Generic (PLEG): container finished" podID="eaa6149f-3c75-4666-8607-5d2686a65b6d" containerID="2386d332bfb4479225eef9bbe398aea56a0926ee89a1875c8ab7148cb8c8c474" exitCode=0
Oct 30 13:16:37.130737 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:37.130558    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"
Oct 30 13:16:37.130737 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:37.130579    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l" event={"ID":"eaa6149f-3c75-4666-8607-5d2686a65b6d","Type":"ContainerDied","Data":"2386d332bfb4479225eef9bbe398aea56a0926ee89a1875c8ab7148cb8c8c474"}
Oct 30 13:16:37.130737 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:37.130643    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l" event={"ID":"eaa6149f-3c75-4666-8607-5d2686a65b6d","Type":"ContainerDied","Data":"c6fa87f045b8e5c52dc31b4d7f52982aa82051a6bf30cfc7c1c7d343aa5f7900"}
Oct 30 13:16:37.130737 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:37.130667    4544 scope.go:122] "RemoveContainer" containerID="2386d332bfb4479225eef9bbe398aea56a0926ee89a1875c8ab7148cb8c8c474"
Oct 30 13:16:37.142466 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:37.142439    4544 scope.go:122] "RemoveContainer" containerID="59a7cd34b4ea8773e5e0aa9ab6cdf2c6113c07eb072a0a4570382665d7280f58"
Oct 30 13:16:37.154696 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:37.154672    4544 scope.go:122] "RemoveContainer" containerID="2386d332bfb4479225eef9bbe398aea56a0926ee89a1875c8ab7148cb8c8c474"
Oct 30 13:16:37.155051 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:16:37.155028    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2386d332bfb4479225eef9bbe398aea56a0926ee89a1875c8ab7148cb8c8c474\": container with ID starting with 2386d332bfb4479225eef9bbe398aea56a0926ee89a1875c8ab7148cb8c8c474 not found: ID does not exist" containerID="2386d332bfb4479225eef9bbe398aea56a0926ee89a1875c8ab7148cb8c8c474"
Oct 30 13:16:37.155124 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:37.155062    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2386d332bfb4479225eef9bbe398aea56a0926ee89a1875c8ab7148cb8c8c474"} err="failed to get container status \"2386d332bfb4479225eef9bbe398aea56a0926ee89a1875c8ab7148cb8c8c474\": rpc error: code = NotFound desc = could not find container \"2386d332bfb4479225eef9bbe398aea56a0926ee89a1875c8ab7148cb8c8c474\": container with ID starting with 2386d332bfb4479225eef9bbe398aea56a0926ee89a1875c8ab7148cb8c8c474 not found: ID does not exist"
Oct 30 13:16:37.155124 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:37.155086    4544 scope.go:122] "RemoveContainer" containerID="59a7cd34b4ea8773e5e0aa9ab6cdf2c6113c07eb072a0a4570382665d7280f58"
Oct 30 13:16:37.155437 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:16:37.155414    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59a7cd34b4ea8773e5e0aa9ab6cdf2c6113c07eb072a0a4570382665d7280f58\": container with ID starting with 59a7cd34b4ea8773e5e0aa9ab6cdf2c6113c07eb072a0a4570382665d7280f58 not found: ID does not exist" containerID="59a7cd34b4ea8773e5e0aa9ab6cdf2c6113c07eb072a0a4570382665d7280f58"
Oct 30 13:16:37.155498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:37.155446    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59a7cd34b4ea8773e5e0aa9ab6cdf2c6113c07eb072a0a4570382665d7280f58"} err="failed to get container status \"59a7cd34b4ea8773e5e0aa9ab6cdf2c6113c07eb072a0a4570382665d7280f58\": rpc error: code = NotFound desc = could not find container \"59a7cd34b4ea8773e5e0aa9ab6cdf2c6113c07eb072a0a4570382665d7280f58\": container with ID starting with 59a7cd34b4ea8773e5e0aa9ab6cdf2c6113c07eb072a0a4570382665d7280f58 not found: ID does not exist"
Oct 30 13:16:37.198796 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:37.198758    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"]
Oct 30 13:16:37.226395 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:37.226354    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l"]
Oct 30 13:16:39.058722 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:39.058667    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh"]
Oct 30 13:16:39.060084 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:39.060057    4544 cpu_manager.go:414] "containerMap: removing container" podUID="eaa6149f-3c75-4666-8607-5d2686a65b6d" containerName="istio-proxy"
Oct 30 13:16:39.060167 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:39.060086    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaa6149f-3c75-4666-8607-5d2686a65b6d" containerName="istio-proxy"
Oct 30 13:16:39.060167 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:39.060118    4544 cpu_manager.go:414] "containerMap: removing container" podUID="eaa6149f-3c75-4666-8607-5d2686a65b6d" containerName="istio-proxy"
Oct 30 13:16:39.060167 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:39.060126    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaa6149f-3c75-4666-8607-5d2686a65b6d" containerName="istio-proxy"
Oct 30 13:16:39.060385 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:39.060368    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaa6149f-3c75-4666-8607-5d2686a65b6d" containerName="istio-proxy"
Oct 30 13:16:39.060421 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:39.060393    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaa6149f-3c75-4666-8607-5d2686a65b6d" containerName="istio-proxy"
Oct 30 13:16:39.066587 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:39.066562    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh"
Oct 30 13:16:39.088909 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:39.088873    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaa6149f-3c75-4666-8607-5d2686a65b6d" path="/var/lib/kubelet/pods/eaa6149f-3c75-4666-8607-5d2686a65b6d/volumes"
Oct 30 13:16:39.090908 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:39.090888    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-80fbd9-serving-cert\""
Oct 30 13:16:39.106003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:39.105966    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55d83701-6415-49ca-877d-d66fc282b2eb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh\" (UID: \"55d83701-6415-49ca-877d-d66fc282b2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh"
Oct 30 13:16:39.106195 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:39.106086    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vl8p\" (UniqueName: \"kubernetes.io/projected/55d83701-6415-49ca-877d-d66fc282b2eb-kube-api-access-5vl8p\") pod \"maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh\" (UID: \"55d83701-6415-49ca-877d-d66fc282b2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh"
Oct 30 13:16:39.106440 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:39.106268    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55d83701-6415-49ca-877d-d66fc282b2eb-maas-api-tls\") pod \"maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh\" (UID: \"55d83701-6415-49ca-877d-d66fc282b2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh"
Oct 30 13:16:39.106440 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:39.106362    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55d83701-6415-49ca-877d-d66fc282b2eb-metrics-certs\") pod \"maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh\" (UID: \"55d83701-6415-49ca-877d-d66fc282b2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh"
Oct 30 13:16:39.112357 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:39.112320    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6"
Oct 30 13:16:39.127244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:39.117427    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh"]
Oct 30 13:16:39.208355 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:39.208249    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vl8p\" (UniqueName: \"kubernetes.io/projected/55d83701-6415-49ca-877d-d66fc282b2eb-kube-api-access-5vl8p\") pod \"maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh\" (UID: \"55d83701-6415-49ca-877d-d66fc282b2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh"
Oct 30 13:16:39.208815 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:39.208783    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55d83701-6415-49ca-877d-d66fc282b2eb-maas-api-tls\") pod \"maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh\" (UID: \"55d83701-6415-49ca-877d-d66fc282b2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh"
Oct 30 13:16:39.208942 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:39.208907    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55d83701-6415-49ca-877d-d66fc282b2eb-metrics-certs\") pod \"maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh\" (UID: \"55d83701-6415-49ca-877d-d66fc282b2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh"
Oct 30 13:16:39.209182 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:39.209159    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55d83701-6415-49ca-877d-d66fc282b2eb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh\" (UID: \"55d83701-6415-49ca-877d-d66fc282b2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh"
Oct 30 13:16:39.210057 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:39.210028    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55d83701-6415-49ca-877d-d66fc282b2eb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh\" (UID: \"55d83701-6415-49ca-877d-d66fc282b2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh"
Oct 30 13:16:39.211828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:39.211799    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55d83701-6415-49ca-877d-d66fc282b2eb-metrics-certs\") pod \"maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh\" (UID: \"55d83701-6415-49ca-877d-d66fc282b2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh"
Oct 30 13:16:39.212744 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:39.212714    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55d83701-6415-49ca-877d-d66fc282b2eb-maas-api-tls\") pod \"maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh\" (UID: \"55d83701-6415-49ca-877d-d66fc282b2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh"
Oct 30 13:16:39.244539 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:39.244501    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vl8p\" (UniqueName: \"kubernetes.io/projected/55d83701-6415-49ca-877d-d66fc282b2eb-kube-api-access-5vl8p\") pod \"maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh\" (UID: \"55d83701-6415-49ca-877d-d66fc282b2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh"
Oct 30 13:16:39.264273 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:39.263057    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:39.384250 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:39.383555    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh"
Oct 30 13:16:39.460841 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:39.460777    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:39.729899 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:39.729865    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh"]
Oct 30 13:16:39.730736 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:16:39.730712    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55d83701_6415_49ca_877d_d66fc282b2eb.slice/crio-457a605c1f03ee686dc76b263d68cbb0c1c2ad75c06857287909cb4edb4133f4 WatchSource:0}: Error finding container 457a605c1f03ee686dc76b263d68cbb0c1c2ad75c06857287909cb4edb4133f4: Status 404 returned error can't find the container with id 457a605c1f03ee686dc76b263d68cbb0c1c2ad75c06857287909cb4edb4133f4
Oct 30 13:16:40.177929 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:40.177890    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh" event={"ID":"55d83701-6415-49ca-877d-d66fc282b2eb","Type":"ContainerStarted","Data":"457a605c1f03ee686dc76b263d68cbb0c1c2ad75c06857287909cb4edb4133f4"}
Oct 30 13:16:40.179926 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:40.179899    4544 generic.go:356] "Generic (PLEG): container finished" podID="da3e4fe2-4f62-4385-9fc3-102dbda6c781" containerID="bc2ef4441794765bc7ae114875739124ce0b10ba8092dee461d869ff0616b0fa" exitCode=0
Oct 30 13:16:40.180089 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:40.179972    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648" event={"ID":"da3e4fe2-4f62-4385-9fc3-102dbda6c781","Type":"ContainerDied","Data":"bc2ef4441794765bc7ae114875739124ce0b10ba8092dee461d869ff0616b0fa"}
Oct 30 13:16:40.180565 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:40.180548    4544 scope.go:122] "RemoveContainer" containerID="bc2ef4441794765bc7ae114875739124ce0b10ba8092dee461d869ff0616b0fa"
Oct 30 13:16:40.182677 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:40.182646    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:16:40.182732 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:40.182720    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:16:40.182770 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:40.182754    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:16:41.191101 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:41.191050    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh" event={"ID":"55d83701-6415-49ca-877d-d66fc282b2eb","Type":"ContainerStarted","Data":"4d23d901d6a8d9c6051cd44d48671c07821bf27c398f906ac0738cc47cf1920b"}
Oct 30 13:16:41.191616 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:41.191130    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh"
Oct 30 13:16:41.193486 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:41.193448    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648" event={"ID":"da3e4fe2-4f62-4385-9fc3-102dbda6c781","Type":"ContainerStarted","Data":"c67196c086b43443840eac544778cbbac903096f6637eaffa683a814a2fb3d98"}
Oct 30 13:16:41.235046 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:41.234986    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh" podStartSLOduration=1.928300069 podStartE2EDuration="2.234973907s" totalImagesPullingTime="306.673838ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:41.233927125 +0000 UTC m=+1896.281061082" watchObservedRunningTime="2026-10-30 13:16:41.234973907 +0000 UTC m=+1896.282107860"
Oct 30 13:16:41.298956 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:41.298925    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:41.909720 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:41.909675    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:16:41.909968 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:41.909940    4544 patch_prober.go:29] interesting pod/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Oct 30 13:16:41.910039 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:41.910013    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648" podUID="da3e4fe2-4f62-4385-9fc3-102dbda6c781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Oct 30 13:16:41.965372 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:41.965331    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:42.113246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:42.112787    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:42.363342 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:42.363303    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:42.702304 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:42.701894    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g"]
Oct 30 13:16:42.803756 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:42.803721    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:43.012649 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:43.012546    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:43.046329 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:43.044236    4544 patch_prober.go:29] interesting pod/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Oct 30 13:16:43.046329 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:43.044299    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648" podUID="da3e4fe2-4f62-4385-9fc3-102dbda6c781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Oct 30 13:16:43.842505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:43.842453    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:43.915770 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:43.915733    4544 patch_prober.go:29] interesting pod/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Oct 30 13:16:43.916091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:43.916061    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648" podUID="da3e4fe2-4f62-4385-9fc3-102dbda6c781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Oct 30 13:16:44.910798 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:44.910757    4544 patch_prober.go:29] interesting pod/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Oct 30 13:16:44.911279 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:44.910815    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648" podUID="da3e4fe2-4f62-4385-9fc3-102dbda6c781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Oct 30 13:16:45.909831 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:45.909792    4544 patch_prober.go:29] interesting pod/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Oct 30 13:16:45.910027 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:45.909848    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648" podUID="da3e4fe2-4f62-4385-9fc3-102dbda6c781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Oct 30 13:16:46.012404 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:46.012356    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:46.909752 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:46.909702    4544 patch_prober.go:29] interesting pod/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Oct 30 13:16:46.909983 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:46.909778    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648" podUID="da3e4fe2-4f62-4385-9fc3-102dbda6c781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Oct 30 13:16:47.055403 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:47.055372    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:16:47.207999 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:47.207924    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh"
Oct 30 13:16:47.702508 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:47.702456    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g" podUID="aad4d3ca-4f7d-418f-b703-cb9c10e101f9" containerName="payload-pre-processing" containerID="cri-o://fa5d5e079221541266f2904a9a1038a8e53af569fedf3c71d51e4ba94f9e889c" gracePeriod=25
Oct 30 13:16:47.917135 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:47.917105    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:16:47.918972 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:47.918949    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:16:47.939652 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:47.938494    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:48.103832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:48.103804    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g"
Oct 30 13:16:48.271850 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:48.271817    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aad4d3ca-4f7d-418f-b703-cb9c10e101f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad4d3ca-4f7d-418f-b703-cb9c10e101f9-plugins-config-volume\") pod \"aad4d3ca-4f7d-418f-b703-cb9c10e101f9\" (UID: \"aad4d3ca-4f7d-418f-b703-cb9c10e101f9\") "
Oct 30 13:16:48.272061 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:48.271968    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aad4d3ca-4f7d-418f-b703-cb9c10e101f9-kube-api-access-4s2v4\" (UniqueName: \"kubernetes.io/projected/aad4d3ca-4f7d-418f-b703-cb9c10e101f9-kube-api-access-4s2v4\") pod \"aad4d3ca-4f7d-418f-b703-cb9c10e101f9\" (UID: \"aad4d3ca-4f7d-418f-b703-cb9c10e101f9\") "
Oct 30 13:16:48.272247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:48.272205    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aad4d3ca-4f7d-418f-b703-cb9c10e101f9-plugins-config-volume" pod "aad4d3ca-4f7d-418f-b703-cb9c10e101f9" (UID: "aad4d3ca-4f7d-418f-b703-cb9c10e101f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:16:48.272489 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:48.272470    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad4d3ca-4f7d-418f-b703-cb9c10e101f9-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:16:48.288365 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:48.288314    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad4d3ca-4f7d-418f-b703-cb9c10e101f9-kube-api-access-4s2v4" pod "aad4d3ca-4f7d-418f-b703-cb9c10e101f9" (UID: "aad4d3ca-4f7d-418f-b703-cb9c10e101f9"). InnerVolumeSpecName "kube-api-access-4s2v4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:16:48.296379 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:48.296336    4544 generic.go:356] "Generic (PLEG): container finished" podID="aad4d3ca-4f7d-418f-b703-cb9c10e101f9" containerID="fa5d5e079221541266f2904a9a1038a8e53af569fedf3c71d51e4ba94f9e889c" exitCode=0
Oct 30 13:16:48.298009 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:48.297981    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g"
Oct 30 13:16:48.303968 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:48.303917    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g" event={"ID":"aad4d3ca-4f7d-418f-b703-cb9c10e101f9","Type":"ContainerDied","Data":"fa5d5e079221541266f2904a9a1038a8e53af569fedf3c71d51e4ba94f9e889c"}
Oct 30 13:16:48.304167 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:48.304087    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g" event={"ID":"aad4d3ca-4f7d-418f-b703-cb9c10e101f9","Type":"ContainerDied","Data":"0de77be0c4e78e4d50bae7ed47f5ffd5d7bb1743fa43c89625820a4bbce6e68e"}
Oct 30 13:16:48.304167 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:48.304146    4544 scope.go:122] "RemoveContainer" containerID="fa5d5e079221541266f2904a9a1038a8e53af569fedf3c71d51e4ba94f9e889c"
Oct 30 13:16:48.316608 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:48.316578    4544 scope.go:122] "RemoveContainer" containerID="fa5d5e079221541266f2904a9a1038a8e53af569fedf3c71d51e4ba94f9e889c"
Oct 30 13:16:48.316964 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:16:48.316932    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa5d5e079221541266f2904a9a1038a8e53af569fedf3c71d51e4ba94f9e889c\": container with ID starting with fa5d5e079221541266f2904a9a1038a8e53af569fedf3c71d51e4ba94f9e889c not found: ID does not exist" containerID="fa5d5e079221541266f2904a9a1038a8e53af569fedf3c71d51e4ba94f9e889c"
Oct 30 13:16:48.317142 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:48.316972    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa5d5e079221541266f2904a9a1038a8e53af569fedf3c71d51e4ba94f9e889c"} err="failed to get container status \"fa5d5e079221541266f2904a9a1038a8e53af569fedf3c71d51e4ba94f9e889c\": rpc error: code = NotFound desc = could not find container \"fa5d5e079221541266f2904a9a1038a8e53af569fedf3c71d51e4ba94f9e889c\": container with ID starting with fa5d5e079221541266f2904a9a1038a8e53af569fedf3c71d51e4ba94f9e889c not found: ID does not exist"
Oct 30 13:16:48.369099 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:48.369052    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g"]
Oct 30 13:16:48.373404 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:48.373369    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g"]
Oct 30 13:16:48.373758 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:48.373723    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s2v4\" (UniqueName: \"kubernetes.io/projected/aad4d3ca-4f7d-418f-b703-cb9c10e101f9-kube-api-access-4s2v4\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:16:49.014039 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:49.013929    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:49.089401 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:49.089365    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aad4d3ca-4f7d-418f-b703-cb9c10e101f9" path="/var/lib/kubelet/pods/aad4d3ca-4f7d-418f-b703-cb9c10e101f9/volumes"
Oct 30 13:16:49.164422 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:49.164388    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:49.804430 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:49.804383    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:51.725244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:51.713029    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:52.419394 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:52.419360    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:53.232692 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:53.232651    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps"]
Oct 30 13:16:53.350483 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:53.350447    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:53.468243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:53.464597    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:54.506484 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:54.506448    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:55.290710 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:55.290670    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:56.550154 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:56.550121    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:56.686138 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:56.686103    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:57.056805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:57.056775    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:58.207545 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:58.207486    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:58.237347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:58.237298    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps" podUID="b885ada7-2fc8-4596-900d-6da86c038051" containerName="payload-pre-processing" containerID="cri-o://34dd37f9ac4aad6edc1a3f9c6a7967711c01ebfadcd87b9f67fde5764289e6eb" gracePeriod=25
Oct 30 13:16:58.401905 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:58.401846    4544 generic.go:356] "Generic (PLEG): container finished" podID="b885ada7-2fc8-4596-900d-6da86c038051" containerID="34dd37f9ac4aad6edc1a3f9c6a7967711c01ebfadcd87b9f67fde5764289e6eb" exitCode=0
Oct 30 13:16:58.402238 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:58.402162    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps" event={"ID":"b885ada7-2fc8-4596-900d-6da86c038051","Type":"ContainerDied","Data":"34dd37f9ac4aad6edc1a3f9c6a7967711c01ebfadcd87b9f67fde5764289e6eb"}
Oct 30 13:16:58.570266 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:58.570189    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps"
Oct 30 13:16:58.602079 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:58.602044    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:58.621746 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:58.621707    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b885ada7-2fc8-4596-900d-6da86c038051-kube-api-access-xxwxt\" (UniqueName: \"kubernetes.io/projected/b885ada7-2fc8-4596-900d-6da86c038051-kube-api-access-xxwxt\") pod \"b885ada7-2fc8-4596-900d-6da86c038051\" (UID: \"b885ada7-2fc8-4596-900d-6da86c038051\") "
Oct 30 13:16:58.621995 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:58.621978    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b885ada7-2fc8-4596-900d-6da86c038051-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b885ada7-2fc8-4596-900d-6da86c038051-plugins-config-volume\") pod \"b885ada7-2fc8-4596-900d-6da86c038051\" (UID: \"b885ada7-2fc8-4596-900d-6da86c038051\") "
Oct 30 13:16:58.622384 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:58.622362    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b885ada7-2fc8-4596-900d-6da86c038051-plugins-config-volume" pod "b885ada7-2fc8-4596-900d-6da86c038051" (UID: "b885ada7-2fc8-4596-900d-6da86c038051"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:16:58.622512 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:58.622492    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b885ada7-2fc8-4596-900d-6da86c038051-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:16:58.623846 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:58.623819    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b885ada7-2fc8-4596-900d-6da86c038051-kube-api-access-xxwxt" pod "b885ada7-2fc8-4596-900d-6da86c038051" (UID: "b885ada7-2fc8-4596-900d-6da86c038051"). InnerVolumeSpecName "kube-api-access-xxwxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:16:58.724797 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:58.724706    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxwxt\" (UniqueName: \"kubernetes.io/projected/b885ada7-2fc8-4596-900d-6da86c038051-kube-api-access-xxwxt\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:16:58.745902 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:58.745861    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:59.222771 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:16:59.220890    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:16:59.365531 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:59.365491    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:59.410937 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:59.410911    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps"
Oct 30 13:16:59.411100 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:59.410954    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps" event={"ID":"b885ada7-2fc8-4596-900d-6da86c038051","Type":"ContainerDied","Data":"ba1edde91787782b7a97e4a7020084a9ce163f90173c930ad229c7f44900be7a"}
Oct 30 13:16:59.411100 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:59.411012    4544 scope.go:122] "RemoveContainer" containerID="34dd37f9ac4aad6edc1a3f9c6a7967711c01ebfadcd87b9f67fde5764289e6eb"
Oct 30 13:16:59.472985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:59.472913    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps"]
Oct 30 13:16:59.484699 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:59.484668    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps"]
Oct 30 13:16:59.507307 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:59.507279    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-716b4c-qqfdl"]
Oct 30 13:16:59.517191 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:59.517160    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-716b4c-qqfdl"]
Oct 30 13:16:59.530545 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:59.530521    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:16:59.663973 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:16:59.663939    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:00.023264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:00.023214    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:01.091389 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:01.091352    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75ec2a60-7de0-4cff-9961-989ee7c2961d" path="/var/lib/kubelet/pods/75ec2a60-7de0-4cff-9961-989ee7c2961d/volumes"
Oct 30 13:17:01.091940 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:01.091922    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b885ada7-2fc8-4596-900d-6da86c038051" path="/var/lib/kubelet/pods/b885ada7-2fc8-4596-900d-6da86c038051/volumes"
Oct 30 13:17:01.972250 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:01.958111    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:02.348286 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:02.348249    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:02.522952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:02.522916    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:04.201455 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:04.201417    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:04.330509 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:04.330452    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:06.539650 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:06.539606    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:06.663187 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:06.663151    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:06.799754 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:06.799640    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:06.968127 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:06.968095    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:07.722749 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:07.722712    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:08.224934 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:08.224895    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:08.347984 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:08.347949    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:08.630478 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:08.630448    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:10.576087 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:10.576051    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:11.675364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:11.675329    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:11.804325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:11.804289    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:12.084129 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:12.084099    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:12.424377 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:12.424274    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:12.585068 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:12.585033    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:13.005675 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:13.005641    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:13.128420 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:13.128389    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:14.291467 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:14.291431    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:14.618912 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:14.618880    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:15.395432 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:15.395394    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:17.375577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:17.375463    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:19.259948 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:19.259909    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:21.311498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:21.311464    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:23.157055 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:23.157020    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:23.785612 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:23.785565    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:25.085624 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:25.085573    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:25.838983 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:25.838923    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:27.386999 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:27.386961    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:27.500837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:27.500796    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:29.010117 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:17:29.010072    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:17:29.761542 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:29.761459    4544 generic.go:356] "Generic (PLEG): container finished" podID="d685dcc9-430c-4f84-aa00-7506306061d1" containerID="b61ba5361033a17783ae4e7598056be0f5835e19e3ede4aae9c71d67d1bac2a0" exitCode=0
Oct 30 13:17:29.761542 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:29.761528    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f" event={"ID":"d685dcc9-430c-4f84-aa00-7506306061d1","Type":"ContainerDied","Data":"b61ba5361033a17783ae4e7598056be0f5835e19e3ede4aae9c71d67d1bac2a0"}
Oct 30 13:17:29.762021 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:29.762003    4544 scope.go:122] "RemoveContainer" containerID="b61ba5361033a17783ae4e7598056be0f5835e19e3ede4aae9c71d67d1bac2a0"
Oct 30 13:17:29.764066 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:29.764038    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:17:29.764152 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:29.764114    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:17:29.764195 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:29.764149    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:17:30.003244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:30.003127    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:30.771416 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:30.771378    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f" event={"ID":"d685dcc9-430c-4f84-aa00-7506306061d1","Type":"ContainerStarted","Data":"a684ee71634864b805b6ef5845f903152c480e87f7342fa9edd32012cffe3ac3"}
Oct 30 13:17:31.041423 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:31.041330    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:17:31.041823 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:31.041778    4544 patch_prober.go:29] interesting pod/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 13:17:31.042018 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:31.041988    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f" podUID="d685dcc9-430c-4f84-aa00-7506306061d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 13:17:32.035118 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:32.035077    4544 patch_prober.go:29] interesting pod/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 13:17:32.035875 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:32.035838    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f" podUID="d685dcc9-430c-4f84-aa00-7506306061d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 13:17:32.144247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:32.140127    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:33.036817 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:33.036779    4544 patch_prober.go:29] interesting pod/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 13:17:33.037441 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:33.037403    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f" podUID="d685dcc9-430c-4f84-aa00-7506306061d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 13:17:33.766412 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:33.766372    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:17:34.037725 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:34.037616    4544 patch_prober.go:29] interesting pod/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 13:17:34.038416 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:34.038378    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f" podUID="d685dcc9-430c-4f84-aa00-7506306061d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 13:17:34.234245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:34.234164    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:35.038260 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:35.037685    4544 patch_prober.go:29] interesting pod/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 13:17:35.038260 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:35.037758    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f" podUID="d685dcc9-430c-4f84-aa00-7506306061d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 13:17:36.038670 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:36.038630    4544 patch_prober.go:29] interesting pod/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 13:17:36.039407 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:36.039355    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f" podUID="d685dcc9-430c-4f84-aa00-7506306061d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 13:17:36.311803 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:36.311701    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:37.131003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:37.130928    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:17:37.132149 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:37.132129    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:17:37.273576 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:37.273492    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:37.729388 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:37.729356    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:37.872575 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:37.872540    4544 generic.go:356] "Generic (PLEG): container finished" podID="1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff" containerID="0e84581c5e263c206940181471dcfde276bbb741244437eeab53f8d17892d386" exitCode=0
Oct 30 13:17:37.872789 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:37.872612    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj" event={"ID":"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff","Type":"ContainerDied","Data":"0e84581c5e263c206940181471dcfde276bbb741244437eeab53f8d17892d386"}
Oct 30 13:17:37.873167 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:37.873149    4544 scope.go:122] "RemoveContainer" containerID="0e84581c5e263c206940181471dcfde276bbb741244437eeab53f8d17892d386"
Oct 30 13:17:37.875029 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:37.874993    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:17:37.875136 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:37.875086    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:17:37.875203 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:37.875140    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:17:38.231704 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.231593    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"]
Oct 30 13:17:38.232492 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.232469    4544 cpu_manager.go:414] "containerMap: removing container" podUID="b885ada7-2fc8-4596-900d-6da86c038051" containerName="payload-pre-processing"
Oct 30 13:17:38.232492 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.232492    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b885ada7-2fc8-4596-900d-6da86c038051" containerName="payload-pre-processing"
Oct 30 13:17:38.232679 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.232541    4544 cpu_manager.go:414] "containerMap: removing container" podUID="aad4d3ca-4f7d-418f-b703-cb9c10e101f9" containerName="payload-pre-processing"
Oct 30 13:17:38.232679 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.232547    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aad4d3ca-4f7d-418f-b703-cb9c10e101f9" containerName="payload-pre-processing"
Oct 30 13:17:38.232790 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.232716    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="aad4d3ca-4f7d-418f-b703-cb9c10e101f9" containerName="payload-pre-processing"
Oct 30 13:17:38.232790 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.232741    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="b885ada7-2fc8-4596-900d-6da86c038051" containerName="payload-pre-processing"
Oct 30 13:17:38.237331 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.237305    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:17:38.262080 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.262045    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-bec983-openshift-default-dockercfg-4ktqh\""
Oct 30 13:17:38.271863 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.271823    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"]
Oct 30 13:17:38.430298 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.422675    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istio-token\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:17:38.430298 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.422736    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-workload-certs\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:17:38.430298 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.422839    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istiod-ca-cert\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:17:38.430298 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.422928    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6cgx\" (UniqueName: \"kubernetes.io/projected/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-kube-api-access-w6cgx\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:17:38.430298 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.422996    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istio-data\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:17:38.430298 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.423040    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istio-podinfo\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:17:38.430298 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.423083    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-workload-socket\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:17:38.430298 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.423170    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-credential-socket\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:17:38.430298 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.423192    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istio-envoy\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:17:38.455260 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.436596    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:38.526309 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.525803    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istio-token\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:17:38.526309 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.525870    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-workload-certs\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:17:38.526309 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.525969    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istiod-ca-cert\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:17:38.526309 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.526052    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6cgx\" (UniqueName: \"kubernetes.io/projected/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-kube-api-access-w6cgx\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:17:38.526309 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.526119    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istio-data\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:17:38.526309 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.526161    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istio-podinfo\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:17:38.526309 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.526204    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-workload-socket\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:17:38.527184 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.526327    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-credential-socket\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:17:38.552393 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.542321    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istio-envoy\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:17:38.552393 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.544537    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-workload-certs\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:17:38.552393 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.545285    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istiod-ca-cert\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:17:38.552393 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.545530    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istio-data\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:17:38.552393 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.545749    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-workload-socket\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:17:38.552393 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.545978    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-credential-socket\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:17:38.592389 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.569424    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:38.592389 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.578116    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6cgx\" (UniqueName: \"kubernetes.io/projected/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-kube-api-access-w6cgx\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:17:38.593159 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.593123    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istio-envoy\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:17:38.611665 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.609987    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istio-podinfo\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:17:38.629136 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.628834    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istio-token\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:17:38.846118 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.846062    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:38.872086 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.872047    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:17:38.882893 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.882850    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj" event={"ID":"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff","Type":"ContainerStarted","Data":"fbcba18ac868587aa34f549c3c9acb5f52895a6da4de09944ec74e4398b89bbd"}
Oct 30 13:17:38.891599 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.891574    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:17:38.892608 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.892566    4544 patch_prober.go:29] interesting pod/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 13:17:38.892711 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:38.892617    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj" podUID="1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 13:17:39.239835 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:17:39.239778    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e32d130_6ba4_4391_bf2a_7bd6548c1c88.slice/crio-0f91ca6113fa978ae028d11e067b8668840718150ac88f8da3e3aa59edc40bf7 WatchSource:0}: Error finding container 0f91ca6113fa978ae028d11e067b8668840718150ac88f8da3e3aa59edc40bf7: Status 404 returned error can't find the container with id 0f91ca6113fa978ae028d11e067b8668840718150ac88f8da3e3aa59edc40bf7
Oct 30 13:17:39.244264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:39.244200    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"]
Oct 30 13:17:39.244891 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:39.244854    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:17:39.244990 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:39.244944    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:17:39.245059 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:39.244993    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:17:39.894678 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:39.894627    4544 patch_prober.go:29] interesting pod/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 13:17:39.894913 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:39.894767    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj" podUID="1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 13:17:39.898508 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:39.898466    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m" event={"ID":"2e32d130-6ba4-4391-bf2a-7bd6548c1c88","Type":"ContainerStarted","Data":"a578a827da5dbae043fd6b228775c44f833308d1c2bb162eb2e7f39f47d1a53f"}
Oct 30 13:17:39.898684 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:39.898567    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m" event={"ID":"2e32d130-6ba4-4391-bf2a-7bd6548c1c88","Type":"ContainerStarted","Data":"0f91ca6113fa978ae028d11e067b8668840718150ac88f8da3e3aa59edc40bf7"}
Oct 30 13:17:39.972172 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:39.972110    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m" podStartSLOduration=1.972096021 podStartE2EDuration="1.972096021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:39.970287389 +0000 UTC m=+1955.017421343" watchObservedRunningTime="2026-10-30 13:17:39.972096021 +0000 UTC m=+1955.019229972"
Oct 30 13:17:40.502764 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:40.502723    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:40.872913 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:40.872865    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:17:40.879314 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:40.879283    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:17:40.891125 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:40.891084    4544 patch_prober.go:29] interesting pod/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 13:17:40.891342 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:40.891159    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj" podUID="1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 13:17:40.905588 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:40.905554    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:17:40.906652 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:40.906629    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:17:41.892068 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:41.892033    4544 patch_prober.go:29] interesting pod/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 13:17:41.892512 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:41.892088    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj" podUID="1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 13:17:42.473693 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:42.473656    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:42.619358 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:42.619318    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:42.641248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:42.639398    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:17:42.894709 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:42.894676    4544 patch_prober.go:29] interesting pod/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 13:17:42.895390 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:42.895331    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj" podUID="1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 13:17:43.891890 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:43.891490    4544 patch_prober.go:29] interesting pod/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 13:17:43.891890 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:43.891559    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj" podUID="1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 13:17:44.240184 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:44.239626    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:44.400985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:44.400945    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:44.891826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:44.891784    4544 patch_prober.go:29] interesting pod/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 13:17:44.892066 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:44.891846    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj" podUID="1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 13:17:45.465657 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:45.465627    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:45.896748 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:45.896716    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:17:45.897964 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:45.897943    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:17:47.557518 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:47.557478    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:49.375328 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:49.375286    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:49.501453 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:49.500087    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:51.522442 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:51.522398    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:53.958064 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:53.958024    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:55.419079 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:55.419041    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:55.723258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:55.723120    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:56.584588 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:56.584552    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:57.882296 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:57.882258    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:17:59.009556 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:17:59.009519    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:17:59.762955 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:17:59.762908    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:01.596901 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:01.596858    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:01.711968 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:01.711932    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:01.838420 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:01.838374    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:03.600551 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:03.600503    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:04.307008 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:04.306974    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:05.424144 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:05.424113    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:07.154955 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:07.154919    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:08.831092 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:08.831059    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:10.805765 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:10.805724    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:12.244551 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:12.244518    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx"]
Oct 30 13:18:12.244975 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:12.244715    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx" podUID="a1a9ed47-9838-43d5-bf54-7133d03c25fd" containerName="maas-api" containerID="cri-o://b17317c2ca8a9df6dffe220eedbcd2e2b088e4f662197ea57e7742e183a63f45" gracePeriod=30
Oct 30 13:18:12.333333 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:12.333298    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:12.821979 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:12.821594    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx"
Oct 30 13:18:13.051579 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:13.051494    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1a9ed47-9838-43d5-bf54-7133d03c25fd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a9ed47-9838-43d5-bf54-7133d03c25fd-metrics-certs\") pod \"a1a9ed47-9838-43d5-bf54-7133d03c25fd\" (UID: \"a1a9ed47-9838-43d5-bf54-7133d03c25fd\") "
Oct 30 13:18:13.051738 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:13.051587    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1a9ed47-9838-43d5-bf54-7133d03c25fd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a9ed47-9838-43d5-bf54-7133d03c25fd-openshift-service-ca\") pod \"a1a9ed47-9838-43d5-bf54-7133d03c25fd\" (UID: \"a1a9ed47-9838-43d5-bf54-7133d03c25fd\") "
Oct 30 13:18:13.051738 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:13.051644    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1a9ed47-9838-43d5-bf54-7133d03c25fd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a9ed47-9838-43d5-bf54-7133d03c25fd-maas-api-tls\") pod \"a1a9ed47-9838-43d5-bf54-7133d03c25fd\" (UID: \"a1a9ed47-9838-43d5-bf54-7133d03c25fd\") "
Oct 30 13:18:13.051871 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:13.051759    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1a9ed47-9838-43d5-bf54-7133d03c25fd-kube-api-access-l2phj\" (UniqueName: \"kubernetes.io/projected/a1a9ed47-9838-43d5-bf54-7133d03c25fd-kube-api-access-l2phj\") pod \"a1a9ed47-9838-43d5-bf54-7133d03c25fd\" (UID: \"a1a9ed47-9838-43d5-bf54-7133d03c25fd\") "
Oct 30 13:18:13.052011 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:13.051986    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a9ed47-9838-43d5-bf54-7133d03c25fd-openshift-service-ca" pod "a1a9ed47-9838-43d5-bf54-7133d03c25fd" (UID: "a1a9ed47-9838-43d5-bf54-7133d03c25fd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:18:13.052630 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:13.052593    4544 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a9ed47-9838-43d5-bf54-7133d03c25fd-openshift-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:18:13.053967 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:13.053941    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a9ed47-9838-43d5-bf54-7133d03c25fd-kube-api-access-l2phj" pod "a1a9ed47-9838-43d5-bf54-7133d03c25fd" (UID: "a1a9ed47-9838-43d5-bf54-7133d03c25fd"). InnerVolumeSpecName "kube-api-access-l2phj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:13.054425 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:13.054398    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a9ed47-9838-43d5-bf54-7133d03c25fd-maas-api-tls" pod "a1a9ed47-9838-43d5-bf54-7133d03c25fd" (UID: "a1a9ed47-9838-43d5-bf54-7133d03c25fd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:18:13.054425 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:13.054415    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a9ed47-9838-43d5-bf54-7133d03c25fd-metrics-certs" pod "a1a9ed47-9838-43d5-bf54-7133d03c25fd" (UID: "a1a9ed47-9838-43d5-bf54-7133d03c25fd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:18:13.077369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:13.077341    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:13.154184 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:13.154151    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2phj\" (UniqueName: \"kubernetes.io/projected/a1a9ed47-9838-43d5-bf54-7133d03c25fd-kube-api-access-l2phj\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:18:13.154184 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:13.154181    4544 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a9ed47-9838-43d5-bf54-7133d03c25fd-metrics-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:18:13.154184 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:13.154192    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a9ed47-9838-43d5-bf54-7133d03c25fd-maas-api-tls\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:18:13.250136 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:13.246142    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:13.299566 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:13.299533    4544 generic.go:356] "Generic (PLEG): container finished" podID="a1a9ed47-9838-43d5-bf54-7133d03c25fd" containerID="b17317c2ca8a9df6dffe220eedbcd2e2b088e4f662197ea57e7742e183a63f45" exitCode=0
Oct 30 13:18:13.299746 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:13.299595    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx"
Oct 30 13:18:13.299746 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:13.299615    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx" event={"ID":"a1a9ed47-9838-43d5-bf54-7133d03c25fd","Type":"ContainerDied","Data":"b17317c2ca8a9df6dffe220eedbcd2e2b088e4f662197ea57e7742e183a63f45"}
Oct 30 13:18:13.299746 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:13.299667    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx" event={"ID":"a1a9ed47-9838-43d5-bf54-7133d03c25fd","Type":"ContainerDied","Data":"208b12cf3a7fd2334e1b489e69ebc79eba8ee7e01c499bb233d285df1f82d3df"}
Oct 30 13:18:13.299746 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:13.299690    4544 scope.go:122] "RemoveContainer" containerID="b17317c2ca8a9df6dffe220eedbcd2e2b088e4f662197ea57e7742e183a63f45"
Oct 30 13:18:13.311904 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:13.311884    4544 scope.go:122] "RemoveContainer" containerID="b17317c2ca8a9df6dffe220eedbcd2e2b088e4f662197ea57e7742e183a63f45"
Oct 30 13:18:13.312215 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:18:13.312196    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b17317c2ca8a9df6dffe220eedbcd2e2b088e4f662197ea57e7742e183a63f45\": container with ID starting with b17317c2ca8a9df6dffe220eedbcd2e2b088e4f662197ea57e7742e183a63f45 not found: ID does not exist" containerID="b17317c2ca8a9df6dffe220eedbcd2e2b088e4f662197ea57e7742e183a63f45"
Oct 30 13:18:13.312303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:13.312260    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b17317c2ca8a9df6dffe220eedbcd2e2b088e4f662197ea57e7742e183a63f45"} err="failed to get container status \"b17317c2ca8a9df6dffe220eedbcd2e2b088e4f662197ea57e7742e183a63f45\": rpc error: code = NotFound desc = could not find container \"b17317c2ca8a9df6dffe220eedbcd2e2b088e4f662197ea57e7742e183a63f45\": container with ID starting with b17317c2ca8a9df6dffe220eedbcd2e2b088e4f662197ea57e7742e183a63f45 not found: ID does not exist"
Oct 30 13:18:13.405346 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:13.405286    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx"]
Oct 30 13:18:13.410253 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:13.409474    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-14e8a4-7d7f655b78-lh5kx"]
Oct 30 13:18:13.464778 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:13.464740    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:13.785906 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:13.785870    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:15.090360 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:15.090329    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1a9ed47-9838-43d5-bf54-7133d03c25fd" path="/var/lib/kubelet/pods/a1a9ed47-9838-43d5-bf54-7133d03c25fd/volumes"
Oct 30 13:18:15.707247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:15.700174    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:17.798708 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:17.798673    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:19.398490 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:19.398460    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:20.388748 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:20.388717    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:20.508632 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:20.508592    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:20.822904 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:20.822825    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:21.484893 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:21.484859    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:21.661347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:21.661310    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:22.100453 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.100417    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"]
Oct 30 13:18:22.101513 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.101488    4544 cpu_manager.go:414] "containerMap: removing container" podUID="a1a9ed47-9838-43d5-bf54-7133d03c25fd" containerName="maas-api"
Oct 30 13:18:22.101513 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.101510    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a9ed47-9838-43d5-bf54-7133d03c25fd" containerName="maas-api"
Oct 30 13:18:22.101714 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.101695    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1a9ed47-9838-43d5-bf54-7133d03c25fd" containerName="maas-api"
Oct 30 13:18:22.107146 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.107125    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:18:22.128932 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.125522    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-60a9a6-openshift-default-dockercfg-f8sb4\""
Oct 30 13:18:22.145785 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.145754    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"]
Oct 30 13:18:22.191806 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.191753    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:22.244664 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.244634    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istio-podinfo\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:18:22.244818 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.244671    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vccxv\" (UniqueName: \"kubernetes.io/projected/ed9d8ff2-2175-4747-a154-d803bb7b54f8-kube-api-access-vccxv\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:18:22.244818 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.244698    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istio-envoy\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:18:22.244896 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.244872    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-workload-socket\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:18:22.245042 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.245012    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istiod-ca-cert\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:18:22.245200 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.245131    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-workload-certs\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:18:22.245327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.245302    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istio-data\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:18:22.245420 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.245405    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istio-token\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:18:22.245637 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.245597    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-credential-socket\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:18:22.334571 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.334446    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:22.347352 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.347312    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-workload-socket\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:18:22.347649 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.347627    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istiod-ca-cert\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:18:22.347887 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.347867    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-workload-certs\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:18:22.359281 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.358367    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istio-data\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:18:22.359281 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.358462    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istio-token\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:18:22.359281 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.358555    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-credential-socket\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:18:22.359281 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.358603    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istio-podinfo\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:18:22.359281 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.358636    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vccxv\" (UniqueName: \"kubernetes.io/projected/ed9d8ff2-2175-4747-a154-d803bb7b54f8-kube-api-access-vccxv\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:18:22.359281 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.358672    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istio-envoy\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:18:22.359698 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.347870    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-workload-socket\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:18:22.363282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.359753    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istio-data\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:18:22.363282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.348520    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istiod-ca-cert\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:18:22.363282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.360029    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-credential-socket\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:18:22.363282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.348792    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-workload-certs\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:18:22.398976 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.398947    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istio-token\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:18:22.399935 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.399728    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istio-envoy\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:18:22.409925 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.409836    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vccxv\" (UniqueName: \"kubernetes.io/projected/ed9d8ff2-2175-4747-a154-d803bb7b54f8-kube-api-access-vccxv\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:18:22.421867 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.421837    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istio-podinfo\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:18:22.444370 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.444343    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:18:22.901157 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.901125    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"]
Oct 30 13:18:22.902021 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:18:22.901991    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded9d8ff2_2175_4747_a154_d803bb7b54f8.slice/crio-e6cafc150bff70d02143e927402efa952fc7cccbc42096464ec7884cd8709781 WatchSource:0}: Error finding container e6cafc150bff70d02143e927402efa952fc7cccbc42096464ec7884cd8709781: Status 404 returned error can't find the container with id e6cafc150bff70d02143e927402efa952fc7cccbc42096464ec7884cd8709781
Oct 30 13:18:22.904804 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.904768    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:18:22.904861 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.904841    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:18:22.904902 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:22.904872    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:18:23.407167 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:23.407136    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d" event={"ID":"ed9d8ff2-2175-4747-a154-d803bb7b54f8","Type":"ContainerStarted","Data":"f856dbfc8cb97f7624945ddbe1fa45919223e6a1bc2abb87c9888021dfeee483"}
Oct 30 13:18:23.407391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:23.407185    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d" event={"ID":"ed9d8ff2-2175-4747-a154-d803bb7b54f8","Type":"ContainerStarted","Data":"e6cafc150bff70d02143e927402efa952fc7cccbc42096464ec7884cd8709781"}
Oct 30 13:18:23.445212 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:23.445183    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:18:23.451738 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:23.451714    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:18:23.561597 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:23.561544    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d" podStartSLOduration=1.561529849 podStartE2EDuration="1.561529849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:23.482687182 +0000 UTC m=+1998.529821147" watchObservedRunningTime="2026-10-30 13:18:23.561529849 +0000 UTC m=+1998.608663801"
Oct 30 13:18:23.626247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:23.626191    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:24.416123 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:24.416080    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:18:24.417108 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:24.417086    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:18:25.723090 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:25.723057    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:25.989695 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:25.989603    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:26.156933 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:26.156898    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:27.888350 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:27.888314    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:28.025548 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:28.025517    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:29.007565 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:18:29.007528    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:18:29.991758 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:29.991719    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:31.598731 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:31.598696    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:32.219642 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:32.219607    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:32.355928 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:32.355888    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:32.492331 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:32.492163    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:32.606164 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:32.606127    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:32.736710 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:32.736661    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:32.902350 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:32.902312    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:34.808933 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:34.808896    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:36.897941 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:36.897902    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:38.628152 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:38.628115    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:39.859471 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:39.859433    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:40.008024 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:40.007982    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:40.233547 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:40.233455    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:40.846920 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:40.846886    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:41.141847 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:41.141758    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:41.885178 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:41.885143    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:42.105669 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:42.105634    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:43.205284 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:43.205242    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:44.906249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:44.905689    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:45.407758 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:45.407720    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:45.684617 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:45.684530    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:45.811517 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:45.811483    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:46.217439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:46.217406    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r"]
Oct 30 13:18:46.222930 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:46.222902    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r"
Oct 30 13:18:46.240317 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:46.240286    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6af611\"/\"kube-root-ca.crt\""
Oct 30 13:18:46.259287 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:46.255118    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6af611\"/\"openshift-service-ca.crt\""
Oct 30 13:18:46.259287 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:46.255435    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6af611\"/\"test-model-6af611-kserve-self-signed-certs\""
Oct 30 13:18:46.259287 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:46.258012    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6af611\"/\"default-dockercfg-hnpsc\""
Oct 30 13:18:46.261091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:46.261057    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r"]
Oct 30 13:18:46.280633 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:46.280594    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:46.317806 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:46.317771    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzb99\" (UniqueName: \"kubernetes.io/projected/0a673362-df63-4d76-81fa-b35ade9305ef-kube-api-access-tzb99\") pod \"test-model-6af611-kserve-5b6bbd8ddf-8ds2r\" (UID: \"0a673362-df63-4d76-81fa-b35ade9305ef\") " pod="ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r"
Oct 30 13:18:46.317990 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:46.317906    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a673362-df63-4d76-81fa-b35ade9305ef-tls-certs\") pod \"test-model-6af611-kserve-5b6bbd8ddf-8ds2r\" (UID: \"0a673362-df63-4d76-81fa-b35ade9305ef\") " pod="ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r"
Oct 30 13:18:46.318064 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:46.317995    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a673362-df63-4d76-81fa-b35ade9305ef-model-cache\") pod \"test-model-6af611-kserve-5b6bbd8ddf-8ds2r\" (UID: \"0a673362-df63-4d76-81fa-b35ade9305ef\") " pod="ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r"
Oct 30 13:18:46.318205 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:46.318187    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a673362-df63-4d76-81fa-b35ade9305ef-home\") pod \"test-model-6af611-kserve-5b6bbd8ddf-8ds2r\" (UID: \"0a673362-df63-4d76-81fa-b35ade9305ef\") " pod="ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r"
Oct 30 13:18:46.318347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:46.318329    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a673362-df63-4d76-81fa-b35ade9305ef-dshm\") pod \"test-model-6af611-kserve-5b6bbd8ddf-8ds2r\" (UID: \"0a673362-df63-4d76-81fa-b35ade9305ef\") " pod="ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r"
Oct 30 13:18:46.318411 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:46.318396    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a673362-df63-4d76-81fa-b35ade9305ef-tmp-dir\") pod \"test-model-6af611-kserve-5b6bbd8ddf-8ds2r\" (UID: \"0a673362-df63-4d76-81fa-b35ade9305ef\") " pod="ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r"
Oct 30 13:18:46.421968 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:46.421925    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a673362-df63-4d76-81fa-b35ade9305ef-home\") pod \"test-model-6af611-kserve-5b6bbd8ddf-8ds2r\" (UID: \"0a673362-df63-4d76-81fa-b35ade9305ef\") " pod="ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r"
Oct 30 13:18:46.422165 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:46.422034    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a673362-df63-4d76-81fa-b35ade9305ef-dshm\") pod \"test-model-6af611-kserve-5b6bbd8ddf-8ds2r\" (UID: \"0a673362-df63-4d76-81fa-b35ade9305ef\") " pod="ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r"
Oct 30 13:18:46.422165 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:46.422083    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a673362-df63-4d76-81fa-b35ade9305ef-tmp-dir\") pod \"test-model-6af611-kserve-5b6bbd8ddf-8ds2r\" (UID: \"0a673362-df63-4d76-81fa-b35ade9305ef\") " pod="ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r"
Oct 30 13:18:46.422324 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:46.422168    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzb99\" (UniqueName: \"kubernetes.io/projected/0a673362-df63-4d76-81fa-b35ade9305ef-kube-api-access-tzb99\") pod \"test-model-6af611-kserve-5b6bbd8ddf-8ds2r\" (UID: \"0a673362-df63-4d76-81fa-b35ade9305ef\") " pod="ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r"
Oct 30 13:18:46.422324 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:46.422280    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a673362-df63-4d76-81fa-b35ade9305ef-tls-certs\") pod \"test-model-6af611-kserve-5b6bbd8ddf-8ds2r\" (UID: \"0a673362-df63-4d76-81fa-b35ade9305ef\") " pod="ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r"
Oct 30 13:18:46.422437 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:46.422364    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a673362-df63-4d76-81fa-b35ade9305ef-home\") pod \"test-model-6af611-kserve-5b6bbd8ddf-8ds2r\" (UID: \"0a673362-df63-4d76-81fa-b35ade9305ef\") " pod="ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r"
Oct 30 13:18:46.422437 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:46.422400    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a673362-df63-4d76-81fa-b35ade9305ef-model-cache\") pod \"test-model-6af611-kserve-5b6bbd8ddf-8ds2r\" (UID: \"0a673362-df63-4d76-81fa-b35ade9305ef\") " pod="ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r"
Oct 30 13:18:46.422535 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:46.422500    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a673362-df63-4d76-81fa-b35ade9305ef-tmp-dir\") pod \"test-model-6af611-kserve-5b6bbd8ddf-8ds2r\" (UID: \"0a673362-df63-4d76-81fa-b35ade9305ef\") " pod="ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r"
Oct 30 13:18:46.422718 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:46.422699    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a673362-df63-4d76-81fa-b35ade9305ef-model-cache\") pod \"test-model-6af611-kserve-5b6bbd8ddf-8ds2r\" (UID: \"0a673362-df63-4d76-81fa-b35ade9305ef\") " pod="ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r"
Oct 30 13:18:46.425656 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:46.425633    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a673362-df63-4d76-81fa-b35ade9305ef-tls-certs\") pod \"test-model-6af611-kserve-5b6bbd8ddf-8ds2r\" (UID: \"0a673362-df63-4d76-81fa-b35ade9305ef\") " pod="ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r"
Oct 30 13:18:46.426714 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:46.426686    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a673362-df63-4d76-81fa-b35ade9305ef-dshm\") pod \"test-model-6af611-kserve-5b6bbd8ddf-8ds2r\" (UID: \"0a673362-df63-4d76-81fa-b35ade9305ef\") " pod="ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r"
Oct 30 13:18:46.445683 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:46.445655    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzb99\" (UniqueName: \"kubernetes.io/projected/0a673362-df63-4d76-81fa-b35ade9305ef-kube-api-access-tzb99\") pod \"test-model-6af611-kserve-5b6bbd8ddf-8ds2r\" (UID: \"0a673362-df63-4d76-81fa-b35ade9305ef\") " pod="ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r"
Oct 30 13:18:46.540746 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:46.540577    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r"
Oct 30 13:18:47.605199 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:47.605162    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r"]
Oct 30 13:18:47.605846 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:18:47.605822    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a673362_df63_4d76_81fa_b35ade9305ef.slice/crio-e8ed30d39d7c81492c91efb98d8e77b79ce2d61c4e3031c6555b98a3f22b4577 WatchSource:0}: Error finding container e8ed30d39d7c81492c91efb98d8e77b79ce2d61c4e3031c6555b98a3f22b4577: Status 404 returned error can't find the container with id e8ed30d39d7c81492c91efb98d8e77b79ce2d61c4e3031c6555b98a3f22b4577
Oct 30 13:18:47.882837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:47.882736    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r" event={"ID":"0a673362-df63-4d76-81fa-b35ade9305ef","Type":"ContainerStarted","Data":"5ea3acfa3f7ca3b951f59aed5cde0d5658ebfc7c63dc5714fa6ac7ada4a3fa5e"}
Oct 30 13:18:47.882837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:47.882800    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r" event={"ID":"0a673362-df63-4d76-81fa-b35ade9305ef","Type":"ContainerStarted","Data":"e8ed30d39d7c81492c91efb98d8e77b79ce2d61c4e3031c6555b98a3f22b4577"}
Oct 30 13:18:47.928061 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:47.927999    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r" podStartSLOduration=1.927982283 podStartE2EDuration="1.927982283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:47.926788946 +0000 UTC m=+2022.973922901" watchObservedRunningTime="2026-10-30 13:18:47.927982283 +0000 UTC m=+2022.975116242"
Oct 30 13:18:48.255874 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:48.255765    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:48.397510 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:48.397447    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:48.532301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:48.532201    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96"]
Oct 30 13:18:48.538328 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:48.538301    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96"
Oct 30 13:18:48.540842 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:48.540820    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r"
Oct 30 13:18:48.556240 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:48.556195    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2b8461\""
Oct 30 13:18:48.557047 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:48.557009    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2b8461-dockercfg-dpv44\""
Oct 30 13:18:48.559624 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:48.559591    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96"]
Oct 30 13:18:48.566994 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:48.566962    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:48.578836 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:48.578801    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp"]
Oct 30 13:18:48.583850 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:48.583824    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp"
Oct 30 13:18:48.740243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:48.739307    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59m4v\" (UniqueName: \"kubernetes.io/projected/e443ff59-e6ec-444d-817a-1b7baaa5faf7-kube-api-access-59m4v\") pod \"payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96\" (UID: \"e443ff59-e6ec-444d-817a-1b7baaa5faf7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96"
Oct 30 13:18:48.740243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:48.739858    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e443ff59-e6ec-444d-817a-1b7baaa5faf7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96\" (UID: \"e443ff59-e6ec-444d-817a-1b7baaa5faf7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96"
Oct 30 13:18:48.846102 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:48.844480    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e443ff59-e6ec-444d-817a-1b7baaa5faf7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96\" (UID: \"e443ff59-e6ec-444d-817a-1b7baaa5faf7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96"
Oct 30 13:18:48.846102 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:48.844616    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59m4v\" (UniqueName: \"kubernetes.io/projected/e443ff59-e6ec-444d-817a-1b7baaa5faf7-kube-api-access-59m4v\") pod \"payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96\" (UID: \"e443ff59-e6ec-444d-817a-1b7baaa5faf7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96"
Oct 30 13:18:48.846102 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:48.844699    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a9a0392-a7ea-4d31-add2-f4faf60c121b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp\" (UID: \"9a9a0392-a7ea-4d31-add2-f4faf60c121b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp"
Oct 30 13:18:48.846102 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:48.844829    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt44z\" (UniqueName: \"kubernetes.io/projected/9a9a0392-a7ea-4d31-add2-f4faf60c121b-kube-api-access-xt44z\") pod \"payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp\" (UID: \"9a9a0392-a7ea-4d31-add2-f4faf60c121b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp"
Oct 30 13:18:48.846102 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:48.845933    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e443ff59-e6ec-444d-817a-1b7baaa5faf7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96\" (UID: \"e443ff59-e6ec-444d-817a-1b7baaa5faf7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96"
Oct 30 13:18:48.903859 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:48.903821    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp"]
Oct 30 13:18:48.935999 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:48.935026    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59m4v\" (UniqueName: \"kubernetes.io/projected/e443ff59-e6ec-444d-817a-1b7baaa5faf7-kube-api-access-59m4v\") pod \"payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96\" (UID: \"e443ff59-e6ec-444d-817a-1b7baaa5faf7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96"
Oct 30 13:18:48.946333 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:48.946300    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a9a0392-a7ea-4d31-add2-f4faf60c121b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp\" (UID: \"9a9a0392-a7ea-4d31-add2-f4faf60c121b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp"
Oct 30 13:18:48.946550 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:48.946409    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt44z\" (UniqueName: \"kubernetes.io/projected/9a9a0392-a7ea-4d31-add2-f4faf60c121b-kube-api-access-xt44z\") pod \"payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp\" (UID: \"9a9a0392-a7ea-4d31-add2-f4faf60c121b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp"
Oct 30 13:18:48.947103 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:48.947077    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a9a0392-a7ea-4d31-add2-f4faf60c121b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp\" (UID: \"9a9a0392-a7ea-4d31-add2-f4faf60c121b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp"
Oct 30 13:18:48.969758 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:48.969716    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt44z\" (UniqueName: \"kubernetes.io/projected/9a9a0392-a7ea-4d31-add2-f4faf60c121b-kube-api-access-xt44z\") pod \"payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp\" (UID: \"9a9a0392-a7ea-4d31-add2-f4faf60c121b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp"
Oct 30 13:18:49.002521 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:49.002483    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp"
Oct 30 13:18:49.153936 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:49.153855    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96"
Oct 30 13:18:49.321202 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:49.321172    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp"]
Oct 30 13:18:49.321579 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:18:49.321552    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a9a0392_a7ea_4d31_add2_f4faf60c121b.slice/crio-5a7deae3f22606a66de62363c74fce150a647829726e5403b09d2e2539b8b676 WatchSource:0}: Error finding container 5a7deae3f22606a66de62363c74fce150a647829726e5403b09d2e2539b8b676: Status 404 returned error can't find the container with id 5a7deae3f22606a66de62363c74fce150a647829726e5403b09d2e2539b8b676
Oct 30 13:18:49.476400 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:49.476371    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96"]
Oct 30 13:18:49.476955 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:18:49.476923    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode443ff59_e6ec_444d_817a_1b7baaa5faf7.slice/crio-f098e396907497c6478cc6e52ce9a4f73ce40bab1fd7621b80e2a52d5a30eca7 WatchSource:0}: Error finding container f098e396907497c6478cc6e52ce9a4f73ce40bab1fd7621b80e2a52d5a30eca7: Status 404 returned error can't find the container with id f098e396907497c6478cc6e52ce9a4f73ce40bab1fd7621b80e2a52d5a30eca7
Oct 30 13:18:49.946328 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:49.946290    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96" event={"ID":"e443ff59-e6ec-444d-817a-1b7baaa5faf7","Type":"ContainerStarted","Data":"d9002011d7106ce68108af211ad6c4b8a3e5b73f1b2ee4b2f52077579674310e"}
Oct 30 13:18:49.946776 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:49.946360    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96" event={"ID":"e443ff59-e6ec-444d-817a-1b7baaa5faf7","Type":"ContainerStarted","Data":"f098e396907497c6478cc6e52ce9a4f73ce40bab1fd7621b80e2a52d5a30eca7"}
Oct 30 13:18:49.946776 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:49.946396    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96"
Oct 30 13:18:49.948281 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:49.948215    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp" event={"ID":"9a9a0392-a7ea-4d31-add2-f4faf60c121b","Type":"ContainerStarted","Data":"24d18cb96a8ece8aea454d2374a2cf341423ab88939b9a9c4bbab66d1e2fcc06"}
Oct 30 13:18:49.948414 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:49.948310    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp" event={"ID":"9a9a0392-a7ea-4d31-add2-f4faf60c121b","Type":"ContainerStarted","Data":"5a7deae3f22606a66de62363c74fce150a647829726e5403b09d2e2539b8b676"}
Oct 30 13:18:49.948414 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:49.948341    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp"
Oct 30 13:18:49.988911 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:49.988836    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96" podStartSLOduration=1.9888203469999999 podStartE2EDuration="1.988820347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:49.986490377 +0000 UTC m=+2025.033624332" watchObservedRunningTime="2026-10-30 13:18:49.988820347 +0000 UTC m=+2025.035954300"
Oct 30 13:18:50.030251 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:50.030185    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp" podStartSLOduration=2.030168714 podStartE2EDuration="2.030168714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:50.02603907 +0000 UTC m=+2025.073173022" watchObservedRunningTime="2026-10-30 13:18:50.030168714 +0000 UTC m=+2025.077302667"
Oct 30 13:18:50.170636 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:50.170605    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:51.894692 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:51.894643    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:52.617213 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:52.617174    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:52.738946 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:52.738907    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:53.778087 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:53.778046    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:55.613827 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:55.613786    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:55.733244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:55.733086    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:56.541274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:56.541196    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r"
Oct 30 13:18:56.555635 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:56.555606    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r"
Oct 30 13:18:57.063476 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:57.063447    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r"
Oct 30 13:18:57.660766 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:57.660729    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:18:59.010021 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:18:59.009983    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:18:59.372804 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:18:59.372761    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:00.017152 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:00.017119    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:00.480145 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:00.480103    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:00.961699 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:00.961671    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96"
Oct 30 13:19:00.964991 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:00.964967    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp"
Oct 30 13:19:01.063407 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:01.063357    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:01.298350 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:01.278944    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:01.735885 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:01.735849    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:03.702344 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:03.702304    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:04.366633 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:04.366598    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:04.529727 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:04.529686    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:05.738025 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:05.737985    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:06.197005 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:06.196966    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:06.334863 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:06.334825    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:08.108974 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:08.108937    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:10.087480 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:10.087438    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:12.230372 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:12.229336    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:14.169540 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:14.169506    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:15.937874 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:15.937841    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:17.771867 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:17.771831    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:17.928779 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:17.928734    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:18.087828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:18.087790    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:19.949365 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:19.949301    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:21.784167 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:21.784124    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:23.794040 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:23.794002    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:24.832198 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:24.832164    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:25.473413 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:25.473382    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11ad6987-rn5cj"]
Oct 30 13:19:25.482068 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:25.482048    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ad6987-rn5cj"
Oct 30 13:19:25.515677 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:25.515645    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11ad6987-rn5cj"]
Oct 30 13:19:25.516514 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:25.516489    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:19:25.516810 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:25.516747    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:19:25.625044 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:25.625014    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11ad6987-rn5cj\" (UID: \"0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ad6987-rn5cj"
Oct 30 13:19:25.625214 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:25.625085    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv9kh\" (UniqueName: \"kubernetes.io/projected/0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8-kube-api-access-dv9kh\") pod \"maas-api-revoke-sub-11ad6987-rn5cj\" (UID: \"0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ad6987-rn5cj"
Oct 30 13:19:25.625289 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:25.625208    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11ad6987-rn5cj\" (UID: \"0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ad6987-rn5cj"
Oct 30 13:19:25.726505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:25.726411    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv9kh\" (UniqueName: \"kubernetes.io/projected/0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8-kube-api-access-dv9kh\") pod \"maas-api-revoke-sub-11ad6987-rn5cj\" (UID: \"0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ad6987-rn5cj"
Oct 30 13:19:25.726505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:25.726474    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11ad6987-rn5cj\" (UID: \"0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ad6987-rn5cj"
Oct 30 13:19:25.726803 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:25.726784    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11ad6987-rn5cj\" (UID: \"0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ad6987-rn5cj"
Oct 30 13:19:25.727112 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:25.727092    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11ad6987-rn5cj\" (UID: \"0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ad6987-rn5cj"
Oct 30 13:19:25.729263 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:25.729212    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11ad6987-rn5cj\" (UID: \"0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ad6987-rn5cj"
Oct 30 13:19:25.750338 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:25.750308    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv9kh\" (UniqueName: \"kubernetes.io/projected/0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8-kube-api-access-dv9kh\") pod \"maas-api-revoke-sub-11ad6987-rn5cj\" (UID: \"0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ad6987-rn5cj"
Oct 30 13:19:25.802251 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:25.797907    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ad6987-rn5cj"
Oct 30 13:19:25.833966 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:25.833925    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:26.143422 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:26.143394    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11ad6987-rn5cj"]
Oct 30 13:19:26.144899 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:19:26.144872    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bdcd77f_9e49_468f_a3f1_96ef6cc33eb8.slice/crio-17177b426c066c3d955901bed5bd902b3e547f632efff92a4c32f65474803f60 WatchSource:0}: Error finding container 17177b426c066c3d955901bed5bd902b3e547f632efff92a4c32f65474803f60: Status 404 returned error can't find the container with id 17177b426c066c3d955901bed5bd902b3e547f632efff92a4c32f65474803f60
Oct 30 13:19:26.420938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:26.420898    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ad6987-rn5cj" event={"ID":"0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8","Type":"ContainerStarted","Data":"17177b426c066c3d955901bed5bd902b3e547f632efff92a4c32f65474803f60"}
Oct 30 13:19:27.517713 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:27.517679    4544 generic.go:356] "Generic (PLEG): container finished" podID="0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8" containerID="a612aa2a754ff5add34ed1c4e1d39a378f8bbbd0f4e51a83c9558b0cd9728085" exitCode=0
Oct 30 13:19:27.519041 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:27.518752    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ad6987-rn5cj" event={"ID":"0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8","Type":"ContainerDied","Data":"a612aa2a754ff5add34ed1c4e1d39a378f8bbbd0f4e51a83c9558b0cd9728085"}
Oct 30 13:19:27.808794 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:27.808694    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:29.292245 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:19:29.174339    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:19:29.322499 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:29.322470    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ad6987-rn5cj"
Oct 30 13:19:29.433961 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:29.433863    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8-maas-api-xks-ca\") pod \"0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8\" (UID: \"0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8\") "
Oct 30 13:19:29.438750 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:29.438718    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8-kube-api-access-dv9kh\" (UniqueName: \"kubernetes.io/projected/0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8-kube-api-access-dv9kh\") pod \"0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8\" (UID: \"0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8\") "
Oct 30 13:19:29.444690 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:29.444651    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8-maas-api-openshift-ca\") pod \"0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8\" (UID: \"0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8\") "
Oct 30 13:19:29.456756 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:29.447696    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8-maas-api-openshift-ca" pod "0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8" (UID: "0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:19:29.550245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:29.546815    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8-kube-api-access-dv9kh" pod "0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8" (UID: "0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8"). InnerVolumeSpecName "kube-api-access-dv9kh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:19:29.550245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:29.546912    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8-maas-api-xks-ca" pod "0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8" (UID: "0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:19:29.573566 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:29.568900    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:19:29.573566 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:29.568942    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:19:29.573566 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:29.568961    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv9kh\" (UniqueName: \"kubernetes.io/projected/0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8-kube-api-access-dv9kh\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:19:29.662270 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:29.661263    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ad6987-rn5cj" event={"ID":"0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8","Type":"ContainerDied","Data":"17177b426c066c3d955901bed5bd902b3e547f632efff92a4c32f65474803f60"}
Oct 30 13:19:29.662270 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:29.661324    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17177b426c066c3d955901bed5bd902b3e547f632efff92a4c32f65474803f60"
Oct 30 13:19:29.662270 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:29.661387    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ad6987-rn5cj"
Oct 30 13:19:30.360164 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:30.360073    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:30.814301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:30.806162    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:32.840063 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:32.838332    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:34.883407 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:34.883369    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:36.767398 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:36.767362    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:38.546094 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:38.546056    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:38.675438 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:38.675402    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:38.766904 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:38.765365    4544 generic.go:356] "Generic (PLEG): container finished" podID="2e32d130-6ba4-4391-bf2a-7bd6548c1c88" containerID="a578a827da5dbae043fd6b228775c44f833308d1c2bb162eb2e7f39f47d1a53f" exitCode=0
Oct 30 13:19:38.766904 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:38.765690    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m" event={"ID":"2e32d130-6ba4-4391-bf2a-7bd6548c1c88","Type":"ContainerDied","Data":"a578a827da5dbae043fd6b228775c44f833308d1c2bb162eb2e7f39f47d1a53f"}
Oct 30 13:19:38.766904 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:38.766383    4544 scope.go:122] "RemoveContainer" containerID="a578a827da5dbae043fd6b228775c44f833308d1c2bb162eb2e7f39f47d1a53f"
Oct 30 13:19:38.769529 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:38.769488    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:19:38.769741 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:38.769590    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:19:38.769741 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:38.769649    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:19:38.835503 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:38.835461    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:39.776070 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:39.776033    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m" event={"ID":"2e32d130-6ba4-4391-bf2a-7bd6548c1c88","Type":"ContainerStarted","Data":"fe886e9d20ad0a1ea488f5d69254df6d276da05e6f3b0d107eeb10c70a8795a1"}
Oct 30 13:19:39.894922 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:39.887717    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:19:39.894922 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:39.887871    4544 patch_prober.go:29] interesting pod/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 13:19:39.894922 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:39.887915    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m" podUID="2e32d130-6ba4-4391-bf2a-7bd6548c1c88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 13:19:40.696419 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:40.696385    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:40.873247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:40.872496    4544 patch_prober.go:29] interesting pod/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 13:19:40.873247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:40.872552    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m" podUID="2e32d130-6ba4-4391-bf2a-7bd6548c1c88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 13:19:40.906477 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:40.906443    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:19:41.873381 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:41.873346    4544 patch_prober.go:29] interesting pod/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 13:19:41.873796 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:41.873396    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m" podUID="2e32d130-6ba4-4391-bf2a-7bd6548c1c88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 13:19:42.690000 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:42.689955    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:42.872976 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:42.872941    4544 patch_prober.go:29] interesting pod/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 13:19:42.873189 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:42.873009    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m" podUID="2e32d130-6ba4-4391-bf2a-7bd6548c1c88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 13:19:43.872571 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:43.872533    4544 patch_prober.go:29] interesting pod/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 13:19:43.873010 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:43.872604    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m" podUID="2e32d130-6ba4-4391-bf2a-7bd6548c1c88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 13:19:44.557806 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:44.557773    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:44.873317 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:44.873268    4544 patch_prober.go:29] interesting pod/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 13:19:44.873745 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:44.873348    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m" podUID="2e32d130-6ba4-4391-bf2a-7bd6548c1c88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 13:19:45.796491 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:45.796448    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:45.878409 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:45.878371    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:19:45.879769 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:45.879746    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:19:45.903180 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:45.903150    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:46.479139 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:46.479102    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:48.034781 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:48.034744    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:48.212933 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:48.212898    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:48.521895 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:48.521849    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:50.240891 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:50.240859    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:52.045195 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:52.045156    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:52.199647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:52.199541    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:53.010023 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:53.009987    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4eabbb8-bbncg"]
Oct 30 13:19:53.010924 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:53.010906    4544 cpu_manager.go:414] "containerMap: removing container" podUID="0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8" containerName="revoke-keys"
Oct 30 13:19:53.010983 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:53.010927    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8" containerName="revoke-keys"
Oct 30 13:19:53.011052 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:53.011043    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8" containerName="revoke-keys"
Oct 30 13:19:53.015464 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:53.015439    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4eabbb8-bbncg"
Oct 30 13:19:53.040299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:53.040261    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4eabbb8-bbncg"]
Oct 30 13:19:53.045415 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:53.045375    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:19:53.045415 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:53.045402    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:19:53.127463 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:53.127419    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4eabbb8-bbncg\" (UID: \"64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4eabbb8-bbncg"
Oct 30 13:19:53.127697 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:53.127485    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6p6c\" (UniqueName: \"kubernetes.io/projected/64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb-kube-api-access-n6p6c\") pod \"maas-api-revoke-sub-b4eabbb8-bbncg\" (UID: \"64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4eabbb8-bbncg"
Oct 30 13:19:53.127770 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:53.127744    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4eabbb8-bbncg\" (UID: \"64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4eabbb8-bbncg"
Oct 30 13:19:53.228878 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:53.228838    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4eabbb8-bbncg\" (UID: \"64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4eabbb8-bbncg"
Oct 30 13:19:53.229068 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:53.229013    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4eabbb8-bbncg\" (UID: \"64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4eabbb8-bbncg"
Oct 30 13:19:53.229142 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:53.229079    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6p6c\" (UniqueName: \"kubernetes.io/projected/64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb-kube-api-access-n6p6c\") pod \"maas-api-revoke-sub-b4eabbb8-bbncg\" (UID: \"64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4eabbb8-bbncg"
Oct 30 13:19:53.229623 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:53.229601    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4eabbb8-bbncg\" (UID: \"64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4eabbb8-bbncg"
Oct 30 13:19:53.231476 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:53.231451    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4eabbb8-bbncg\" (UID: \"64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4eabbb8-bbncg"
Oct 30 13:19:53.255722 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:53.255683    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6p6c\" (UniqueName: \"kubernetes.io/projected/64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb-kube-api-access-n6p6c\") pod \"maas-api-revoke-sub-b4eabbb8-bbncg\" (UID: \"64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4eabbb8-bbncg"
Oct 30 13:19:53.331048 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:53.331015    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4eabbb8-bbncg"
Oct 30 13:19:53.654032 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:53.653998    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4eabbb8-bbncg"]
Oct 30 13:19:53.656306 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:19:53.656257    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a5d1f0_bcfe_4bf9_bb2f_791a3180b3fb.slice/crio-e37a0eebc1de6d3a9672d3a43e331c07a284876ddceed54d3006ba451ef9c4cf WatchSource:0}: Error finding container e37a0eebc1de6d3a9672d3a43e331c07a284876ddceed54d3006ba451ef9c4cf: Status 404 returned error can't find the container with id e37a0eebc1de6d3a9672d3a43e331c07a284876ddceed54d3006ba451ef9c4cf
Oct 30 13:19:53.974378 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:53.974272    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4eabbb8-bbncg" event={"ID":"64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb","Type":"ContainerStarted","Data":"e37a0eebc1de6d3a9672d3a43e331c07a284876ddceed54d3006ba451ef9c4cf"}
Oct 30 13:19:53.981415 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:53.981380    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:54.985048 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:54.985006    4544 generic.go:356] "Generic (PLEG): container finished" podID="64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb" containerID="8df076bfb3c1061c01514a7b1ca3dea2b44f15c368ad4ba0a04c79824eeef0d2" exitCode=0
Oct 30 13:19:54.985541 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:54.985074    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4eabbb8-bbncg" event={"ID":"64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb","Type":"ContainerDied","Data":"8df076bfb3c1061c01514a7b1ca3dea2b44f15c368ad4ba0a04c79824eeef0d2"}
Oct 30 13:19:55.347037 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:55.346983    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:55.816139 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:55.815982    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:56.180416 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:56.180385    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4eabbb8-bbncg"
Oct 30 13:19:56.276877 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:56.276834    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:56.291128 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:56.291071    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb-maas-api-openshift-ca\") pod \"64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb\" (UID: \"64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb\") "
Oct 30 13:19:56.292126 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:56.292102    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb-kube-api-access-n6p6c\" (UniqueName: \"kubernetes.io/projected/64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb-kube-api-access-n6p6c\") pod \"64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb\" (UID: \"64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb\") "
Oct 30 13:19:56.292862 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:56.292838    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb-maas-api-xks-ca\") pod \"64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb\" (UID: \"64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb\") "
Oct 30 13:19:56.299346 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:56.291868    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb-maas-api-openshift-ca" pod "64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb" (UID: "64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:19:56.299592 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:56.298814    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb-maas-api-xks-ca" pod "64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb" (UID: "64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:19:56.305819 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:56.305782    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb-kube-api-access-n6p6c" pod "64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb" (UID: "64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb"). InnerVolumeSpecName "kube-api-access-n6p6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:19:56.398775 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:56.398667    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:19:56.398775 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:56.398710    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6p6c\" (UniqueName: \"kubernetes.io/projected/64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb-kube-api-access-n6p6c\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:19:56.398775 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:56.398726    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:19:57.002961 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:57.002933    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4eabbb8-bbncg"
Oct 30 13:19:57.003159 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:57.002965    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4eabbb8-bbncg" event={"ID":"64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb","Type":"ContainerDied","Data":"e37a0eebc1de6d3a9672d3a43e331c07a284876ddceed54d3006ba451ef9c4cf"}
Oct 30 13:19:57.003159 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:57.003047    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e37a0eebc1de6d3a9672d3a43e331c07a284876ddceed54d3006ba451ef9c4cf"
Oct 30 13:19:58.616400 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:58.549348    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:59.011734 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:19:59.011645    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:19:59.238109 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:59.238052    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a3e01-p8hjg"]
Oct 30 13:19:59.239845 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:59.239813    4544 cpu_manager.go:414] "containerMap: removing container" podUID="64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb" containerName="revoke-keys"
Oct 30 13:19:59.240072 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:59.240054    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb" containerName="revoke-keys"
Oct 30 13:19:59.240473 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:59.240453    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb" containerName="revoke-keys"
Oct 30 13:19:59.245279 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:59.245209    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a3e01-p8hjg"
Oct 30 13:19:59.293533 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:59.293447    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a3e01-p8hjg"]
Oct 30 13:19:59.302735 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:59.302699    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58w86\" (UniqueName: \"kubernetes.io/projected/8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6-kube-api-access-58w86\") pod \"maas-api-revoke-keys-e2e-inf-a-2a3e01-p8hjg\" (UID: \"8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a3e01-p8hjg"
Oct 30 13:19:59.302908 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:59.302885    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2a3e01-p8hjg\" (UID: \"8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a3e01-p8hjg"
Oct 30 13:19:59.302970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:59.302955    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2a3e01-p8hjg\" (UID: \"8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a3e01-p8hjg"
Oct 30 13:19:59.336984 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:59.336944    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:19:59.337209 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:59.337006    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:19:59.371197 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:59.371159    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:59.404756 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:59.404713    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2a3e01-p8hjg\" (UID: \"8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a3e01-p8hjg"
Oct 30 13:19:59.404955 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:59.404807    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2a3e01-p8hjg\" (UID: \"8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a3e01-p8hjg"
Oct 30 13:19:59.405002 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:59.404966    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58w86\" (UniqueName: \"kubernetes.io/projected/8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6-kube-api-access-58w86\") pod \"maas-api-revoke-keys-e2e-inf-a-2a3e01-p8hjg\" (UID: \"8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a3e01-p8hjg"
Oct 30 13:19:59.405589 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:59.405560    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2a3e01-p8hjg\" (UID: \"8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a3e01-p8hjg"
Oct 30 13:19:59.407299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:59.407275    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2a3e01-p8hjg\" (UID: \"8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a3e01-p8hjg"
Oct 30 13:19:59.432742 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:59.432702    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58w86\" (UniqueName: \"kubernetes.io/projected/8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6-kube-api-access-58w86\") pod \"maas-api-revoke-keys-e2e-inf-a-2a3e01-p8hjg\" (UID: \"8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a3e01-p8hjg"
Oct 30 13:19:59.583247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:59.581184    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:19:59.603794 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:59.603410    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a3e01-p8hjg"
Oct 30 13:19:59.940248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:19:59.940200    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a3e01-p8hjg"]
Oct 30 13:19:59.940987 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:19:59.940953    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b3a8cb5_9d41_4cac_a9d2_226c5d76a9a6.slice/crio-0b95765ef86c1f950e14d37691538b82bbc748610dd6ab59f6fccec6fca12d3c WatchSource:0}: Error finding container 0b95765ef86c1f950e14d37691538b82bbc748610dd6ab59f6fccec6fca12d3c: Status 404 returned error can't find the container with id 0b95765ef86c1f950e14d37691538b82bbc748610dd6ab59f6fccec6fca12d3c
Oct 30 13:20:00.133170 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:00.133135    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a3e01-p8hjg" event={"ID":"8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6","Type":"ContainerStarted","Data":"0b95765ef86c1f950e14d37691538b82bbc748610dd6ab59f6fccec6fca12d3c"}
Oct 30 13:20:01.207472 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:01.207374    4544 generic.go:356] "Generic (PLEG): container finished" podID="8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6" containerID="724c0732e2f42ea495c2b006f5b2355b777a854b6038b6018f7d83354e1b82dc" exitCode=0
Oct 30 13:20:01.207966 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:01.207606    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a3e01-p8hjg" event={"ID":"8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6","Type":"ContainerDied","Data":"724c0732e2f42ea495c2b006f5b2355b777a854b6038b6018f7d83354e1b82dc"}
Oct 30 13:20:02.165281 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:02.165066    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:02.410411 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:02.410380    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a3e01-p8hjg"
Oct 30 13:20:02.527504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:02.527407    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6-kube-api-access-58w86\" (UniqueName: \"kubernetes.io/projected/8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6-kube-api-access-58w86\") pod \"8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6\" (UID: \"8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6\") "
Oct 30 13:20:02.527504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:02.527460    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6-maas-api-xks-ca\") pod \"8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6\" (UID: \"8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6\") "
Oct 30 13:20:02.527805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:02.527783    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6-maas-api-openshift-ca\") pod \"8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6\" (UID: \"8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6\") "
Oct 30 13:20:02.528781 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:02.528346    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6-maas-api-openshift-ca" pod "8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6" (UID: "8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:20:02.528781 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:02.528739    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:02.530749 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:02.530723    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6-maas-api-xks-ca" pod "8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6" (UID: "8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:20:02.530968 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:02.530850    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6-kube-api-access-58w86" pod "8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6" (UID: "8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6"). InnerVolumeSpecName "kube-api-access-58w86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:02.630740 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:02.630707    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58w86\" (UniqueName: \"kubernetes.io/projected/8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6-kube-api-access-58w86\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:02.630740 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:02.630736    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:03.238631 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:03.238601    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a3e01-p8hjg"
Oct 30 13:20:03.238828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:03.238603    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a3e01-p8hjg" event={"ID":"8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6","Type":"ContainerDied","Data":"0b95765ef86c1f950e14d37691538b82bbc748610dd6ab59f6fccec6fca12d3c"}
Oct 30 13:20:03.238828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:03.238735    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b95765ef86c1f950e14d37691538b82bbc748610dd6ab59f6fccec6fca12d3c"
Oct 30 13:20:04.135094 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:04.135063    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz"]
Oct 30 13:20:04.136034 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:04.136013    4544 cpu_manager.go:414] "containerMap: removing container" podUID="8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6" containerName="revoke-keys"
Oct 30 13:20:04.136034 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:04.136032    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6" containerName="revoke-keys"
Oct 30 13:20:04.136302 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:04.136186    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6" containerName="revoke-keys"
Oct 30 13:20:04.141264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:04.141205    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz"
Oct 30 13:20:04.167889 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:04.167860    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz"]
Oct 30 13:20:04.173653 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:04.173628    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-bec983\"/\"openshift-service-ca.crt\""
Oct 30 13:20:04.173858 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:04.173628    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-bec983\"/\"kube-root-ca.crt\""
Oct 30 13:20:04.174203 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:04.174185    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-bec983\"/\"default-dockercfg-xlbp8\""
Oct 30 13:20:04.178003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:04.177981    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-bec983\"/\"test-model-bec983-kserve-self-signed-certs\""
Oct 30 13:20:04.219181 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:04.219148    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:04.259209 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:04.259174    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41182c29-4530-4a65-87b6-781a2fd06ff8-model-cache\") pod \"test-model-bec983-kserve-564766445f-5xbrz\" (UID: \"41182c29-4530-4a65-87b6-781a2fd06ff8\") " pod="ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz"
Oct 30 13:20:04.259424 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:04.259366    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swjj6\" (UniqueName: \"kubernetes.io/projected/41182c29-4530-4a65-87b6-781a2fd06ff8-kube-api-access-swjj6\") pod \"test-model-bec983-kserve-564766445f-5xbrz\" (UID: \"41182c29-4530-4a65-87b6-781a2fd06ff8\") " pod="ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz"
Oct 30 13:20:04.259557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:04.259540    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41182c29-4530-4a65-87b6-781a2fd06ff8-dshm\") pod \"test-model-bec983-kserve-564766445f-5xbrz\" (UID: \"41182c29-4530-4a65-87b6-781a2fd06ff8\") " pod="ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz"
Oct 30 13:20:04.259621 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:04.259608    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41182c29-4530-4a65-87b6-781a2fd06ff8-tls-certs\") pod \"test-model-bec983-kserve-564766445f-5xbrz\" (UID: \"41182c29-4530-4a65-87b6-781a2fd06ff8\") " pod="ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz"
Oct 30 13:20:04.259712 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:04.259695    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41182c29-4530-4a65-87b6-781a2fd06ff8-tmp-dir\") pod \"test-model-bec983-kserve-564766445f-5xbrz\" (UID: \"41182c29-4530-4a65-87b6-781a2fd06ff8\") " pod="ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz"
Oct 30 13:20:04.259787 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:04.259770    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41182c29-4530-4a65-87b6-781a2fd06ff8-home\") pod \"test-model-bec983-kserve-564766445f-5xbrz\" (UID: \"41182c29-4530-4a65-87b6-781a2fd06ff8\") " pod="ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz"
Oct 30 13:20:04.361177 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:04.361137    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41182c29-4530-4a65-87b6-781a2fd06ff8-dshm\") pod \"test-model-bec983-kserve-564766445f-5xbrz\" (UID: \"41182c29-4530-4a65-87b6-781a2fd06ff8\") " pod="ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz"
Oct 30 13:20:04.361379 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:04.361266    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41182c29-4530-4a65-87b6-781a2fd06ff8-tls-certs\") pod \"test-model-bec983-kserve-564766445f-5xbrz\" (UID: \"41182c29-4530-4a65-87b6-781a2fd06ff8\") " pod="ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz"
Oct 30 13:20:04.361379 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:04.361320    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41182c29-4530-4a65-87b6-781a2fd06ff8-tmp-dir\") pod \"test-model-bec983-kserve-564766445f-5xbrz\" (UID: \"41182c29-4530-4a65-87b6-781a2fd06ff8\") " pod="ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz"
Oct 30 13:20:04.361460 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:04.361382    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41182c29-4530-4a65-87b6-781a2fd06ff8-home\") pod \"test-model-bec983-kserve-564766445f-5xbrz\" (UID: \"41182c29-4530-4a65-87b6-781a2fd06ff8\") " pod="ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz"
Oct 30 13:20:04.361510 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:04.361493    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41182c29-4530-4a65-87b6-781a2fd06ff8-model-cache\") pod \"test-model-bec983-kserve-564766445f-5xbrz\" (UID: \"41182c29-4530-4a65-87b6-781a2fd06ff8\") " pod="ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz"
Oct 30 13:20:04.361629 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:04.361610    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swjj6\" (UniqueName: \"kubernetes.io/projected/41182c29-4530-4a65-87b6-781a2fd06ff8-kube-api-access-swjj6\") pod \"test-model-bec983-kserve-564766445f-5xbrz\" (UID: \"41182c29-4530-4a65-87b6-781a2fd06ff8\") " pod="ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz"
Oct 30 13:20:04.361822 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:04.361790    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41182c29-4530-4a65-87b6-781a2fd06ff8-home\") pod \"test-model-bec983-kserve-564766445f-5xbrz\" (UID: \"41182c29-4530-4a65-87b6-781a2fd06ff8\") " pod="ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz"
Oct 30 13:20:04.361822 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:04.361790    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41182c29-4530-4a65-87b6-781a2fd06ff8-tmp-dir\") pod \"test-model-bec983-kserve-564766445f-5xbrz\" (UID: \"41182c29-4530-4a65-87b6-781a2fd06ff8\") " pod="ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz"
Oct 30 13:20:04.362175 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:04.362150    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41182c29-4530-4a65-87b6-781a2fd06ff8-model-cache\") pod \"test-model-bec983-kserve-564766445f-5xbrz\" (UID: \"41182c29-4530-4a65-87b6-781a2fd06ff8\") " pod="ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz"
Oct 30 13:20:04.363846 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:04.363824    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41182c29-4530-4a65-87b6-781a2fd06ff8-dshm\") pod \"test-model-bec983-kserve-564766445f-5xbrz\" (UID: \"41182c29-4530-4a65-87b6-781a2fd06ff8\") " pod="ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz"
Oct 30 13:20:04.363972 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:04.363953    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41182c29-4530-4a65-87b6-781a2fd06ff8-tls-certs\") pod \"test-model-bec983-kserve-564766445f-5xbrz\" (UID: \"41182c29-4530-4a65-87b6-781a2fd06ff8\") " pod="ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz"
Oct 30 13:20:04.393709 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:04.393619    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swjj6\" (UniqueName: \"kubernetes.io/projected/41182c29-4530-4a65-87b6-781a2fd06ff8-kube-api-access-swjj6\") pod \"test-model-bec983-kserve-564766445f-5xbrz\" (UID: \"41182c29-4530-4a65-87b6-781a2fd06ff8\") " pod="ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz"
Oct 30 13:20:04.418162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:04.418126    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:04.462597 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:04.462560    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz"
Oct 30 13:20:04.955970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:04.955941    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz"]
Oct 30 13:20:04.957303 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:20:04.957241    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41182c29_4530_4a65_87b6_781a2fd06ff8.slice/crio-cb6eb8f8f02e7903f6f951f4f95c2d1f28d21f6e0e7dcdfd48d775b35c771543 WatchSource:0}: Error finding container cb6eb8f8f02e7903f6f951f4f95c2d1f28d21f6e0e7dcdfd48d775b35c771543: Status 404 returned error can't find the container with id cb6eb8f8f02e7903f6f951f4f95c2d1f28d21f6e0e7dcdfd48d775b35c771543
Oct 30 13:20:05.260970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:05.260837    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz" event={"ID":"41182c29-4530-4a65-87b6-781a2fd06ff8","Type":"ContainerStarted","Data":"1c266916894d619ed4527ccfc5ac608560bfc85c40984188e1657620c40f6fe3"}
Oct 30 13:20:05.260970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:05.260911    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz" event={"ID":"41182c29-4530-4a65-87b6-781a2fd06ff8","Type":"ContainerStarted","Data":"cb6eb8f8f02e7903f6f951f4f95c2d1f28d21f6e0e7dcdfd48d775b35c771543"}
Oct 30 13:20:05.307417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:05.307336    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz" podStartSLOduration=1.307315241 podStartE2EDuration="1.307315241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:05.302995519 +0000 UTC m=+2100.350129474" watchObservedRunningTime="2026-10-30 13:20:05.307315241 +0000 UTC m=+2100.354449255"
Oct 30 13:20:05.466244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:05.465949    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz"
Oct 30 13:20:06.546917 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:06.546878    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:08.541244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:08.540449    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:09.747167 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:09.747133    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq"]
Oct 30 13:20:09.912201 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:09.912168    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:10.050928 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:10.050821    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:10.385566 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:10.385529    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:12.437796 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:12.437749    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:13.241274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:13.241201    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:13.803574 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:13.803538    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9a91dcc-vprwx"]
Oct 30 13:20:13.825684 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:13.825648    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a91dcc-vprwx"
Oct 30 13:20:13.837628 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:13.837594    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9a91dcc-vprwx"]
Oct 30 13:20:13.846783 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:13.845900    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:20:13.856960 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:13.856925    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:20:13.944757 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:13.944725    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nddgb\" (UniqueName: \"kubernetes.io/projected/983f6389-65d1-4dea-b68b-4fc4d23098fe-kube-api-access-nddgb\") pod \"maas-api-revoke-sub-b9a91dcc-vprwx\" (UID: \"983f6389-65d1-4dea-b68b-4fc4d23098fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a91dcc-vprwx"
Oct 30 13:20:13.944937 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:13.944904    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983f6389-65d1-4dea-b68b-4fc4d23098fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9a91dcc-vprwx\" (UID: \"983f6389-65d1-4dea-b68b-4fc4d23098fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a91dcc-vprwx"
Oct 30 13:20:13.945096 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:13.945081    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983f6389-65d1-4dea-b68b-4fc4d23098fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9a91dcc-vprwx\" (UID: \"983f6389-65d1-4dea-b68b-4fc4d23098fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a91dcc-vprwx"
Oct 30 13:20:14.047041 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:14.047010    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nddgb\" (UniqueName: \"kubernetes.io/projected/983f6389-65d1-4dea-b68b-4fc4d23098fe-kube-api-access-nddgb\") pod \"maas-api-revoke-sub-b9a91dcc-vprwx\" (UID: \"983f6389-65d1-4dea-b68b-4fc4d23098fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a91dcc-vprwx"
Oct 30 13:20:14.047278 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:14.047124    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983f6389-65d1-4dea-b68b-4fc4d23098fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9a91dcc-vprwx\" (UID: \"983f6389-65d1-4dea-b68b-4fc4d23098fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a91dcc-vprwx"
Oct 30 13:20:14.047361 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:14.047281    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983f6389-65d1-4dea-b68b-4fc4d23098fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9a91dcc-vprwx\" (UID: \"983f6389-65d1-4dea-b68b-4fc4d23098fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a91dcc-vprwx"
Oct 30 13:20:14.047882 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:14.047857    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983f6389-65d1-4dea-b68b-4fc4d23098fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9a91dcc-vprwx\" (UID: \"983f6389-65d1-4dea-b68b-4fc4d23098fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a91dcc-vprwx"
Oct 30 13:20:14.049728 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:14.049707    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983f6389-65d1-4dea-b68b-4fc4d23098fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9a91dcc-vprwx\" (UID: \"983f6389-65d1-4dea-b68b-4fc4d23098fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a91dcc-vprwx"
Oct 30 13:20:14.073112 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:14.073085    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nddgb\" (UniqueName: \"kubernetes.io/projected/983f6389-65d1-4dea-b68b-4fc4d23098fe-kube-api-access-nddgb\") pod \"maas-api-revoke-sub-b9a91dcc-vprwx\" (UID: \"983f6389-65d1-4dea-b68b-4fc4d23098fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a91dcc-vprwx"
Oct 30 13:20:14.156146 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:14.156111    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a91dcc-vprwx"
Oct 30 13:20:14.464852 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:14.464812    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz"
Oct 30 13:20:14.465530 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:14.465500    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9a91dcc-vprwx"]
Oct 30 13:20:14.467463 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:20:14.467433    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod983f6389_65d1_4dea_b68b_4fc4d23098fe.slice/crio-a15234e284976beb3a2b643bcefe4c886be249d0d996df982a29a58aa3698a15 WatchSource:0}: Error finding container a15234e284976beb3a2b643bcefe4c886be249d0d996df982a29a58aa3698a15: Status 404 returned error can't find the container with id a15234e284976beb3a2b643bcefe4c886be249d0d996df982a29a58aa3698a15
Oct 30 13:20:14.481854 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:14.481817    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz"
Oct 30 13:20:14.596428 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:14.596396    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:14.745697 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:14.745657    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:14.747560 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:14.747534    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq" podUID="2d6dcafc-5203-4f02-b8b8-c06fc9eb9353" containerName="payload-pre-processing" containerID="cri-o://fb2c21eff2c65186cc0647473c773103f43566a8891cc512c4e99893460de2d2" gracePeriod=25
Oct 30 13:20:15.384327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:15.384282    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a91dcc-vprwx" event={"ID":"983f6389-65d1-4dea-b68b-4fc4d23098fe","Type":"ContainerStarted","Data":"6e0aaf3cf0c7dd43cd3db6c159314a7eb63ea0d3a83fa2e5e67b2eaf4c028a94"}
Oct 30 13:20:15.384775 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:15.384350    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a91dcc-vprwx" event={"ID":"983f6389-65d1-4dea-b68b-4fc4d23098fe","Type":"ContainerStarted","Data":"a15234e284976beb3a2b643bcefe4c886be249d0d996df982a29a58aa3698a15"}
Oct 30 13:20:15.395922 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:15.395889    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz"
Oct 30 13:20:15.452571 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:15.452510    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a91dcc-vprwx" podStartSLOduration=2.452492264 podStartE2EDuration="2.452492264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:15.450511163 +0000 UTC m=+2110.497645117" watchObservedRunningTime="2026-10-30 13:20:15.452492264 +0000 UTC m=+2110.499626216"
Oct 30 13:20:16.394712 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:16.394674    4544 generic.go:356] "Generic (PLEG): container finished" podID="983f6389-65d1-4dea-b68b-4fc4d23098fe" containerID="6e0aaf3cf0c7dd43cd3db6c159314a7eb63ea0d3a83fa2e5e67b2eaf4c028a94" exitCode=0
Oct 30 13:20:16.395332 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:16.394785    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a91dcc-vprwx" event={"ID":"983f6389-65d1-4dea-b68b-4fc4d23098fe","Type":"ContainerDied","Data":"6e0aaf3cf0c7dd43cd3db6c159314a7eb63ea0d3a83fa2e5e67b2eaf4c028a94"}
Oct 30 13:20:16.695355 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:16.690880    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:17.579729 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:17.579692    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a91dcc-vprwx"
Oct 30 13:20:17.620449 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:17.620417    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/983f6389-65d1-4dea-b68b-4fc4d23098fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983f6389-65d1-4dea-b68b-4fc4d23098fe-maas-api-openshift-ca\") pod \"983f6389-65d1-4dea-b68b-4fc4d23098fe\" (UID: \"983f6389-65d1-4dea-b68b-4fc4d23098fe\") "
Oct 30 13:20:17.620641 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:17.620538    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/983f6389-65d1-4dea-b68b-4fc4d23098fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983f6389-65d1-4dea-b68b-4fc4d23098fe-maas-api-xks-ca\") pod \"983f6389-65d1-4dea-b68b-4fc4d23098fe\" (UID: \"983f6389-65d1-4dea-b68b-4fc4d23098fe\") "
Oct 30 13:20:17.620641 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:17.620606    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/983f6389-65d1-4dea-b68b-4fc4d23098fe-kube-api-access-nddgb\" (UniqueName: \"kubernetes.io/projected/983f6389-65d1-4dea-b68b-4fc4d23098fe-kube-api-access-nddgb\") pod \"983f6389-65d1-4dea-b68b-4fc4d23098fe\" (UID: \"983f6389-65d1-4dea-b68b-4fc4d23098fe\") "
Oct 30 13:20:17.620845 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:17.620817    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/983f6389-65d1-4dea-b68b-4fc4d23098fe-maas-api-openshift-ca" pod "983f6389-65d1-4dea-b68b-4fc4d23098fe" (UID: "983f6389-65d1-4dea-b68b-4fc4d23098fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:20:17.621173 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:17.621150    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983f6389-65d1-4dea-b68b-4fc4d23098fe-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:17.622797 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:17.622774    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/983f6389-65d1-4dea-b68b-4fc4d23098fe-kube-api-access-nddgb" pod "983f6389-65d1-4dea-b68b-4fc4d23098fe" (UID: "983f6389-65d1-4dea-b68b-4fc4d23098fe"). InnerVolumeSpecName "kube-api-access-nddgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:17.622916 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:17.622824    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/983f6389-65d1-4dea-b68b-4fc4d23098fe-maas-api-xks-ca" pod "983f6389-65d1-4dea-b68b-4fc4d23098fe" (UID: "983f6389-65d1-4dea-b68b-4fc4d23098fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:20:17.723408 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:17.723322    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983f6389-65d1-4dea-b68b-4fc4d23098fe-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:17.723408 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:17.723353    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nddgb\" (UniqueName: \"kubernetes.io/projected/983f6389-65d1-4dea-b68b-4fc4d23098fe-kube-api-access-nddgb\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:18.463208 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:18.463163    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a91dcc-vprwx" event={"ID":"983f6389-65d1-4dea-b68b-4fc4d23098fe","Type":"ContainerDied","Data":"a15234e284976beb3a2b643bcefe4c886be249d0d996df982a29a58aa3698a15"}
Oct 30 13:20:18.463408 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:18.463365    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a91dcc-vprwx"
Oct 30 13:20:18.481943 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:18.481325    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a15234e284976beb3a2b643bcefe4c886be249d0d996df982a29a58aa3698a15"
Oct 30 13:20:18.738093 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:18.737985    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:20.879337 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:20.871649    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:21.154404 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:21.154309    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq" podUID="2d6dcafc-5203-4f02-b8b8-c06fc9eb9353" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:20:22.884122 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:22.884086    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:24.214386 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:24.214347    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:24.721741 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:24.721696    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:25.003627 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:25.003545    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:25.281292 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:25.281192    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:25.471246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:25.470372    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:25.473253 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:25.472322    4544 patch_prober.go:29] interesting pod/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:20:25.473253 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:25.472417    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d" podUID="ed9d8ff2-2175-4747-a154-d803bb7b54f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:20:27.186101 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:27.182718    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:28.565518 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:28.565421    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"]
Oct 30 13:20:28.566104 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:28.565647    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp" podUID="20cef20e-1c1b-4d57-b260-019f2de104a1" containerName="istio-proxy" containerID="cri-o://0815d1389d8b9b41e98b4b4bb3b047315250308667ec1c516be9e83a569ffea6" gracePeriod=30
Oct 30 13:20:28.631568 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:28.631531    4544 generic.go:356] "Generic (PLEG): container finished" podID="ed9d8ff2-2175-4747-a154-d803bb7b54f8" containerID="f856dbfc8cb97f7624945ddbe1fa45919223e6a1bc2abb87c9888021dfeee483" exitCode=0
Oct 30 13:20:28.631865 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:28.631841    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d" event={"ID":"ed9d8ff2-2175-4747-a154-d803bb7b54f8","Type":"ContainerDied","Data":"f856dbfc8cb97f7624945ddbe1fa45919223e6a1bc2abb87c9888021dfeee483"}
Oct 30 13:20:28.632725 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:28.632703    4544 scope.go:122] "RemoveContainer" containerID="f856dbfc8cb97f7624945ddbe1fa45919223e6a1bc2abb87c9888021dfeee483"
Oct 30 13:20:28.638239 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:28.638165    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:20:28.638598 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:28.638559    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:20:28.638732 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:28.638626    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:20:28.754172 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:28.754133    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7"]
Oct 30 13:20:28.754534 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:28.754353    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7" podUID="040d15f4-178f-41c8-8932-80bdfcfb6780" containerName="maas-api" containerID="cri-o://22b47f8343f1b706ff113d7a6a56ca27fcfec0bd651d42ddf36ab8b4b54fb156" gracePeriod=30
Oct 30 13:20:28.866559 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:28.866524    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:29.043708 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:20:29.043663    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:20:29.327671 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:29.327634    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"]
Oct 30 13:20:29.327893 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:29.327709    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"]
Oct 30 13:20:29.327963 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:29.327891    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m" podUID="3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf" containerName="istio-proxy" containerID="cri-o://e72b6fd6e3230ccb48f7e93bfb1bb132da0338be2ca04492b8bf7f9a42306f51" gracePeriod=30
Oct 30 13:20:29.328175 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:29.328149    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s" podUID="9c24a948-3812-4568-a7a0-e58641a7427f" containerName="istio-proxy" containerID="cri-o://457f928f5e6b51b4d2611a04116db827bc438cbe60f1ad787f08093538e028f6" gracePeriod=30
Oct 30 13:20:29.343017 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:29.342978    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7"
Oct 30 13:20:29.362311 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:29.362266    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"]
Oct 30 13:20:29.362529 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:29.362481    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt" podUID="f8eb30f8-aa71-4121-9542-4f65aff15344" containerName="istio-proxy" containerID="cri-o://31a2df703c050cfa05e2788849a61fd6ec015897fc2a483ad33723fdac46d76e" gracePeriod=30
Oct 30 13:20:29.412956 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:29.412913    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/040d15f4-178f-41c8-8932-80bdfcfb6780-metrics-certs\" (UniqueName: \"kubernetes.io/secret/040d15f4-178f-41c8-8932-80bdfcfb6780-metrics-certs\") pod \"040d15f4-178f-41c8-8932-80bdfcfb6780\" (UID: \"040d15f4-178f-41c8-8932-80bdfcfb6780\") "
Oct 30 13:20:29.413352 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:29.413320    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/040d15f4-178f-41c8-8932-80bdfcfb6780-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/040d15f4-178f-41c8-8932-80bdfcfb6780-maas-api-tls\") pod \"040d15f4-178f-41c8-8932-80bdfcfb6780\" (UID: \"040d15f4-178f-41c8-8932-80bdfcfb6780\") "
Oct 30 13:20:29.413579 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:29.413555    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/040d15f4-178f-41c8-8932-80bdfcfb6780-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/040d15f4-178f-41c8-8932-80bdfcfb6780-openshift-service-ca\") pod \"040d15f4-178f-41c8-8932-80bdfcfb6780\" (UID: \"040d15f4-178f-41c8-8932-80bdfcfb6780\") "
Oct 30 13:20:29.413711 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:29.413690    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/040d15f4-178f-41c8-8932-80bdfcfb6780-kube-api-access-6qtrf\" (UniqueName: \"kubernetes.io/projected/040d15f4-178f-41c8-8932-80bdfcfb6780-kube-api-access-6qtrf\") pod \"040d15f4-178f-41c8-8932-80bdfcfb6780\" (UID: \"040d15f4-178f-41c8-8932-80bdfcfb6780\") "
Oct 30 13:20:29.455990 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:29.455948    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/040d15f4-178f-41c8-8932-80bdfcfb6780-openshift-service-ca" pod "040d15f4-178f-41c8-8932-80bdfcfb6780" (UID: "040d15f4-178f-41c8-8932-80bdfcfb6780"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:20:29.484268 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:29.478428    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:29.484268 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:29.478612    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/040d15f4-178f-41c8-8932-80bdfcfb6780-maas-api-tls" pod "040d15f4-178f-41c8-8932-80bdfcfb6780" (UID: "040d15f4-178f-41c8-8932-80bdfcfb6780"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:20:29.484268 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:29.482589    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/040d15f4-178f-41c8-8932-80bdfcfb6780-kube-api-access-6qtrf" pod "040d15f4-178f-41c8-8932-80bdfcfb6780" (UID: "040d15f4-178f-41c8-8932-80bdfcfb6780"). InnerVolumeSpecName "kube-api-access-6qtrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:29.495246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:29.491703    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/040d15f4-178f-41c8-8932-80bdfcfb6780-metrics-certs" pod "040d15f4-178f-41c8-8932-80bdfcfb6780" (UID: "040d15f4-178f-41c8-8932-80bdfcfb6780"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:20:29.541472 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:29.541434    4544 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/040d15f4-178f-41c8-8932-80bdfcfb6780-metrics-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:29.541472 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:29.541473    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/040d15f4-178f-41c8-8932-80bdfcfb6780-maas-api-tls\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:29.541801 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:29.541487    4544 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/040d15f4-178f-41c8-8932-80bdfcfb6780-openshift-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:29.541801 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:29.541502    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qtrf\" (UniqueName: \"kubernetes.io/projected/040d15f4-178f-41c8-8932-80bdfcfb6780-kube-api-access-6qtrf\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:29.657076 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:29.654905    4544 generic.go:356] "Generic (PLEG): container finished" podID="040d15f4-178f-41c8-8932-80bdfcfb6780" containerID="22b47f8343f1b706ff113d7a6a56ca27fcfec0bd651d42ddf36ab8b4b54fb156" exitCode=0
Oct 30 13:20:29.657076 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:29.655051    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7" event={"ID":"040d15f4-178f-41c8-8932-80bdfcfb6780","Type":"ContainerDied","Data":"22b47f8343f1b706ff113d7a6a56ca27fcfec0bd651d42ddf36ab8b4b54fb156"}
Oct 30 13:20:29.657076 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:29.655110    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7" event={"ID":"040d15f4-178f-41c8-8932-80bdfcfb6780","Type":"ContainerDied","Data":"62ea5b4469b587425cb13de12465311532ec9e43a1e56e71b85661835230a434"}
Oct 30 13:20:29.657076 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:29.655149    4544 scope.go:122] "RemoveContainer" containerID="22b47f8343f1b706ff113d7a6a56ca27fcfec0bd651d42ddf36ab8b4b54fb156"
Oct 30 13:20:29.657076 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:29.655379    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7"
Oct 30 13:20:29.664073 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:29.664035    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d" event={"ID":"ed9d8ff2-2175-4747-a154-d803bb7b54f8","Type":"ContainerStarted","Data":"9e4dac145ea7dee8fe37327650f8acaa27afe024af64c862036c61dc36b57ac5"}
Oct 30 13:20:29.686157 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:29.686135    4544 scope.go:122] "RemoveContainer" containerID="22b47f8343f1b706ff113d7a6a56ca27fcfec0bd651d42ddf36ab8b4b54fb156"
Oct 30 13:20:29.686789 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:20:29.686764    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22b47f8343f1b706ff113d7a6a56ca27fcfec0bd651d42ddf36ab8b4b54fb156\": container with ID starting with 22b47f8343f1b706ff113d7a6a56ca27fcfec0bd651d42ddf36ab8b4b54fb156 not found: ID does not exist" containerID="22b47f8343f1b706ff113d7a6a56ca27fcfec0bd651d42ddf36ab8b4b54fb156"
Oct 30 13:20:29.686968 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:29.686944    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22b47f8343f1b706ff113d7a6a56ca27fcfec0bd651d42ddf36ab8b4b54fb156"} err="failed to get container status \"22b47f8343f1b706ff113d7a6a56ca27fcfec0bd651d42ddf36ab8b4b54fb156\": rpc error: code = NotFound desc = could not find container \"22b47f8343f1b706ff113d7a6a56ca27fcfec0bd651d42ddf36ab8b4b54fb156\": container with ID starting with 22b47f8343f1b706ff113d7a6a56ca27fcfec0bd651d42ddf36ab8b4b54fb156 not found: ID does not exist"
Oct 30 13:20:29.718664 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:29.718615    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:29.882257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:29.880051    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7"]
Oct 30 13:20:29.903314 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:29.894298    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7"]
Oct 30 13:20:30.009185 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:30.009099    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:30.187243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:30.175154    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:30.237342 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:30.237300    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a3e01-p8hjg"]
Oct 30 13:20:30.248498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:30.248408    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a3e01-p8hjg"]
Oct 30 13:20:30.400764 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:30.400724    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:30.445159 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:30.445071    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:20:30.445159 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:30.445094    4544 patch_prober.go:29] interesting pod/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 13:20:30.445159 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:30.445136    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d" podUID="ed9d8ff2-2175-4747-a154-d803bb7b54f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 13:20:30.745666 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:30.743166    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:30.939243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:30.939110    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:31.158454 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:31.158411    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="040d15f4-178f-41c8-8932-80bdfcfb6780" path="/var/lib/kubelet/pods/040d15f4-178f-41c8-8932-80bdfcfb6780/volumes"
Oct 30 13:20:31.159779 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:31.159374    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6" path="/var/lib/kubelet/pods/8b3a8cb5-9d41-4cac-a9d2-226c5d76a9a6/volumes"
Oct 30 13:20:31.160712 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:31.160043    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq" podUID="2d6dcafc-5203-4f02-b8b8-c06fc9eb9353" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:20:31.445795 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:31.445698    4544 patch_prober.go:29] interesting pod/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 13:20:31.445795 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:31.445766    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d" podUID="ed9d8ff2-2175-4747-a154-d803bb7b54f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 13:20:31.545072 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:31.545038    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a8ee1-nmkb2"]
Oct 30 13:20:31.545948 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:31.545917    4544 cpu_manager.go:414] "containerMap: removing container" podUID="040d15f4-178f-41c8-8932-80bdfcfb6780" containerName="maas-api"
Oct 30 13:20:31.545948 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:31.545948    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="040d15f4-178f-41c8-8932-80bdfcfb6780" containerName="maas-api"
Oct 30 13:20:31.546078 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:31.545983    4544 cpu_manager.go:414] "containerMap: removing container" podUID="983f6389-65d1-4dea-b68b-4fc4d23098fe" containerName="revoke-keys"
Oct 30 13:20:31.546078 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:31.545989    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="983f6389-65d1-4dea-b68b-4fc4d23098fe" containerName="revoke-keys"
Oct 30 13:20:31.546144 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:31.546102    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="040d15f4-178f-41c8-8932-80bdfcfb6780" containerName="maas-api"
Oct 30 13:20:31.546144 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:31.546118    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="983f6389-65d1-4dea-b68b-4fc4d23098fe" containerName="revoke-keys"
Oct 30 13:20:31.552404 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:31.552379    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a8ee1-nmkb2"
Oct 30 13:20:31.573412 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:31.573374    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:20:31.580011 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:31.574618    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a8ee1-nmkb2"]
Oct 30 13:20:31.580269 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:31.580245    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:20:31.661713 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:31.661638    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f3613a9-0816-4310-986d-43dbd8b16288-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4a8ee1-nmkb2\" (UID: \"7f3613a9-0816-4310-986d-43dbd8b16288\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a8ee1-nmkb2"
Oct 30 13:20:31.661713 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:31.661711    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5x2t\" (UniqueName: \"kubernetes.io/projected/7f3613a9-0816-4310-986d-43dbd8b16288-kube-api-access-r5x2t\") pod \"maas-api-revoke-keys-e2e-inf-a-4a8ee1-nmkb2\" (UID: \"7f3613a9-0816-4310-986d-43dbd8b16288\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a8ee1-nmkb2"
Oct 30 13:20:31.662000 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:31.661786    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f3613a9-0816-4310-986d-43dbd8b16288-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4a8ee1-nmkb2\" (UID: \"7f3613a9-0816-4310-986d-43dbd8b16288\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a8ee1-nmkb2"
Oct 30 13:20:31.763692 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:31.763607    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f3613a9-0816-4310-986d-43dbd8b16288-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4a8ee1-nmkb2\" (UID: \"7f3613a9-0816-4310-986d-43dbd8b16288\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a8ee1-nmkb2"
Oct 30 13:20:31.763692 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:31.763682    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5x2t\" (UniqueName: \"kubernetes.io/projected/7f3613a9-0816-4310-986d-43dbd8b16288-kube-api-access-r5x2t\") pod \"maas-api-revoke-keys-e2e-inf-a-4a8ee1-nmkb2\" (UID: \"7f3613a9-0816-4310-986d-43dbd8b16288\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a8ee1-nmkb2"
Oct 30 13:20:31.764125 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:31.763743    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f3613a9-0816-4310-986d-43dbd8b16288-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4a8ee1-nmkb2\" (UID: \"7f3613a9-0816-4310-986d-43dbd8b16288\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a8ee1-nmkb2"
Oct 30 13:20:31.764419 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:31.764390    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f3613a9-0816-4310-986d-43dbd8b16288-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4a8ee1-nmkb2\" (UID: \"7f3613a9-0816-4310-986d-43dbd8b16288\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a8ee1-nmkb2"
Oct 30 13:20:31.766173 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:31.766153    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f3613a9-0816-4310-986d-43dbd8b16288-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4a8ee1-nmkb2\" (UID: \"7f3613a9-0816-4310-986d-43dbd8b16288\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a8ee1-nmkb2"
Oct 30 13:20:31.787190 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:31.787159    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5x2t\" (UniqueName: \"kubernetes.io/projected/7f3613a9-0816-4310-986d-43dbd8b16288-kube-api-access-r5x2t\") pod \"maas-api-revoke-keys-e2e-inf-a-4a8ee1-nmkb2\" (UID: \"7f3613a9-0816-4310-986d-43dbd8b16288\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a8ee1-nmkb2"
Oct 30 13:20:31.872964 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:31.872929    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a8ee1-nmkb2"
Oct 30 13:20:32.021619 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:32.021189    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2a3e01-689797fdf5-s9rc7" podUID="040d15f4-178f-41c8-8932-80bdfcfb6780" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.185:8443/health\": context deadline exceeded"
Oct 30 13:20:32.243774 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:32.243745    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a8ee1-nmkb2"]
Oct 30 13:20:32.245408 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:20:32.245379    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f3613a9_0816_4310_986d_43dbd8b16288.slice/crio-8db86a0a82fbbc76821b5e064bed04660333817becd6927814ccb88ae9348684 WatchSource:0}: Error finding container 8db86a0a82fbbc76821b5e064bed04660333817becd6927814ccb88ae9348684: Status 404 returned error can't find the container with id 8db86a0a82fbbc76821b5e064bed04660333817becd6927814ccb88ae9348684
Oct 30 13:20:32.446212 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:32.446179    4544 patch_prober.go:29] interesting pod/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 13:20:32.446393 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:32.446242    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d" podUID="ed9d8ff2-2175-4747-a154-d803bb7b54f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 13:20:32.697449 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:32.697412    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a8ee1-nmkb2" event={"ID":"7f3613a9-0816-4310-986d-43dbd8b16288","Type":"ContainerStarted","Data":"8db86a0a82fbbc76821b5e064bed04660333817becd6927814ccb88ae9348684"}
Oct 30 13:20:32.845182 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:32.845129    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:33.006145 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:33.006057    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:33.443318 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:33.440986    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd"]
Oct 30 13:20:33.447038 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:33.447003    4544 patch_prober.go:29] interesting pod/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 13:20:33.447325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:33.447296    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d" podUID="ed9d8ff2-2175-4747-a154-d803bb7b54f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 13:20:33.447797 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:33.447658    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd"]
Oct 30 13:20:33.447954 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:33.447736    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd"
Oct 30 13:20:33.455316 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:33.455179    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7mx6s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:20:33.455650 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:33.455623    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s" podUID="9c24a948-3812-4568-a7a0-e58641a7427f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:20:33.480275 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:33.480208    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f3338e-dockercfg-gjrdb\""
Oct 30 13:20:33.484110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:33.483860    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f3338e\""
Oct 30 13:20:33.538693 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:33.538397    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbrbt\" (UniqueName: \"kubernetes.io/projected/05c58d32-2ee6-4197-a19c-8431668ca3eb-kube-api-access-qbrbt\") pod \"payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd\" (UID: \"05c58d32-2ee6-4197-a19c-8431668ca3eb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd"
Oct 30 13:20:33.538693 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:33.538599    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c58d32-2ee6-4197-a19c-8431668ca3eb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd\" (UID: \"05c58d32-2ee6-4197-a19c-8431668ca3eb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd"
Oct 30 13:20:33.641518 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:33.641477    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbrbt\" (UniqueName: \"kubernetes.io/projected/05c58d32-2ee6-4197-a19c-8431668ca3eb-kube-api-access-qbrbt\") pod \"payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd\" (UID: \"05c58d32-2ee6-4197-a19c-8431668ca3eb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd"
Oct 30 13:20:33.641766 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:33.641619    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c58d32-2ee6-4197-a19c-8431668ca3eb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd\" (UID: \"05c58d32-2ee6-4197-a19c-8431668ca3eb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd"
Oct 30 13:20:33.642398 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:33.642375    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c58d32-2ee6-4197-a19c-8431668ca3eb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd\" (UID: \"05c58d32-2ee6-4197-a19c-8431668ca3eb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd"
Oct 30 13:20:33.665111 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:33.665074    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbrbt\" (UniqueName: \"kubernetes.io/projected/05c58d32-2ee6-4197-a19c-8431668ca3eb-kube-api-access-qbrbt\") pod \"payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd\" (UID: \"05c58d32-2ee6-4197-a19c-8431668ca3eb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd"
Oct 30 13:20:33.710347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:33.710253    4544 generic.go:356] "Generic (PLEG): container finished" podID="7f3613a9-0816-4310-986d-43dbd8b16288" containerID="aaaeb82775ca79364dd59bb6104b50d7f433b28a057cbc9620133b3b1053a7e1" exitCode=0
Oct 30 13:20:33.710347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:33.710324    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a8ee1-nmkb2" event={"ID":"7f3613a9-0816-4310-986d-43dbd8b16288","Type":"ContainerDied","Data":"aaaeb82775ca79364dd59bb6104b50d7f433b28a057cbc9620133b3b1053a7e1"}
Oct 30 13:20:33.720247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:33.712294    4544 generic.go:356] "Generic (PLEG): container finished" podID="20cef20e-1c1b-4d57-b260-019f2de104a1" containerID="0815d1389d8b9b41e98b4b4bb3b047315250308667ec1c516be9e83a569ffea6" exitCode=0
Oct 30 13:20:33.720247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:33.712391    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp" event={"ID":"20cef20e-1c1b-4d57-b260-019f2de104a1","Type":"ContainerDied","Data":"0815d1389d8b9b41e98b4b4bb3b047315250308667ec1c516be9e83a569ffea6"}
Oct 30 13:20:33.791018 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:33.790967    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd"
Oct 30 13:20:33.934698 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:33.934284    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:20:34.052046 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.051945    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20cef20e-1c1b-4d57-b260-019f2de104a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20cef20e-1c1b-4d57-b260-019f2de104a1-istio-podinfo\") pod \"20cef20e-1c1b-4d57-b260-019f2de104a1\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") "
Oct 30 13:20:34.052295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.052071    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20cef20e-1c1b-4d57-b260-019f2de104a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20cef20e-1c1b-4d57-b260-019f2de104a1-istiod-ca-cert\") pod \"20cef20e-1c1b-4d57-b260-019f2de104a1\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") "
Oct 30 13:20:34.052295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.052122    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-credential-socket\") pod \"20cef20e-1c1b-4d57-b260-019f2de104a1\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") "
Oct 30 13:20:34.052295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.052177    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20cef20e-1c1b-4d57-b260-019f2de104a1-istio-token\" (UniqueName: \"kubernetes.io/projected/20cef20e-1c1b-4d57-b260-019f2de104a1-istio-token\") pod \"20cef20e-1c1b-4d57-b260-019f2de104a1\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") "
Oct 30 13:20:34.052504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.052344    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20cef20e-1c1b-4d57-b260-019f2de104a1-kube-api-access-56frj\" (UniqueName: \"kubernetes.io/projected/20cef20e-1c1b-4d57-b260-019f2de104a1-kube-api-access-56frj\") pod \"20cef20e-1c1b-4d57-b260-019f2de104a1\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") "
Oct 30 13:20:34.052504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.052382    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-workload-certs\") pod \"20cef20e-1c1b-4d57-b260-019f2de104a1\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") "
Oct 30 13:20:34.052504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.052449    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-workload-socket\") pod \"20cef20e-1c1b-4d57-b260-019f2de104a1\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") "
Oct 30 13:20:34.052688 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.052504    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-credential-socket" pod "20cef20e-1c1b-4d57-b260-019f2de104a1" (UID: "20cef20e-1c1b-4d57-b260-019f2de104a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:34.052688 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.052534    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-istio-envoy\") pod \"20cef20e-1c1b-4d57-b260-019f2de104a1\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") "
Oct 30 13:20:34.052688 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.052662    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-istio-data\") pod \"20cef20e-1c1b-4d57-b260-019f2de104a1\" (UID: \"20cef20e-1c1b-4d57-b260-019f2de104a1\") "
Oct 30 13:20:34.052688 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.052662    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20cef20e-1c1b-4d57-b260-019f2de104a1-istiod-ca-cert" pod "20cef20e-1c1b-4d57-b260-019f2de104a1" (UID: "20cef20e-1c1b-4d57-b260-019f2de104a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:20:34.052915 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.052746    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-workload-socket" pod "20cef20e-1c1b-4d57-b260-019f2de104a1" (UID: "20cef20e-1c1b-4d57-b260-019f2de104a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:34.053046 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.053021    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-istio-data" pod "20cef20e-1c1b-4d57-b260-019f2de104a1" (UID: "20cef20e-1c1b-4d57-b260-019f2de104a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:34.053162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.053111    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-workload-certs" pod "20cef20e-1c1b-4d57-b260-019f2de104a1" (UID: "20cef20e-1c1b-4d57-b260-019f2de104a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:34.053668 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.053520    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:34.053668 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.053547    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:34.053668 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.053561    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:34.053668 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.053574    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20cef20e-1c1b-4d57-b260-019f2de104a1-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:34.053668 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.053586    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:34.055022 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.054994    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-istio-envoy" pod "20cef20e-1c1b-4d57-b260-019f2de104a1" (UID: "20cef20e-1c1b-4d57-b260-019f2de104a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:34.055022 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.055008    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20cef20e-1c1b-4d57-b260-019f2de104a1-kube-api-access-56frj" pod "20cef20e-1c1b-4d57-b260-019f2de104a1" (UID: "20cef20e-1c1b-4d57-b260-019f2de104a1"). InnerVolumeSpecName "kube-api-access-56frj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:34.056419 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.056398    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20cef20e-1c1b-4d57-b260-019f2de104a1-istio-token" pod "20cef20e-1c1b-4d57-b260-019f2de104a1" (UID: "20cef20e-1c1b-4d57-b260-019f2de104a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:34.056537 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.056429    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20cef20e-1c1b-4d57-b260-019f2de104a1-istio-podinfo" pod "20cef20e-1c1b-4d57-b260-019f2de104a1" (UID: "20cef20e-1c1b-4d57-b260-019f2de104a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:20:34.155536 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.155497    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20cef20e-1c1b-4d57-b260-019f2de104a1-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:34.155536 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.155533    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20cef20e-1c1b-4d57-b260-019f2de104a1-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:34.155536 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.155545    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56frj\" (UniqueName: \"kubernetes.io/projected/20cef20e-1c1b-4d57-b260-019f2de104a1-kube-api-access-56frj\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:34.155890 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.155558    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20cef20e-1c1b-4d57-b260-019f2de104a1-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:34.162565 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.162534    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd"]
Oct 30 13:20:34.163939 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:20:34.163916    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c58d32_2ee6_4197_a19c_8431668ca3eb.slice/crio-6b92ab558300e3fa64865863d44e2a3f5ecafa3f871d87b021032055c8db3c39 WatchSource:0}: Error finding container 6b92ab558300e3fa64865863d44e2a3f5ecafa3f871d87b021032055c8db3c39: Status 404 returned error can't find the container with id 6b92ab558300e3fa64865863d44e2a3f5ecafa3f871d87b021032055c8db3c39
Oct 30 13:20:34.446045 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.446007    4544 patch_prober.go:29] interesting pod/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 13:20:34.446232 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.446067    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d" podUID="ed9d8ff2-2175-4747-a154-d803bb7b54f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 13:20:34.760732 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.760705    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"
Oct 30 13:20:34.761037 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.760707    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp" event={"ID":"20cef20e-1c1b-4d57-b260-019f2de104a1","Type":"ContainerDied","Data":"5754c51bb6a2522144bcd347ef34b240b0bde3a9f0ceb8c543ab3a55a1921e1f"}
Oct 30 13:20:34.761102 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.761078    4544 scope.go:122] "RemoveContainer" containerID="0815d1389d8b9b41e98b4b4bb3b047315250308667ec1c516be9e83a569ffea6"
Oct 30 13:20:34.766951 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.765136    4544 generic.go:356] "Generic (PLEG): container finished" podID="3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf" containerID="e72b6fd6e3230ccb48f7e93bfb1bb132da0338be2ca04492b8bf7f9a42306f51" exitCode=0
Oct 30 13:20:34.766951 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.765594    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m" event={"ID":"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf","Type":"ContainerDied","Data":"e72b6fd6e3230ccb48f7e93bfb1bb132da0338be2ca04492b8bf7f9a42306f51"}
Oct 30 13:20:34.776282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.769913    4544 generic.go:356] "Generic (PLEG): container finished" podID="f8eb30f8-aa71-4121-9542-4f65aff15344" containerID="31a2df703c050cfa05e2788849a61fd6ec015897fc2a483ad33723fdac46d76e" exitCode=0
Oct 30 13:20:34.776282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.769989    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt" event={"ID":"f8eb30f8-aa71-4121-9542-4f65aff15344","Type":"ContainerDied","Data":"31a2df703c050cfa05e2788849a61fd6ec015897fc2a483ad33723fdac46d76e"}
Oct 30 13:20:34.776282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.771913    4544 generic.go:356] "Generic (PLEG): container finished" podID="9c24a948-3812-4568-a7a0-e58641a7427f" containerID="457f928f5e6b51b4d2611a04116db827bc438cbe60f1ad787f08093538e028f6" exitCode=0
Oct 30 13:20:34.776282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.771970    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s" event={"ID":"9c24a948-3812-4568-a7a0-e58641a7427f","Type":"ContainerDied","Data":"457f928f5e6b51b4d2611a04116db827bc438cbe60f1ad787f08093538e028f6"}
Oct 30 13:20:34.776282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.773783    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd" event={"ID":"05c58d32-2ee6-4197-a19c-8431668ca3eb","Type":"ContainerStarted","Data":"6ed969f4bc2fa0939d9d0fe8be01690b2aa3ba9a9ad5fb6c0fb7c950c0ceed7a"}
Oct 30 13:20:34.776282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.773828    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd" event={"ID":"05c58d32-2ee6-4197-a19c-8431668ca3eb","Type":"ContainerStarted","Data":"6b92ab558300e3fa64865863d44e2a3f5ecafa3f871d87b021032055c8db3c39"}
Oct 30 13:20:34.776282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.773886    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd"
Oct 30 13:20:34.860251 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.851787    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd" podStartSLOduration=1.851763931 podStartE2EDuration="1.851763931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:34.844806559 +0000 UTC m=+2129.891940512" watchObservedRunningTime="2026-10-30 13:20:34.851763931 +0000 UTC m=+2129.898897884"
Oct 30 13:20:34.904493 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.903491    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 13:20:34.970544 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.970428    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 13:20:34.978384 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.978296    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"]
Oct 30 13:20:34.985257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.979140    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c24a948-3812-4568-a7a0-e58641a7427f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c24a948-3812-4568-a7a0-e58641a7427f-istio-podinfo\") pod \"9c24a948-3812-4568-a7a0-e58641a7427f\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") "
Oct 30 13:20:34.985257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.979440    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c24a948-3812-4568-a7a0-e58641a7427f-istio-token\" (UniqueName: \"kubernetes.io/projected/9c24a948-3812-4568-a7a0-e58641a7427f-istio-token\") pod \"9c24a948-3812-4568-a7a0-e58641a7427f\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") "
Oct 30 13:20:34.985257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.979488    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-workload-certs\") pod \"9c24a948-3812-4568-a7a0-e58641a7427f\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") "
Oct 30 13:20:34.985257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.979547    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c24a948-3812-4568-a7a0-e58641a7427f-kube-api-access-zghds\" (UniqueName: \"kubernetes.io/projected/9c24a948-3812-4568-a7a0-e58641a7427f-kube-api-access-zghds\") pod \"9c24a948-3812-4568-a7a0-e58641a7427f\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") "
Oct 30 13:20:34.985257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.979634    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c24a948-3812-4568-a7a0-e58641a7427f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c24a948-3812-4568-a7a0-e58641a7427f-istiod-ca-cert\") pod \"9c24a948-3812-4568-a7a0-e58641a7427f\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") "
Oct 30 13:20:34.985257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.979718    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-istio-data\") pod \"9c24a948-3812-4568-a7a0-e58641a7427f\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") "
Oct 30 13:20:34.985257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.979782    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-credential-socket\") pod \"9c24a948-3812-4568-a7a0-e58641a7427f\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") "
Oct 30 13:20:34.985257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.979818    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-istio-envoy\") pod \"9c24a948-3812-4568-a7a0-e58641a7427f\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") "
Oct 30 13:20:34.985257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.979877    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-workload-socket\") pod \"9c24a948-3812-4568-a7a0-e58641a7427f\" (UID: \"9c24a948-3812-4568-a7a0-e58641a7427f\") "
Oct 30 13:20:34.986870 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.986417    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-workload-socket" pod "9c24a948-3812-4568-a7a0-e58641a7427f" (UID: "9c24a948-3812-4568-a7a0-e58641a7427f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:35.000636 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:34.999800    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-credential-socket" pod "9c24a948-3812-4568-a7a0-e58641a7427f" (UID: "9c24a948-3812-4568-a7a0-e58641a7427f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:35.000636 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.000043    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-istio-data" pod "9c24a948-3812-4568-a7a0-e58641a7427f" (UID: "9c24a948-3812-4568-a7a0-e58641a7427f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:35.000636 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.000352    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c24a948-3812-4568-a7a0-e58641a7427f-istio-token" pod "9c24a948-3812-4568-a7a0-e58641a7427f" (UID: "9c24a948-3812-4568-a7a0-e58641a7427f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:35.009716 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.009670    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-workload-certs" pod "9c24a948-3812-4568-a7a0-e58641a7427f" (UID: "9c24a948-3812-4568-a7a0-e58641a7427f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:35.023877 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.023810    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c24a948-3812-4568-a7a0-e58641a7427f-kube-api-access-zghds" pod "9c24a948-3812-4568-a7a0-e58641a7427f" (UID: "9c24a948-3812-4568-a7a0-e58641a7427f"). InnerVolumeSpecName "kube-api-access-zghds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:35.079522 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.079475    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c24a948-3812-4568-a7a0-e58641a7427f-istiod-ca-cert" pod "9c24a948-3812-4568-a7a0-e58641a7427f" (UID: "9c24a948-3812-4568-a7a0-e58641a7427f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:20:35.108434 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.108378    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:35.108658 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.108645    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:35.108745 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.108734    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:35.108829 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.108819    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c24a948-3812-4568-a7a0-e58641a7427f-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:35.108906 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.108896    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:35.108988 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.108978    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zghds\" (UniqueName: \"kubernetes.io/projected/9c24a948-3812-4568-a7a0-e58641a7427f-kube-api-access-zghds\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:35.109075 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.109065    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c24a948-3812-4568-a7a0-e58641a7427f-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:35.114624 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.114592    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-istio-envoy" pod "9c24a948-3812-4568-a7a0-e58641a7427f" (UID: "9c24a948-3812-4568-a7a0-e58641a7427f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:35.132387 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.132335    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c24a948-3812-4568-a7a0-e58641a7427f-istio-podinfo" pod "9c24a948-3812-4568-a7a0-e58641a7427f" (UID: "9c24a948-3812-4568-a7a0-e58641a7427f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:20:35.189413 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.189385    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp"]
Oct 30 13:20:35.189564 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.189537    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a8ee1-nmkb2"
Oct 30 13:20:35.194289 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.194265    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:20:35.209952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.209914    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-workload-certs\") pod \"f8eb30f8-aa71-4121-9542-4f65aff15344\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") "
Oct 30 13:20:35.210112 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.209973    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8eb30f8-aa71-4121-9542-4f65aff15344-istio-token\" (UniqueName: \"kubernetes.io/projected/f8eb30f8-aa71-4121-9542-4f65aff15344-istio-token\") pod \"f8eb30f8-aa71-4121-9542-4f65aff15344\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") "
Oct 30 13:20:35.210112 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.210002    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-credential-socket\") pod \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") "
Oct 30 13:20:35.210112 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.210028    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-istio-envoy\") pod \"f8eb30f8-aa71-4121-9542-4f65aff15344\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") "
Oct 30 13:20:35.210112 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.210087    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8eb30f8-aa71-4121-9542-4f65aff15344-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8eb30f8-aa71-4121-9542-4f65aff15344-istiod-ca-cert\") pod \"f8eb30f8-aa71-4121-9542-4f65aff15344\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") "
Oct 30 13:20:35.210424 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.210157    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-workload-certs\") pod \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") "
Oct 30 13:20:35.210424 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.210190    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istio-token\" (UniqueName: \"kubernetes.io/projected/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istio-token\") pod \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") "
Oct 30 13:20:35.210424 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.210258    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-istio-data\") pod \"f8eb30f8-aa71-4121-9542-4f65aff15344\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") "
Oct 30 13:20:35.210424 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.210290    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-credential-socket\") pod \"f8eb30f8-aa71-4121-9542-4f65aff15344\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") "
Oct 30 13:20:35.210424 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.210357    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-kube-api-access-xld8z\" (UniqueName: \"kubernetes.io/projected/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-kube-api-access-xld8z\") pod \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") "
Oct 30 13:20:35.210424 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.210409    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f3613a9-0816-4310-986d-43dbd8b16288-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f3613a9-0816-4310-986d-43dbd8b16288-maas-api-openshift-ca\") pod \"7f3613a9-0816-4310-986d-43dbd8b16288\" (UID: \"7f3613a9-0816-4310-986d-43dbd8b16288\") "
Oct 30 13:20:35.210736 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.210431    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-workload-socket\") pod \"f8eb30f8-aa71-4121-9542-4f65aff15344\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") "
Oct 30 13:20:35.210736 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.210474    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-workload-socket\") pod \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") "
Oct 30 13:20:35.210736 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.210534    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istio-data\") pod \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") "
Oct 30 13:20:35.211362 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.210575    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istio-podinfo\") pod \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") "
Oct 30 13:20:35.212034 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.211402    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f3613a9-0816-4310-986d-43dbd8b16288-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f3613a9-0816-4310-986d-43dbd8b16288-maas-api-xks-ca\") pod \"7f3613a9-0816-4310-986d-43dbd8b16288\" (UID: \"7f3613a9-0816-4310-986d-43dbd8b16288\") "
Oct 30 13:20:35.212170 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.212151    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istio-envoy\") pod \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") "
Oct 30 13:20:35.244936 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.220254    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8eb30f8-aa71-4121-9542-4f65aff15344-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8eb30f8-aa71-4121-9542-4f65aff15344-istio-podinfo\") pod \"f8eb30f8-aa71-4121-9542-4f65aff15344\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") "
Oct 30 13:20:35.244936 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.220368    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8eb30f8-aa71-4121-9542-4f65aff15344-kube-api-access-f2pdw\" (UniqueName: \"kubernetes.io/projected/f8eb30f8-aa71-4121-9542-4f65aff15344-kube-api-access-f2pdw\") pod \"f8eb30f8-aa71-4121-9542-4f65aff15344\" (UID: \"f8eb30f8-aa71-4121-9542-4f65aff15344\") "
Oct 30 13:20:35.244936 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.220421    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istiod-ca-cert\") pod \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\" (UID: \"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf\") "
Oct 30 13:20:35.244936 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.220499    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f3613a9-0816-4310-986d-43dbd8b16288-kube-api-access-r5x2t\" (UniqueName: \"kubernetes.io/projected/7f3613a9-0816-4310-986d-43dbd8b16288-kube-api-access-r5x2t\") pod \"7f3613a9-0816-4310-986d-43dbd8b16288\" (UID: \"7f3613a9-0816-4310-986d-43dbd8b16288\") "
Oct 30 13:20:35.244936 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.221531    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c24a948-3812-4568-a7a0-e58641a7427f-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:35.244936 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.221559    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c24a948-3812-4568-a7a0-e58641a7427f-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:35.245462 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.210925    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-workload-certs" pod "f8eb30f8-aa71-4121-9542-4f65aff15344" (UID: "f8eb30f8-aa71-4121-9542-4f65aff15344"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:35.245462 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.211183    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-istio-data" pod "f8eb30f8-aa71-4121-9542-4f65aff15344" (UID: "f8eb30f8-aa71-4121-9542-4f65aff15344"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:35.245462 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.211255    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f3613a9-0816-4310-986d-43dbd8b16288-maas-api-openshift-ca" pod "7f3613a9-0816-4310-986d-43dbd8b16288" (UID: "7f3613a9-0816-4310-986d-43dbd8b16288"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:20:35.245462 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.211488    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-credential-socket" pod "f8eb30f8-aa71-4121-9542-4f65aff15344" (UID: "f8eb30f8-aa71-4121-9542-4f65aff15344"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:35.245462 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.211997    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-credential-socket" pod "3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf" (UID: "3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:35.245462 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.212100    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-workload-socket" pod "f8eb30f8-aa71-4121-9542-4f65aff15344" (UID: "f8eb30f8-aa71-4121-9542-4f65aff15344"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:35.245462 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.212249    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-workload-socket" pod "3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf" (UID: "3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:35.245462 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.242378    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istio-envoy" pod "3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf" (UID: "3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:35.245462 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.242544    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8eb30f8-aa71-4121-9542-4f65aff15344-istio-token" pod "f8eb30f8-aa71-4121-9542-4f65aff15344" (UID: "f8eb30f8-aa71-4121-9542-4f65aff15344"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:35.245462 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.242764    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istiod-ca-cert" pod "3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf" (UID: "3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:20:35.246594 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.246001    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-workload-certs" pod "3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf" (UID: "3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:35.246594 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.246309    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f3613a9-0816-4310-986d-43dbd8b16288-kube-api-access-r5x2t" pod "7f3613a9-0816-4310-986d-43dbd8b16288" (UID: "7f3613a9-0816-4310-986d-43dbd8b16288"). InnerVolumeSpecName "kube-api-access-r5x2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:35.248467 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.246926    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8eb30f8-aa71-4121-9542-4f65aff15344-istiod-ca-cert" pod "f8eb30f8-aa71-4121-9542-4f65aff15344" (UID: "f8eb30f8-aa71-4121-9542-4f65aff15344"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:20:35.248467 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.246985    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istio-data" pod "3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf" (UID: "3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:35.248467 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.247942    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istio-podinfo" pod "3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf" (UID: "3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:20:35.248467 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.248014    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f3613a9-0816-4310-986d-43dbd8b16288-maas-api-xks-ca" pod "7f3613a9-0816-4310-986d-43dbd8b16288" (UID: "7f3613a9-0816-4310-986d-43dbd8b16288"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:20:35.248467 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.248113    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-kube-api-access-xld8z" pod "3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf" (UID: "3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf"). InnerVolumeSpecName "kube-api-access-xld8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:35.250539 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.250441    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8eb30f8-aa71-4121-9542-4f65aff15344-istio-podinfo" pod "f8eb30f8-aa71-4121-9542-4f65aff15344" (UID: "f8eb30f8-aa71-4121-9542-4f65aff15344"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:20:35.250943 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.250916    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-istio-envoy" pod "f8eb30f8-aa71-4121-9542-4f65aff15344" (UID: "f8eb30f8-aa71-4121-9542-4f65aff15344"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:35.251035 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.250916    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istio-token" pod "3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf" (UID: "3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:35.251431 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.251206    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8eb30f8-aa71-4121-9542-4f65aff15344-kube-api-access-f2pdw" pod "f8eb30f8-aa71-4121-9542-4f65aff15344" (UID: "f8eb30f8-aa71-4121-9542-4f65aff15344"). InnerVolumeSpecName "kube-api-access-f2pdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:35.334274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.330657    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:35.334274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.330699    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8eb30f8-aa71-4121-9542-4f65aff15344-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:35.334274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.330715    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2pdw\" (UniqueName: \"kubernetes.io/projected/f8eb30f8-aa71-4121-9542-4f65aff15344-kube-api-access-f2pdw\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:35.334274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.330733    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:35.334274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.330748    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5x2t\" (UniqueName: \"kubernetes.io/projected/7f3613a9-0816-4310-986d-43dbd8b16288-kube-api-access-r5x2t\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:35.334274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.330762    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:35.334274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.330777    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8eb30f8-aa71-4121-9542-4f65aff15344-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:35.334274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.330790    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:35.334274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.330803    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:35.334274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.330818    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8eb30f8-aa71-4121-9542-4f65aff15344-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:35.334274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.330831    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:35.334274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.330845    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:35.334274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.330858    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:35.334274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.330871    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:35.334274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.330885    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xld8z\" (UniqueName: \"kubernetes.io/projected/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-kube-api-access-xld8z\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:35.334274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.330900    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f3613a9-0816-4310-986d-43dbd8b16288-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:35.334274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.330914    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eb30f8-aa71-4121-9542-4f65aff15344-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:35.334274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.330927    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:35.334274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.330941    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:35.334274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.330956    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:35.334274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.330971    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f3613a9-0816-4310-986d-43dbd8b16288-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:35.364705 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.364674    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq"
Oct 30 13:20:35.432578 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.432547    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d6dcafc-5203-4f02-b8b8-c06fc9eb9353-kube-api-access-ds745\" (UniqueName: \"kubernetes.io/projected/2d6dcafc-5203-4f02-b8b8-c06fc9eb9353-kube-api-access-ds745\") pod \"2d6dcafc-5203-4f02-b8b8-c06fc9eb9353\" (UID: \"2d6dcafc-5203-4f02-b8b8-c06fc9eb9353\") "
Oct 30 13:20:35.432971 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.432951    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d6dcafc-5203-4f02-b8b8-c06fc9eb9353-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d6dcafc-5203-4f02-b8b8-c06fc9eb9353-plugins-config-volume\") pod \"2d6dcafc-5203-4f02-b8b8-c06fc9eb9353\" (UID: \"2d6dcafc-5203-4f02-b8b8-c06fc9eb9353\") "
Oct 30 13:20:35.433731 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.433377    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d6dcafc-5203-4f02-b8b8-c06fc9eb9353-plugins-config-volume" pod "2d6dcafc-5203-4f02-b8b8-c06fc9eb9353" (UID: "2d6dcafc-5203-4f02-b8b8-c06fc9eb9353"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:20:35.433731 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.433630    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d6dcafc-5203-4f02-b8b8-c06fc9eb9353-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:35.435446 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.435390    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d6dcafc-5203-4f02-b8b8-c06fc9eb9353-kube-api-access-ds745" pod "2d6dcafc-5203-4f02-b8b8-c06fc9eb9353" (UID: "2d6dcafc-5203-4f02-b8b8-c06fc9eb9353"). InnerVolumeSpecName "kube-api-access-ds745". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:35.445871 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.445840    4544 patch_prober.go:29] interesting pod/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 13:20:35.446018 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.445901    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d" podUID="ed9d8ff2-2175-4747-a154-d803bb7b54f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 13:20:35.461014 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.460865    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:35.548248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.535691    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds745\" (UniqueName: \"kubernetes.io/projected/2d6dcafc-5203-4f02-b8b8-c06fc9eb9353-kube-api-access-ds745\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:35.769638 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.769502    4544 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:20:35.769638 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.769591    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt" podUID="f8eb30f8-aa71-4121-9542-4f65aff15344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:20:35.785739 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.785707    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"
Oct 30 13:20:35.785921 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.785743    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m" event={"ID":"3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf","Type":"ContainerDied","Data":"aa6999a2e89b62387e2c6216f171cb1b5186db82aebe8be3c2297ede7cd6f5a0"}
Oct 30 13:20:35.785921 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.785809    4544 scope.go:122] "RemoveContainer" containerID="e72b6fd6e3230ccb48f7e93bfb1bb132da0338be2ca04492b8bf7f9a42306f51"
Oct 30 13:20:35.787627 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.787602    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"
Oct 30 13:20:35.787755 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.787629    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt" event={"ID":"f8eb30f8-aa71-4121-9542-4f65aff15344","Type":"ContainerDied","Data":"f792af01042fe795300890784ffaad82a0a9ecb96436fa07a16c85de4458a0a3"}
Oct 30 13:20:35.789204 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.789180    4544 generic.go:356] "Generic (PLEG): container finished" podID="2d6dcafc-5203-4f02-b8b8-c06fc9eb9353" containerID="fb2c21eff2c65186cc0647473c773103f43566a8891cc512c4e99893460de2d2" exitCode=0
Oct 30 13:20:35.789351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.789292    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq"
Oct 30 13:20:35.789351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.789326    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq" event={"ID":"2d6dcafc-5203-4f02-b8b8-c06fc9eb9353","Type":"ContainerDied","Data":"fb2c21eff2c65186cc0647473c773103f43566a8891cc512c4e99893460de2d2"}
Oct 30 13:20:35.789511 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.789388    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq" event={"ID":"2d6dcafc-5203-4f02-b8b8-c06fc9eb9353","Type":"ContainerDied","Data":"18b1180297e4524fe065a0904f990da7c3c4248449bb85c6e03a4699cc4cd4dd"}
Oct 30 13:20:35.790964 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.790945    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a8ee1-nmkb2"
Oct 30 13:20:35.791118 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.790979    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a8ee1-nmkb2" event={"ID":"7f3613a9-0816-4310-986d-43dbd8b16288","Type":"ContainerDied","Data":"8db86a0a82fbbc76821b5e064bed04660333817becd6927814ccb88ae9348684"}
Oct 30 13:20:35.791118 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.791014    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8db86a0a82fbbc76821b5e064bed04660333817becd6927814ccb88ae9348684"
Oct 30 13:20:35.796613 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.796593    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"
Oct 30 13:20:35.796702 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.796630    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s" event={"ID":"9c24a948-3812-4568-a7a0-e58641a7427f","Type":"ContainerDied","Data":"80db4a4a138695eaf976a0507c460dd7c72e140144dde65d20d89c4c276020d2"}
Oct 30 13:20:35.802056 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.801959    4544 scope.go:122] "RemoveContainer" containerID="31a2df703c050cfa05e2788849a61fd6ec015897fc2a483ad33723fdac46d76e"
Oct 30 13:20:35.813776 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.813751    4544 scope.go:122] "RemoveContainer" containerID="fb2c21eff2c65186cc0647473c773103f43566a8891cc512c4e99893460de2d2"
Oct 30 13:20:35.828210 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.828183    4544 scope.go:122] "RemoveContainer" containerID="fb2c21eff2c65186cc0647473c773103f43566a8891cc512c4e99893460de2d2"
Oct 30 13:20:35.828635 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:20:35.828613    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb2c21eff2c65186cc0647473c773103f43566a8891cc512c4e99893460de2d2\": container with ID starting with fb2c21eff2c65186cc0647473c773103f43566a8891cc512c4e99893460de2d2 not found: ID does not exist" containerID="fb2c21eff2c65186cc0647473c773103f43566a8891cc512c4e99893460de2d2"
Oct 30 13:20:35.828717 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.828648    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb2c21eff2c65186cc0647473c773103f43566a8891cc512c4e99893460de2d2"} err="failed to get container status \"fb2c21eff2c65186cc0647473c773103f43566a8891cc512c4e99893460de2d2\": rpc error: code = NotFound desc = could not find container \"fb2c21eff2c65186cc0647473c773103f43566a8891cc512c4e99893460de2d2\": container with ID starting with fb2c21eff2c65186cc0647473c773103f43566a8891cc512c4e99893460de2d2 not found: ID does not exist"
Oct 30 13:20:35.828717 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.828675    4544 scope.go:122] "RemoveContainer" containerID="457f928f5e6b51b4d2611a04116db827bc438cbe60f1ad787f08093538e028f6"
Oct 30 13:20:35.878001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.877970    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"]
Oct 30 13:20:35.891018 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.890981    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m"]
Oct 30 13:20:35.956142 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.956108    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq"]
Oct 30 13:20:35.964399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:35.964367    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq"]
Oct 30 13:20:36.001761 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:36.001714    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"]
Oct 30 13:20:36.013021 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:36.012988    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt"]
Oct 30 13:20:36.055487 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:36.055387    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"]
Oct 30 13:20:36.062491 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:36.062460    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s"]
Oct 30 13:20:36.450703 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:36.450668    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:20:36.451210 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:36.451187    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:20:36.452018 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:36.452000    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:20:37.089566 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:37.089531    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20cef20e-1c1b-4d57-b260-019f2de104a1" path="/var/lib/kubelet/pods/20cef20e-1c1b-4d57-b260-019f2de104a1/volumes"
Oct 30 13:20:37.090082 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:37.090064    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d6dcafc-5203-4f02-b8b8-c06fc9eb9353" path="/var/lib/kubelet/pods/2d6dcafc-5203-4f02-b8b8-c06fc9eb9353/volumes"
Oct 30 13:20:37.090602 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:37.090586    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf" path="/var/lib/kubelet/pods/3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf/volumes"
Oct 30 13:20:37.091451 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:37.091430    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c24a948-3812-4568-a7a0-e58641a7427f" path="/var/lib/kubelet/pods/9c24a948-3812-4568-a7a0-e58641a7427f/volumes"
Oct 30 13:20:37.092000 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:37.091979    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8eb30f8-aa71-4121-9542-4f65aff15344" path="/var/lib/kubelet/pods/f8eb30f8-aa71-4121-9542-4f65aff15344/volumes"
Oct 30 13:20:37.496793 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:37.496690    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:37.805081 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:37.804994    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:38.129178 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:38.129148    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:38.380796 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:38.380709    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"]
Oct 30 13:20:38.380983 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:38.380956    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6" podUID="476505c3-3236-4789-846f-c13caa710b36" containerName="istio-proxy" containerID="cri-o://b0b7fe58678bf51c96306fd7784d2a86c1e89d59d71c9f0c54901f3be76c79f4" gracePeriod=30
Oct 30 13:20:38.423667 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:38.423623    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:38.882817 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:38.882780    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-649303-8kg5n"]
Oct 30 13:20:38.883817 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:38.883767    4544 cpu_manager.go:414] "containerMap: removing container" podUID="7f3613a9-0816-4310-986d-43dbd8b16288" containerName="revoke-keys"
Oct 30 13:20:38.883817 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:38.883791    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f3613a9-0816-4310-986d-43dbd8b16288" containerName="revoke-keys"
Oct 30 13:20:38.883817 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:38.883814    4544 cpu_manager.go:414] "containerMap: removing container" podUID="9c24a948-3812-4568-a7a0-e58641a7427f" containerName="istio-proxy"
Oct 30 13:20:38.883817 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:38.883822    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c24a948-3812-4568-a7a0-e58641a7427f" containerName="istio-proxy"
Oct 30 13:20:38.884038 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:38.883852    4544 cpu_manager.go:414] "containerMap: removing container" podUID="2d6dcafc-5203-4f02-b8b8-c06fc9eb9353" containerName="payload-pre-processing"
Oct 30 13:20:38.884038 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:38.883858    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d6dcafc-5203-4f02-b8b8-c06fc9eb9353" containerName="payload-pre-processing"
Oct 30 13:20:38.884038 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:38.883868    4544 cpu_manager.go:414] "containerMap: removing container" podUID="3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf" containerName="istio-proxy"
Oct 30 13:20:38.884038 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:38.883872    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf" containerName="istio-proxy"
Oct 30 13:20:38.884038 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:38.883888    4544 cpu_manager.go:414] "containerMap: removing container" podUID="20cef20e-1c1b-4d57-b260-019f2de104a1" containerName="istio-proxy"
Oct 30 13:20:38.884038 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:38.883895    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20cef20e-1c1b-4d57-b260-019f2de104a1" containerName="istio-proxy"
Oct 30 13:20:38.884038 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:38.883907    4544 cpu_manager.go:414] "containerMap: removing container" podUID="f8eb30f8-aa71-4121-9542-4f65aff15344" containerName="istio-proxy"
Oct 30 13:20:38.884038 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:38.883911    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8eb30f8-aa71-4121-9542-4f65aff15344" containerName="istio-proxy"
Oct 30 13:20:38.884294 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:38.884043    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8eb30f8-aa71-4121-9542-4f65aff15344" containerName="istio-proxy"
Oct 30 13:20:38.884294 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:38.884058    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f3613a9-0816-4310-986d-43dbd8b16288" containerName="revoke-keys"
Oct 30 13:20:38.884294 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:38.884070    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c24a948-3812-4568-a7a0-e58641a7427f" containerName="istio-proxy"
Oct 30 13:20:38.884294 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:38.884082    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d6dcafc-5203-4f02-b8b8-c06fc9eb9353" containerName="payload-pre-processing"
Oct 30 13:20:38.884294 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:38.884090    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a3bf11c-7722-47cd-a2ff-0e8cf5bacdcf" containerName="istio-proxy"
Oct 30 13:20:38.884294 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:38.884097    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="20cef20e-1c1b-4d57-b260-019f2de104a1" containerName="istio-proxy"
Oct 30 13:20:38.889675 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:38.889645    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-649303-8kg5n"
Oct 30 13:20:38.905892 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:38.905870    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:20:38.906287 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:38.906267    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:20:38.919952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:38.919864    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-649303-8kg5n"]
Oct 30 13:20:38.998659 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:38.998624    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056cc0ce-a210-4506-a6dc-7f43b733324d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-649303-8kg5n\" (UID: \"056cc0ce-a210-4506-a6dc-7f43b733324d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-649303-8kg5n"
Oct 30 13:20:38.998848 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:38.998665    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056cc0ce-a210-4506-a6dc-7f43b733324d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-649303-8kg5n\" (UID: \"056cc0ce-a210-4506-a6dc-7f43b733324d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-649303-8kg5n"
Oct 30 13:20:38.998892 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:38.998838    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k9nk\" (UniqueName: \"kubernetes.io/projected/056cc0ce-a210-4506-a6dc-7f43b733324d-kube-api-access-5k9nk\") pod \"maas-api-revoke-keys-e2e-inf-b-649303-8kg5n\" (UID: \"056cc0ce-a210-4506-a6dc-7f43b733324d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-649303-8kg5n"
Oct 30 13:20:39.099953 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:39.099912    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k9nk\" (UniqueName: \"kubernetes.io/projected/056cc0ce-a210-4506-a6dc-7f43b733324d-kube-api-access-5k9nk\") pod \"maas-api-revoke-keys-e2e-inf-b-649303-8kg5n\" (UID: \"056cc0ce-a210-4506-a6dc-7f43b733324d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-649303-8kg5n"
Oct 30 13:20:39.100171 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:39.100114    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056cc0ce-a210-4506-a6dc-7f43b733324d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-649303-8kg5n\" (UID: \"056cc0ce-a210-4506-a6dc-7f43b733324d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-649303-8kg5n"
Oct 30 13:20:39.100252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:39.100175    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056cc0ce-a210-4506-a6dc-7f43b733324d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-649303-8kg5n\" (UID: \"056cc0ce-a210-4506-a6dc-7f43b733324d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-649303-8kg5n"
Oct 30 13:20:39.100820 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:39.100787    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056cc0ce-a210-4506-a6dc-7f43b733324d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-649303-8kg5n\" (UID: \"056cc0ce-a210-4506-a6dc-7f43b733324d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-649303-8kg5n"
Oct 30 13:20:39.102649 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:39.102627    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056cc0ce-a210-4506-a6dc-7f43b733324d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-649303-8kg5n\" (UID: \"056cc0ce-a210-4506-a6dc-7f43b733324d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-649303-8kg5n"
Oct 30 13:20:39.125205 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:39.125129    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k9nk\" (UniqueName: \"kubernetes.io/projected/056cc0ce-a210-4506-a6dc-7f43b733324d-kube-api-access-5k9nk\") pod \"maas-api-revoke-keys-e2e-inf-b-649303-8kg5n\" (UID: \"056cc0ce-a210-4506-a6dc-7f43b733324d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-649303-8kg5n"
Oct 30 13:20:39.204534 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:39.204443    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-649303-8kg5n"
Oct 30 13:20:39.610974 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:39.602972    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-649303-8kg5n"]
Oct 30 13:20:39.808714 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:39.808679    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:39.909292 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:39.909196    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-649303-8kg5n" event={"ID":"056cc0ce-a210-4506-a6dc-7f43b733324d","Type":"ContainerStarted","Data":"4c71d0290aa552b98f86c92c11252d2c428b88fd2d5541fc7973e7b458a17797"}
Oct 30 13:20:40.918929 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:40.918892    4544 generic.go:356] "Generic (PLEG): container finished" podID="056cc0ce-a210-4506-a6dc-7f43b733324d" containerID="ccfd0efc4145503f874ce92d9810c3bee3aa12d124bae96772c5c12202dcf655" exitCode=0
Oct 30 13:20:40.919352 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:40.918970    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-649303-8kg5n" event={"ID":"056cc0ce-a210-4506-a6dc-7f43b733324d","Type":"ContainerDied","Data":"ccfd0efc4145503f874ce92d9810c3bee3aa12d124bae96772c5c12202dcf655"}
Oct 30 13:20:42.008271 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:42.003318    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:42.175955 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:42.175886    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:42.200472 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:42.200395    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-649303-8kg5n"
Oct 30 13:20:42.294320 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:42.288560    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/056cc0ce-a210-4506-a6dc-7f43b733324d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056cc0ce-a210-4506-a6dc-7f43b733324d-maas-api-openshift-ca\") pod \"056cc0ce-a210-4506-a6dc-7f43b733324d\" (UID: \"056cc0ce-a210-4506-a6dc-7f43b733324d\") "
Oct 30 13:20:42.294320 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:42.289733    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/056cc0ce-a210-4506-a6dc-7f43b733324d-maas-api-openshift-ca" pod "056cc0ce-a210-4506-a6dc-7f43b733324d" (UID: "056cc0ce-a210-4506-a6dc-7f43b733324d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:20:42.294320 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:42.290031    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/056cc0ce-a210-4506-a6dc-7f43b733324d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056cc0ce-a210-4506-a6dc-7f43b733324d-maas-api-xks-ca\") pod \"056cc0ce-a210-4506-a6dc-7f43b733324d\" (UID: \"056cc0ce-a210-4506-a6dc-7f43b733324d\") "
Oct 30 13:20:42.294320 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:42.291021    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/056cc0ce-a210-4506-a6dc-7f43b733324d-kube-api-access-5k9nk\" (UniqueName: \"kubernetes.io/projected/056cc0ce-a210-4506-a6dc-7f43b733324d-kube-api-access-5k9nk\") pod \"056cc0ce-a210-4506-a6dc-7f43b733324d\" (UID: \"056cc0ce-a210-4506-a6dc-7f43b733324d\") "
Oct 30 13:20:42.294320 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:42.291815    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056cc0ce-a210-4506-a6dc-7f43b733324d-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:42.301874 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:42.301820    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/056cc0ce-a210-4506-a6dc-7f43b733324d-maas-api-xks-ca" pod "056cc0ce-a210-4506-a6dc-7f43b733324d" (UID: "056cc0ce-a210-4506-a6dc-7f43b733324d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:20:42.302603 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:42.302565    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:42.356896 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:42.356822    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/056cc0ce-a210-4506-a6dc-7f43b733324d-kube-api-access-5k9nk" pod "056cc0ce-a210-4506-a6dc-7f43b733324d" (UID: "056cc0ce-a210-4506-a6dc-7f43b733324d"). InnerVolumeSpecName "kube-api-access-5k9nk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:42.393489 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:42.393440    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k9nk\" (UniqueName: \"kubernetes.io/projected/056cc0ce-a210-4506-a6dc-7f43b733324d-kube-api-access-5k9nk\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:42.393489 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:42.393483    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056cc0ce-a210-4506-a6dc-7f43b733324d-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:42.400415 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:42.400379    4544 patch_prober.go:29] interesting pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:20:42.400559 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:42.400439    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6" podUID="476505c3-3236-4789-846f-c13caa710b36" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:20:42.943847 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:42.943803    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-649303-8kg5n" event={"ID":"056cc0ce-a210-4506-a6dc-7f43b733324d","Type":"ContainerDied","Data":"4c71d0290aa552b98f86c92c11252d2c428b88fd2d5541fc7973e7b458a17797"}
Oct 30 13:20:42.944054 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:42.943876    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c71d0290aa552b98f86c92c11252d2c428b88fd2d5541fc7973e7b458a17797"
Oct 30 13:20:42.944054 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:42.943959    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-649303-8kg5n"
Oct 30 13:20:43.655832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.655762    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:20:43.745397 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.745364    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/476505c3-3236-4789-846f-c13caa710b36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/476505c3-3236-4789-846f-c13caa710b36-istiod-ca-cert\") pod \"476505c3-3236-4789-846f-c13caa710b36\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") "
Oct 30 13:20:43.745574 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.745535    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/476505c3-3236-4789-846f-c13caa710b36-kube-api-access-652h2\" (UniqueName: \"kubernetes.io/projected/476505c3-3236-4789-846f-c13caa710b36-kube-api-access-652h2\") pod \"476505c3-3236-4789-846f-c13caa710b36\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") "
Oct 30 13:20:43.745681 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.745664    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/476505c3-3236-4789-846f-c13caa710b36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/476505c3-3236-4789-846f-c13caa710b36-istio-podinfo\") pod \"476505c3-3236-4789-846f-c13caa710b36\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") "
Oct 30 13:20:43.746162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.745790    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-istio-data\") pod \"476505c3-3236-4789-846f-c13caa710b36\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") "
Oct 30 13:20:43.746162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.745861    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-istio-envoy\") pod \"476505c3-3236-4789-846f-c13caa710b36\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") "
Oct 30 13:20:43.746162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.745873    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/476505c3-3236-4789-846f-c13caa710b36-istiod-ca-cert" pod "476505c3-3236-4789-846f-c13caa710b36" (UID: "476505c3-3236-4789-846f-c13caa710b36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:20:43.746162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.745951    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-credential-socket\") pod \"476505c3-3236-4789-846f-c13caa710b36\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") "
Oct 30 13:20:43.746162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.746013    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-workload-certs\") pod \"476505c3-3236-4789-846f-c13caa710b36\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") "
Oct 30 13:20:43.746162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.746050    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-workload-socket\") pod \"476505c3-3236-4789-846f-c13caa710b36\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") "
Oct 30 13:20:43.746162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.746089    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/476505c3-3236-4789-846f-c13caa710b36-istio-token\" (UniqueName: \"kubernetes.io/projected/476505c3-3236-4789-846f-c13caa710b36-istio-token\") pod \"476505c3-3236-4789-846f-c13caa710b36\" (UID: \"476505c3-3236-4789-846f-c13caa710b36\") "
Oct 30 13:20:43.746611 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.746181    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-istio-data" pod "476505c3-3236-4789-846f-c13caa710b36" (UID: "476505c3-3236-4789-846f-c13caa710b36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:43.746611 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.746457    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-workload-certs" pod "476505c3-3236-4789-846f-c13caa710b36" (UID: "476505c3-3236-4789-846f-c13caa710b36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:43.747045 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.747016    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:43.747137 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.747054    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:43.747137 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.747069    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/476505c3-3236-4789-846f-c13caa710b36-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:43.747332 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.747311    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-credential-socket" pod "476505c3-3236-4789-846f-c13caa710b36" (UID: "476505c3-3236-4789-846f-c13caa710b36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:43.747513 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.747491    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-workload-socket" pod "476505c3-3236-4789-846f-c13caa710b36" (UID: "476505c3-3236-4789-846f-c13caa710b36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:43.748998 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.748966    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/476505c3-3236-4789-846f-c13caa710b36-kube-api-access-652h2" pod "476505c3-3236-4789-846f-c13caa710b36" (UID: "476505c3-3236-4789-846f-c13caa710b36"). InnerVolumeSpecName "kube-api-access-652h2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:43.749432 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.749406    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/476505c3-3236-4789-846f-c13caa710b36-istio-token" pod "476505c3-3236-4789-846f-c13caa710b36" (UID: "476505c3-3236-4789-846f-c13caa710b36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:43.749591 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.749566    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/476505c3-3236-4789-846f-c13caa710b36-istio-podinfo" pod "476505c3-3236-4789-846f-c13caa710b36" (UID: "476505c3-3236-4789-846f-c13caa710b36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:20:43.751166 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.751133    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-istio-envoy" pod "476505c3-3236-4789-846f-c13caa710b36" (UID: "476505c3-3236-4789-846f-c13caa710b36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:43.812335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.812296    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:43.848308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.848269    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-652h2\" (UniqueName: \"kubernetes.io/projected/476505c3-3236-4789-846f-c13caa710b36-kube-api-access-652h2\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:43.848308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.848298    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/476505c3-3236-4789-846f-c13caa710b36-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:43.848308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.848313    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:43.848577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.848325    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:43.848577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.848343    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/476505c3-3236-4789-846f-c13caa710b36-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:43.848577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.848355    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/476505c3-3236-4789-846f-c13caa710b36-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:43.954124 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.954032    4544 generic.go:356] "Generic (PLEG): container finished" podID="476505c3-3236-4789-846f-c13caa710b36" containerID="b0b7fe58678bf51c96306fd7784d2a86c1e89d59d71c9f0c54901f3be76c79f4" exitCode=0
Oct 30 13:20:43.954124 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.954094    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"
Oct 30 13:20:43.954408 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.954117    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6" event={"ID":"476505c3-3236-4789-846f-c13caa710b36","Type":"ContainerDied","Data":"b0b7fe58678bf51c96306fd7784d2a86c1e89d59d71c9f0c54901f3be76c79f4"}
Oct 30 13:20:43.954408 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.954174    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6" event={"ID":"476505c3-3236-4789-846f-c13caa710b36","Type":"ContainerDied","Data":"ab89f5db184a4fb433af771414977ce9123c8eabfc7800f8e8cad13659eaf3d9"}
Oct 30 13:20:43.954408 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.954200    4544 scope.go:122] "RemoveContainer" containerID="b0b7fe58678bf51c96306fd7784d2a86c1e89d59d71c9f0c54901f3be76c79f4"
Oct 30 13:20:43.966549 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.966531    4544 scope.go:122] "RemoveContainer" containerID="56400f2249f611470566cc3ca65eee52ef00ac4e02853170e613fd42d51eb337"
Oct 30 13:20:43.981771 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.981745    4544 scope.go:122] "RemoveContainer" containerID="b0b7fe58678bf51c96306fd7784d2a86c1e89d59d71c9f0c54901f3be76c79f4"
Oct 30 13:20:43.982083 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:20:43.982060    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0b7fe58678bf51c96306fd7784d2a86c1e89d59d71c9f0c54901f3be76c79f4\": container with ID starting with b0b7fe58678bf51c96306fd7784d2a86c1e89d59d71c9f0c54901f3be76c79f4 not found: ID does not exist" containerID="b0b7fe58678bf51c96306fd7784d2a86c1e89d59d71c9f0c54901f3be76c79f4"
Oct 30 13:20:43.982141 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.982094    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0b7fe58678bf51c96306fd7784d2a86c1e89d59d71c9f0c54901f3be76c79f4"} err="failed to get container status \"b0b7fe58678bf51c96306fd7784d2a86c1e89d59d71c9f0c54901f3be76c79f4\": rpc error: code = NotFound desc = could not find container \"b0b7fe58678bf51c96306fd7784d2a86c1e89d59d71c9f0c54901f3be76c79f4\": container with ID starting with b0b7fe58678bf51c96306fd7784d2a86c1e89d59d71c9f0c54901f3be76c79f4 not found: ID does not exist"
Oct 30 13:20:43.982141 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.982112    4544 scope.go:122] "RemoveContainer" containerID="56400f2249f611470566cc3ca65eee52ef00ac4e02853170e613fd42d51eb337"
Oct 30 13:20:43.982459 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:20:43.982433    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56400f2249f611470566cc3ca65eee52ef00ac4e02853170e613fd42d51eb337\": container with ID starting with 56400f2249f611470566cc3ca65eee52ef00ac4e02853170e613fd42d51eb337 not found: ID does not exist" containerID="56400f2249f611470566cc3ca65eee52ef00ac4e02853170e613fd42d51eb337"
Oct 30 13:20:43.982527 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:43.982466    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56400f2249f611470566cc3ca65eee52ef00ac4e02853170e613fd42d51eb337"} err="failed to get container status \"56400f2249f611470566cc3ca65eee52ef00ac4e02853170e613fd42d51eb337\": rpc error: code = NotFound desc = could not find container \"56400f2249f611470566cc3ca65eee52ef00ac4e02853170e613fd42d51eb337\": container with ID starting with 56400f2249f611470566cc3ca65eee52ef00ac4e02853170e613fd42d51eb337 not found: ID does not exist"
Oct 30 13:20:44.029744 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:44.029712    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"]
Oct 30 13:20:44.041833 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:44.041803    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6"]
Oct 30 13:20:44.348252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:44.345169    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:44.500839 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:44.500805    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:45.180244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.179947    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:45.266565 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.266528    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="476505c3-3236-4789-846f-c13caa710b36" path="/var/lib/kubelet/pods/476505c3-3236-4789-846f-c13caa710b36/volumes"
Oct 30 13:20:45.266926 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.266909    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"]
Oct 30 13:20:45.267768 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.267744    4544 cpu_manager.go:414] "containerMap: removing container" podUID="476505c3-3236-4789-846f-c13caa710b36" containerName="istio-proxy"
Oct 30 13:20:45.267768 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.267763    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="476505c3-3236-4789-846f-c13caa710b36" containerName="istio-proxy"
Oct 30 13:20:45.267962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.267782    4544 cpu_manager.go:414] "containerMap: removing container" podUID="056cc0ce-a210-4506-a6dc-7f43b733324d" containerName="revoke-keys"
Oct 30 13:20:45.267962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.267788    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="056cc0ce-a210-4506-a6dc-7f43b733324d" containerName="revoke-keys"
Oct 30 13:20:45.267962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.267798    4544 cpu_manager.go:414] "containerMap: removing container" podUID="476505c3-3236-4789-846f-c13caa710b36" containerName="istio-proxy"
Oct 30 13:20:45.267962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.267803    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="476505c3-3236-4789-846f-c13caa710b36" containerName="istio-proxy"
Oct 30 13:20:45.267962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.267941    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="476505c3-3236-4789-846f-c13caa710b36" containerName="istio-proxy"
Oct 30 13:20:45.267962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.267950    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="056cc0ce-a210-4506-a6dc-7f43b733324d" containerName="revoke-keys"
Oct 30 13:20:45.267962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.267956    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="476505c3-3236-4789-846f-c13caa710b36" containerName="istio-proxy"
Oct 30 13:20:45.273361 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.273340    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"]
Oct 30 13:20:45.273472 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.273432    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:20:45.301689 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.301662    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a5e47c-openshift-default-dockercfg-5hq7k\""
Oct 30 13:20:45.398485 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.398444    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkjwf\" (UniqueName: \"kubernetes.io/projected/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-kube-api-access-kkjwf\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:20:45.398655 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.398513    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istio-token\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:20:45.398655 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.398534    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istio-envoy\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:20:45.398768 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.398648    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-workload-socket\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:20:45.398810 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.398787    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istio-data\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:20:45.398921 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.398904    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-credential-socket\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:20:45.398971 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.398938    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-workload-certs\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:20:45.399144 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.399131    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istio-podinfo\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:20:45.399189 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.399169    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istiod-ca-cert\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:20:45.493579 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.493484    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:45.501090 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.501060    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-workload-socket\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:20:45.501254 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.501161    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istio-data\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:20:45.501307 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.501254    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-credential-socket\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:20:45.501307 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.501278    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-workload-certs\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:20:45.501390 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.501376    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istio-podinfo\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:20:45.501441 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.501426    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istiod-ca-cert\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:20:45.501496 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.501465    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-workload-socket\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:20:45.501577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.501535    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istio-data\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:20:45.501641 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.501590    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkjwf\" (UniqueName: \"kubernetes.io/projected/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-kube-api-access-kkjwf\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:20:45.501641 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.501597    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-credential-socket\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:20:45.501641 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.501618    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istio-token\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:20:45.501891 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.501648    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istio-envoy\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:20:45.501891 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.501669    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-workload-certs\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:20:45.502123 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.502100    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istiod-ca-cert\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:20:45.503795 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.503769    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istio-envoy\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:20:45.504098 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.504074    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istio-podinfo\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:20:45.531693 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.531661    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istio-token\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:20:45.531693 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.531677    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkjwf\" (UniqueName: \"kubernetes.io/projected/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-kube-api-access-kkjwf\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:20:45.588836 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.588802    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:20:45.800936 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.800843    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd"
Oct 30 13:20:45.965927 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.965893    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"]
Oct 30 13:20:45.968708 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:20:45.968676    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd4c038f_d0d4_4fd5_ad51_2dbae210cb26.slice/crio-e29c90e0a3d4cbbd2119a8cace01f4847a5139579ae2224c7c92fbaa6cde9381 WatchSource:0}: Error finding container e29c90e0a3d4cbbd2119a8cace01f4847a5139579ae2224c7c92fbaa6cde9381: Status 404 returned error can't find the container with id e29c90e0a3d4cbbd2119a8cace01f4847a5139579ae2224c7c92fbaa6cde9381
Oct 30 13:20:45.973194 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.973144    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:20:45.973364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.973270    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:20:45.973364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:45.973322    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:20:46.035513 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:46.035480    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl" event={"ID":"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26","Type":"ContainerStarted","Data":"e29c90e0a3d4cbbd2119a8cace01f4847a5139579ae2224c7c92fbaa6cde9381"}
Oct 30 13:20:47.095244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:47.083650    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl" event={"ID":"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26","Type":"ContainerStarted","Data":"8065e62d66aaf050576444ef5dd78cae779752f4f4e2c60b2f551ced1d07dd5d"}
Oct 30 13:20:47.173418 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:47.170548    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl" podStartSLOduration=3.170529711 podStartE2EDuration="3.170529711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:47.156460154 +0000 UTC m=+2142.203594108" watchObservedRunningTime="2026-10-30 13:20:47.170529711 +0000 UTC m=+2142.217663667"
Oct 30 13:20:47.494568 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:47.494471    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:47.596399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:47.596141    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:20:47.815490 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:47.815404    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:20:48.093940 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:48.093909    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:20:48.095779 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:48.095756    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:20:49.294621 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:49.294585    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:49.419291 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:49.419249    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-649303-5c6ff5544-429vh"]
Oct 30 13:20:49.479292 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:49.479211    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:49.495680 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:49.495649    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w"]
Oct 30 13:20:49.684855 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:49.684818    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:49.868986 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:49.868955    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:50.096023 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:50.095988    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:52.304005 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:52.303966    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:52.390714 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:52.390680    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9"]
Oct 30 13:20:52.400162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:52.400130    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9"
Oct 30 13:20:52.421771 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:52.421736    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-a5e47c\""
Oct 30 13:20:52.422655 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:52.422629    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9"]
Oct 30 13:20:52.463398 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:52.463357    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbhms\" (UniqueName: \"kubernetes.io/projected/872e259b-51ea-477d-b898-e8d4bbb086f5-kube-api-access-hbhms\") pod \"payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9\" (UID: \"872e259b-51ea-477d-b898-e8d4bbb086f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9"
Oct 30 13:20:52.463613 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:52.463510    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872e259b-51ea-477d-b898-e8d4bbb086f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9\" (UID: \"872e259b-51ea-477d-b898-e8d4bbb086f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9"
Oct 30 13:20:52.506146 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:52.506110    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj"]
Oct 30 13:20:52.511403 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:52.511379    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj"
Oct 30 13:20:52.529086 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:52.529054    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-a5e47c-dockercfg-c645m\""
Oct 30 13:20:52.529848 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:52.529824    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:52.547013 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:52.546982    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj"]
Oct 30 13:20:52.565101 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:52.564999    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbhms\" (UniqueName: \"kubernetes.io/projected/872e259b-51ea-477d-b898-e8d4bbb086f5-kube-api-access-hbhms\") pod \"payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9\" (UID: \"872e259b-51ea-477d-b898-e8d4bbb086f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9"
Oct 30 13:20:52.565305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:52.565098    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872e259b-51ea-477d-b898-e8d4bbb086f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9\" (UID: \"872e259b-51ea-477d-b898-e8d4bbb086f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9"
Oct 30 13:20:52.566339 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:52.566312    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872e259b-51ea-477d-b898-e8d4bbb086f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9\" (UID: \"872e259b-51ea-477d-b898-e8d4bbb086f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9"
Oct 30 13:20:52.591481 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:52.591446    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbhms\" (UniqueName: \"kubernetes.io/projected/872e259b-51ea-477d-b898-e8d4bbb086f5-kube-api-access-hbhms\") pod \"payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9\" (UID: \"872e259b-51ea-477d-b898-e8d4bbb086f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9"
Oct 30 13:20:52.710244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:52.683268    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a9b6789-4b42-43b8-a815-bc9f835452c7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj\" (UID: \"2a9b6789-4b42-43b8-a815-bc9f835452c7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj"
Oct 30 13:20:52.710244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:52.683631    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt9t2\" (UniqueName: \"kubernetes.io/projected/2a9b6789-4b42-43b8-a815-bc9f835452c7-kube-api-access-vt9t2\") pod \"payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj\" (UID: \"2a9b6789-4b42-43b8-a815-bc9f835452c7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj"
Oct 30 13:20:52.720379 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:52.720078    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9"
Oct 30 13:20:52.757291 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:52.744912    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:52.806459 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:52.798849    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt9t2\" (UniqueName: \"kubernetes.io/projected/2a9b6789-4b42-43b8-a815-bc9f835452c7-kube-api-access-vt9t2\") pod \"payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj\" (UID: \"2a9b6789-4b42-43b8-a815-bc9f835452c7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj"
Oct 30 13:20:52.806459 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:52.798953    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a9b6789-4b42-43b8-a815-bc9f835452c7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj\" (UID: \"2a9b6789-4b42-43b8-a815-bc9f835452c7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj"
Oct 30 13:20:52.814264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:52.807616    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a9b6789-4b42-43b8-a815-bc9f835452c7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj\" (UID: \"2a9b6789-4b42-43b8-a815-bc9f835452c7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj"
Oct 30 13:20:52.870339 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:52.866131    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt9t2\" (UniqueName: \"kubernetes.io/projected/2a9b6789-4b42-43b8-a815-bc9f835452c7-kube-api-access-vt9t2\") pod \"payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj\" (UID: \"2a9b6789-4b42-43b8-a815-bc9f835452c7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj"
Oct 30 13:20:53.126735 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:53.126654    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj"
Oct 30 13:20:53.169080 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:53.169046    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9"]
Oct 30 13:20:53.170650 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:20:53.170616    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872e259b_51ea_477d_b898_e8d4bbb086f5.slice/crio-2c784cc29ccabda944e84812729fb6638cc069abd16111450f8280a641f58de8 WatchSource:0}: Error finding container 2c784cc29ccabda944e84812729fb6638cc069abd16111450f8280a641f58de8: Status 404 returned error can't find the container with id 2c784cc29ccabda944e84812729fb6638cc069abd16111450f8280a641f58de8
Oct 30 13:20:53.442545 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:20:53.442484    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a9b6789_4b42_43b8_a815_bc9f835452c7.slice/crio-3a919582a6ddecb2b134e2f3a580cc2c1f7adbc662f085eb9dcc7d095e1ba5f3 WatchSource:0}: Error finding container 3a919582a6ddecb2b134e2f3a580cc2c1f7adbc662f085eb9dcc7d095e1ba5f3: Status 404 returned error can't find the container with id 3a919582a6ddecb2b134e2f3a580cc2c1f7adbc662f085eb9dcc7d095e1ba5f3
Oct 30 13:20:53.442920 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:53.442708    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj"]
Oct 30 13:20:54.146592 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:54.146556    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9" event={"ID":"872e259b-51ea-477d-b898-e8d4bbb086f5","Type":"ContainerStarted","Data":"81a19b3c7be5e67ceb8a4d553f6de5ea437599f43f81dc99e970905d9a2df212"}
Oct 30 13:20:54.146800 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:54.146611    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9" event={"ID":"872e259b-51ea-477d-b898-e8d4bbb086f5","Type":"ContainerStarted","Data":"2c784cc29ccabda944e84812729fb6638cc069abd16111450f8280a641f58de8"}
Oct 30 13:20:54.146800 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:54.146635    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9"
Oct 30 13:20:54.148441 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:54.148412    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj" event={"ID":"2a9b6789-4b42-43b8-a815-bc9f835452c7","Type":"ContainerStarted","Data":"61c150bc17cff9173ff5d6c5f32dc9c1b8b4a8b8c4486fef770420a3af8e8bbc"}
Oct 30 13:20:54.148577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:54.148457    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj" event={"ID":"2a9b6789-4b42-43b8-a815-bc9f835452c7","Type":"ContainerStarted","Data":"3a919582a6ddecb2b134e2f3a580cc2c1f7adbc662f085eb9dcc7d095e1ba5f3"}
Oct 30 13:20:54.148577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:54.148480    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj"
Oct 30 13:20:54.192813 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:54.192739    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9" podStartSLOduration=2.192719297 podStartE2EDuration="2.192719297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:54.188766988 +0000 UTC m=+2149.235900943" watchObservedRunningTime="2026-10-30 13:20:54.192719297 +0000 UTC m=+2149.239853249"
Oct 30 13:20:54.244922 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:54.244826    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj" podStartSLOduration=2.244800729 podStartE2EDuration="2.244800729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:54.239269748 +0000 UTC m=+2149.286403693" watchObservedRunningTime="2026-10-30 13:20:54.244800729 +0000 UTC m=+2149.291934744"
Oct 30 13:20:54.425401 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:54.421439    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-649303-5c6ff5544-429vh" podUID="cdc909e9-a85b-4c23-8218-c0a982b13829" containerName="payload-processing" containerID="cri-o://7acb3bd725ca5bcb002422ff88669751cac7784c19650ecf692cbb5cd23fb23e" gracePeriod=25
Oct 30 13:20:54.496765 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:54.496720    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w" podUID="8dfe63f2-8812-4b0b-a4e2-e0684d7ba985" containerName="payload-pre-processing" containerID="cri-o://e0f9c9172ea33a197ff4e79bbc6e634ae2ebaaa34a8d8e6e119c9f7bdf6c0e66" gracePeriod=25
Oct 30 13:20:54.720369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:54.720339    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-649303-5c6ff5544-429vh"
Oct 30 13:20:54.735530 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:54.735482    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdc909e9-a85b-4c23-8218-c0a982b13829-kube-api-access-lh2sf\" (UniqueName: \"kubernetes.io/projected/cdc909e9-a85b-4c23-8218-c0a982b13829-kube-api-access-lh2sf\") pod \"cdc909e9-a85b-4c23-8218-c0a982b13829\" (UID: \"cdc909e9-a85b-4c23-8218-c0a982b13829\") "
Oct 30 13:20:54.735692 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:54.735584    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdc909e9-a85b-4c23-8218-c0a982b13829-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc909e9-a85b-4c23-8218-c0a982b13829-plugins-config-volume\") pod \"cdc909e9-a85b-4c23-8218-c0a982b13829\" (UID: \"cdc909e9-a85b-4c23-8218-c0a982b13829\") "
Oct 30 13:20:54.735955 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:54.735931    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdc909e9-a85b-4c23-8218-c0a982b13829-plugins-config-volume" pod "cdc909e9-a85b-4c23-8218-c0a982b13829" (UID: "cdc909e9-a85b-4c23-8218-c0a982b13829"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:20:54.736173 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:54.736158    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc909e9-a85b-4c23-8218-c0a982b13829-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:54.737901 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:54.737872    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc909e9-a85b-4c23-8218-c0a982b13829-kube-api-access-lh2sf" pod "cdc909e9-a85b-4c23-8218-c0a982b13829" (UID: "cdc909e9-a85b-4c23-8218-c0a982b13829"). InnerVolumeSpecName "kube-api-access-lh2sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:54.837855 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:54.837808    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh2sf\" (UniqueName: \"kubernetes.io/projected/cdc909e9-a85b-4c23-8218-c0a982b13829-kube-api-access-lh2sf\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:20:54.994927 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:54.994837    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:55.157948 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:55.157915    4544 generic.go:356] "Generic (PLEG): container finished" podID="cdc909e9-a85b-4c23-8218-c0a982b13829" containerID="7acb3bd725ca5bcb002422ff88669751cac7784c19650ecf692cbb5cd23fb23e" exitCode=0
Oct 30 13:20:55.158134 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:55.157977    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-649303-5c6ff5544-429vh"
Oct 30 13:20:55.158134 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:55.158007    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-649303-5c6ff5544-429vh" event={"ID":"cdc909e9-a85b-4c23-8218-c0a982b13829","Type":"ContainerDied","Data":"7acb3bd725ca5bcb002422ff88669751cac7784c19650ecf692cbb5cd23fb23e"}
Oct 30 13:20:55.158134 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:55.158093    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-649303-5c6ff5544-429vh" event={"ID":"cdc909e9-a85b-4c23-8218-c0a982b13829","Type":"ContainerDied","Data":"6bfdfe15370f68e709f6ad288b906fa7dd406252e7b2320f31f3e8264be46758"}
Oct 30 13:20:55.158134 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:55.158122    4544 scope.go:122] "RemoveContainer" containerID="7acb3bd725ca5bcb002422ff88669751cac7784c19650ecf692cbb5cd23fb23e"
Oct 30 13:20:55.169483 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:55.169461    4544 scope.go:122] "RemoveContainer" containerID="7acb3bd725ca5bcb002422ff88669751cac7784c19650ecf692cbb5cd23fb23e"
Oct 30 13:20:55.169771 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:20:55.169750    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7acb3bd725ca5bcb002422ff88669751cac7784c19650ecf692cbb5cd23fb23e\": container with ID starting with 7acb3bd725ca5bcb002422ff88669751cac7784c19650ecf692cbb5cd23fb23e not found: ID does not exist" containerID="7acb3bd725ca5bcb002422ff88669751cac7784c19650ecf692cbb5cd23fb23e"
Oct 30 13:20:55.169852 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:55.169779    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7acb3bd725ca5bcb002422ff88669751cac7784c19650ecf692cbb5cd23fb23e"} err="failed to get container status \"7acb3bd725ca5bcb002422ff88669751cac7784c19650ecf692cbb5cd23fb23e\": rpc error: code = NotFound desc = could not find container \"7acb3bd725ca5bcb002422ff88669751cac7784c19650ecf692cbb5cd23fb23e\": container with ID starting with 7acb3bd725ca5bcb002422ff88669751cac7784c19650ecf692cbb5cd23fb23e not found: ID does not exist"
Oct 30 13:20:55.214578 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:55.214534    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-649303-5c6ff5544-429vh"]
Oct 30 13:20:55.225697 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:55.225664    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-649303-5c6ff5544-429vh"]
Oct 30 13:20:55.539461 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:55.539419    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w" podUID="8dfe63f2-8812-4b0b-a4e2-e0684d7ba985" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:20:55.906877 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:55.906830    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:56.189697 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:56.189609    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:56.600889 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:56.600853    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:57.169662 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:57.169619    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdc909e9-a85b-4c23-8218-c0a982b13829" path="/var/lib/kubelet/pods/cdc909e9-a85b-4c23-8218-c0a982b13829/volumes"
Oct 30 13:20:57.315407 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:57.315372    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:57.438101 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:57.438019    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a8ee1-nmkb2"]
Oct 30 13:20:57.442582 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:57.442558    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:57.449437 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:57.449409    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4a8ee1-nmkb2"]
Oct 30 13:20:57.581699 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:57.578790    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:57.983369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:57.983332    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:20:59.010361 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:20:59.010324    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:20:59.090466 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:59.090423    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f3613a9-0816-4310-986d-43dbd8b16288" path="/var/lib/kubelet/pods/7f3613a9-0816-4310-986d-43dbd8b16288/volumes"
Oct 30 13:20:59.752457 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:20:59.752392    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:02.068096 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:02.068057    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:04.309213 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:04.309162    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:05.162313 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:05.162281    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj"
Oct 30 13:21:05.162521 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:05.162371    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9"
Oct 30 13:21:05.539607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:05.539507    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w" podUID="8dfe63f2-8812-4b0b-a4e2-e0684d7ba985" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:21:05.722658 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:05.722626    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:05.865678 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:05.865628    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:06.042103 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:06.042062    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"]
Oct 30 13:21:06.042406 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:06.042370    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v" podUID="4fc60c82-a658-4c5d-9dd4-a4eae3af4846" containerName="istio-proxy" containerID="cri-o://a41fea898496fdd20f3ebf27e37c1b9fea9c32967ed3ae16660249892a6a1f82" gracePeriod=30
Oct 30 13:21:06.123532 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:06.123433    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:06.279894 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:06.279790    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:06.608981 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:06.608942    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f9071-qxtqd"]
Oct 30 13:21:06.610001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:06.609978    4544 cpu_manager.go:414] "containerMap: removing container" podUID="cdc909e9-a85b-4c23-8218-c0a982b13829" containerName="payload-processing"
Oct 30 13:21:06.610001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:06.610001    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdc909e9-a85b-4c23-8218-c0a982b13829" containerName="payload-processing"
Oct 30 13:21:06.610140 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:06.610131    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdc909e9-a85b-4c23-8218-c0a982b13829" containerName="payload-processing"
Oct 30 13:21:06.614435 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:06.614405    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f9071-qxtqd"
Oct 30 13:21:06.630497 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:06.630465    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:21:06.630497 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:06.630473    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:21:06.636528 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:06.636491    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f9071-qxtqd"]
Oct 30 13:21:06.676303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:06.676254    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxfdr\" (UniqueName: \"kubernetes.io/projected/c07719a1-a141-463d-9191-c060d4ca35c1-kube-api-access-bxfdr\") pod \"maas-api-revoke-keys-e2e-inf-b-6f9071-qxtqd\" (UID: \"c07719a1-a141-463d-9191-c060d4ca35c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f9071-qxtqd"
Oct 30 13:21:06.676478 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:06.676449    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07719a1-a141-463d-9191-c060d4ca35c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6f9071-qxtqd\" (UID: \"c07719a1-a141-463d-9191-c060d4ca35c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f9071-qxtqd"
Oct 30 13:21:06.676631 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:06.676613    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07719a1-a141-463d-9191-c060d4ca35c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6f9071-qxtqd\" (UID: \"c07719a1-a141-463d-9191-c060d4ca35c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f9071-qxtqd"
Oct 30 13:21:06.777747 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:06.777702    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07719a1-a141-463d-9191-c060d4ca35c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6f9071-qxtqd\" (UID: \"c07719a1-a141-463d-9191-c060d4ca35c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f9071-qxtqd"
Oct 30 13:21:06.777951 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:06.777854    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07719a1-a141-463d-9191-c060d4ca35c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6f9071-qxtqd\" (UID: \"c07719a1-a141-463d-9191-c060d4ca35c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f9071-qxtqd"
Oct 30 13:21:06.778060 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:06.778042    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxfdr\" (UniqueName: \"kubernetes.io/projected/c07719a1-a141-463d-9191-c060d4ca35c1-kube-api-access-bxfdr\") pod \"maas-api-revoke-keys-e2e-inf-b-6f9071-qxtqd\" (UID: \"c07719a1-a141-463d-9191-c060d4ca35c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f9071-qxtqd"
Oct 30 13:21:06.778495 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:06.778471    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07719a1-a141-463d-9191-c060d4ca35c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6f9071-qxtqd\" (UID: \"c07719a1-a141-463d-9191-c060d4ca35c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f9071-qxtqd"
Oct 30 13:21:06.780639 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:06.780615    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07719a1-a141-463d-9191-c060d4ca35c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6f9071-qxtqd\" (UID: \"c07719a1-a141-463d-9191-c060d4ca35c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f9071-qxtqd"
Oct 30 13:21:06.802055 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:06.802021    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxfdr\" (UniqueName: \"kubernetes.io/projected/c07719a1-a141-463d-9191-c060d4ca35c1-kube-api-access-bxfdr\") pod \"maas-api-revoke-keys-e2e-inf-b-6f9071-qxtqd\" (UID: \"c07719a1-a141-463d-9191-c060d4ca35c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f9071-qxtqd"
Oct 30 13:21:06.929698 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:06.929610    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f9071-qxtqd"
Oct 30 13:21:07.295716 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:07.295681    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f9071-qxtqd"]
Oct 30 13:21:07.296804 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:21:07.296770    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc07719a1_a141_463d_9191_c060d4ca35c1.slice/crio-beb7603247e1940e019eac7335585b2feca662539cbe4e71c856187f63e380d8 WatchSource:0}: Error finding container beb7603247e1940e019eac7335585b2feca662539cbe4e71c856187f63e380d8: Status 404 returned error can't find the container with id beb7603247e1940e019eac7335585b2feca662539cbe4e71c856187f63e380d8
Oct 30 13:21:07.385262 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:07.385210    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f9071-qxtqd" event={"ID":"c07719a1-a141-463d-9191-c060d4ca35c1","Type":"ContainerStarted","Data":"beb7603247e1940e019eac7335585b2feca662539cbe4e71c856187f63e380d8"}
Oct 30 13:21:07.482931 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:07.482886    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:08.397009 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:08.396958    4544 generic.go:356] "Generic (PLEG): container finished" podID="c07719a1-a141-463d-9191-c060d4ca35c1" containerID="2a3912b63efae2656d38f4f4aa0f39bdc8a89032d91c3a54e0da275ada1333c9" exitCode=0
Oct 30 13:21:08.397570 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:08.397036    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f9071-qxtqd" event={"ID":"c07719a1-a141-463d-9191-c060d4ca35c1","Type":"ContainerDied","Data":"2a3912b63efae2656d38f4f4aa0f39bdc8a89032d91c3a54e0da275ada1333c9"}
Oct 30 13:21:08.448539 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:08.448502    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:09.519390 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:09.519334    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:09.607015 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:09.606986    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f9071-qxtqd"
Oct 30 13:21:09.739309 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:09.738764    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c07719a1-a141-463d-9191-c060d4ca35c1-kube-api-access-bxfdr\" (UniqueName: \"kubernetes.io/projected/c07719a1-a141-463d-9191-c060d4ca35c1-kube-api-access-bxfdr\") pod \"c07719a1-a141-463d-9191-c060d4ca35c1\" (UID: \"c07719a1-a141-463d-9191-c060d4ca35c1\") "
Oct 30 13:21:09.739309 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:09.738853    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c07719a1-a141-463d-9191-c060d4ca35c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07719a1-a141-463d-9191-c060d4ca35c1-maas-api-openshift-ca\") pod \"c07719a1-a141-463d-9191-c060d4ca35c1\" (UID: \"c07719a1-a141-463d-9191-c060d4ca35c1\") "
Oct 30 13:21:09.739309 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:09.739120    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c07719a1-a141-463d-9191-c060d4ca35c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07719a1-a141-463d-9191-c060d4ca35c1-maas-api-xks-ca\") pod \"c07719a1-a141-463d-9191-c060d4ca35c1\" (UID: \"c07719a1-a141-463d-9191-c060d4ca35c1\") "
Oct 30 13:21:09.803246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:09.773986    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c07719a1-a141-463d-9191-c060d4ca35c1-maas-api-openshift-ca" pod "c07719a1-a141-463d-9191-c060d4ca35c1" (UID: "c07719a1-a141-463d-9191-c060d4ca35c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:21:09.803246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:09.774903    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07719a1-a141-463d-9191-c060d4ca35c1-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:21:09.818472 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:09.818407    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c07719a1-a141-463d-9191-c060d4ca35c1-kube-api-access-bxfdr" pod "c07719a1-a141-463d-9191-c060d4ca35c1" (UID: "c07719a1-a141-463d-9191-c060d4ca35c1"). InnerVolumeSpecName "kube-api-access-bxfdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:21:09.886696 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:09.886658    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxfdr\" (UniqueName: \"kubernetes.io/projected/c07719a1-a141-463d-9191-c060d4ca35c1-kube-api-access-bxfdr\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:21:09.889809 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:09.889767    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c07719a1-a141-463d-9191-c060d4ca35c1-maas-api-xks-ca" pod "c07719a1-a141-463d-9191-c060d4ca35c1" (UID: "c07719a1-a141-463d-9191-c060d4ca35c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:21:09.988502 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:09.988464    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07719a1-a141-463d-9191-c060d4ca35c1-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:21:10.418655 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:10.418606    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f9071-qxtqd" event={"ID":"c07719a1-a141-463d-9191-c060d4ca35c1","Type":"ContainerDied","Data":"beb7603247e1940e019eac7335585b2feca662539cbe4e71c856187f63e380d8"}
Oct 30 13:21:10.418845 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:10.418679    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="beb7603247e1940e019eac7335585b2feca662539cbe4e71c856187f63e380d8"
Oct 30 13:21:10.418845 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:10.418631    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f9071-qxtqd"
Oct 30 13:21:10.556944 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:10.556907    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:11.345482 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.345455    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:21:11.432671 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.432642    4544 generic.go:356] "Generic (PLEG): container finished" podID="4fc60c82-a658-4c5d-9dd4-a4eae3af4846" containerID="a41fea898496fdd20f3ebf27e37c1b9fea9c32967ed3ae16660249892a6a1f82" exitCode=0
Oct 30 13:21:11.432878 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.432691    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v" event={"ID":"4fc60c82-a658-4c5d-9dd4-a4eae3af4846","Type":"ContainerDied","Data":"a41fea898496fdd20f3ebf27e37c1b9fea9c32967ed3ae16660249892a6a1f82"}
Oct 30 13:21:11.432878 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.432699    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"
Oct 30 13:21:11.432878 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.432736    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v" event={"ID":"4fc60c82-a658-4c5d-9dd4-a4eae3af4846","Type":"ContainerDied","Data":"aedfbd547f5a4f42a6be4ae98e5166ba31ae708a287648754ec9c98ac344240e"}
Oct 30 13:21:11.432878 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.432761    4544 scope.go:122] "RemoveContainer" containerID="a41fea898496fdd20f3ebf27e37c1b9fea9c32967ed3ae16660249892a6a1f82"
Oct 30 13:21:11.448536 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.448485    4544 scope.go:122] "RemoveContainer" containerID="8eb07ad10148fc577f31de8b2c4715f2668e48fbb32527af0083cbf48d637553"
Oct 30 13:21:11.470516 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.470483    4544 scope.go:122] "RemoveContainer" containerID="a41fea898496fdd20f3ebf27e37c1b9fea9c32967ed3ae16660249892a6a1f82"
Oct 30 13:21:11.470868 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:21:11.470848    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a41fea898496fdd20f3ebf27e37c1b9fea9c32967ed3ae16660249892a6a1f82\": container with ID starting with a41fea898496fdd20f3ebf27e37c1b9fea9c32967ed3ae16660249892a6a1f82 not found: ID does not exist" containerID="a41fea898496fdd20f3ebf27e37c1b9fea9c32967ed3ae16660249892a6a1f82"
Oct 30 13:21:11.470938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.470879    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a41fea898496fdd20f3ebf27e37c1b9fea9c32967ed3ae16660249892a6a1f82"} err="failed to get container status \"a41fea898496fdd20f3ebf27e37c1b9fea9c32967ed3ae16660249892a6a1f82\": rpc error: code = NotFound desc = could not find container \"a41fea898496fdd20f3ebf27e37c1b9fea9c32967ed3ae16660249892a6a1f82\": container with ID starting with a41fea898496fdd20f3ebf27e37c1b9fea9c32967ed3ae16660249892a6a1f82 not found: ID does not exist"
Oct 30 13:21:11.470938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.470898    4544 scope.go:122] "RemoveContainer" containerID="8eb07ad10148fc577f31de8b2c4715f2668e48fbb32527af0083cbf48d637553"
Oct 30 13:21:11.471237 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:21:11.471200    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eb07ad10148fc577f31de8b2c4715f2668e48fbb32527af0083cbf48d637553\": container with ID starting with 8eb07ad10148fc577f31de8b2c4715f2668e48fbb32527af0083cbf48d637553 not found: ID does not exist" containerID="8eb07ad10148fc577f31de8b2c4715f2668e48fbb32527af0083cbf48d637553"
Oct 30 13:21:11.471312 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.471241    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eb07ad10148fc577f31de8b2c4715f2668e48fbb32527af0083cbf48d637553"} err="failed to get container status \"8eb07ad10148fc577f31de8b2c4715f2668e48fbb32527af0083cbf48d637553\": rpc error: code = NotFound desc = could not find container \"8eb07ad10148fc577f31de8b2c4715f2668e48fbb32527af0083cbf48d637553\": container with ID starting with 8eb07ad10148fc577f31de8b2c4715f2668e48fbb32527af0083cbf48d637553 not found: ID does not exist"
Oct 30 13:21:11.531899 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.531857    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istio-envoy\") pod \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") "
Oct 30 13:21:11.532092 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.531916    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-kube-api-access-w9lpb\" (UniqueName: \"kubernetes.io/projected/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-kube-api-access-w9lpb\") pod \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") "
Oct 30 13:21:11.532092 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.531956    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istiod-ca-cert\") pod \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") "
Oct 30 13:21:11.532092 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.531994    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-workload-certs\") pod \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") "
Oct 30 13:21:11.532092 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.532033    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istio-podinfo\") pod \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") "
Oct 30 13:21:11.532368 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.532159    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-credential-socket\") pod \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") "
Oct 30 13:21:11.532428 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.532365    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istio-data\") pod \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") "
Oct 30 13:21:11.532428 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.532412    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istio-token\" (UniqueName: \"kubernetes.io/projected/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istio-token\") pod \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") "
Oct 30 13:21:11.532542 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.532402    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-workload-certs" pod "4fc60c82-a658-4c5d-9dd4-a4eae3af4846" (UID: "4fc60c82-a658-4c5d-9dd4-a4eae3af4846"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:21:11.532542 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.532441    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-workload-socket\") pod \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\" (UID: \"4fc60c82-a658-4c5d-9dd4-a4eae3af4846\") "
Oct 30 13:21:11.532542 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.532498    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istiod-ca-cert" pod "4fc60c82-a658-4c5d-9dd4-a4eae3af4846" (UID: "4fc60c82-a658-4c5d-9dd4-a4eae3af4846"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:21:11.532695 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.532646    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istio-data" pod "4fc60c82-a658-4c5d-9dd4-a4eae3af4846" (UID: "4fc60c82-a658-4c5d-9dd4-a4eae3af4846"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:21:11.532763 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.532715    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-credential-socket" pod "4fc60c82-a658-4c5d-9dd4-a4eae3af4846" (UID: "4fc60c82-a658-4c5d-9dd4-a4eae3af4846"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:21:11.532906 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.532878    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-workload-socket" pod "4fc60c82-a658-4c5d-9dd4-a4eae3af4846" (UID: "4fc60c82-a658-4c5d-9dd4-a4eae3af4846"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:21:11.533345 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.533320    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:21:11.533471 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.533349    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:21:11.533471 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.533365    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:21:11.533471 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.533378    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:21:11.533471 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.533396    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:21:11.534638 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.534577    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istio-envoy" pod "4fc60c82-a658-4c5d-9dd4-a4eae3af4846" (UID: "4fc60c82-a658-4c5d-9dd4-a4eae3af4846"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:21:11.534758 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.534652    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istio-podinfo" pod "4fc60c82-a658-4c5d-9dd4-a4eae3af4846" (UID: "4fc60c82-a658-4c5d-9dd4-a4eae3af4846"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:21:11.534758 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.534674    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-kube-api-access-w9lpb" pod "4fc60c82-a658-4c5d-9dd4-a4eae3af4846" (UID: "4fc60c82-a658-4c5d-9dd4-a4eae3af4846"). InnerVolumeSpecName "kube-api-access-w9lpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:21:11.534758 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.534583    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istio-token" pod "4fc60c82-a658-4c5d-9dd4-a4eae3af4846" (UID: "4fc60c82-a658-4c5d-9dd4-a4eae3af4846"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:21:11.635507 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.635461    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:21:11.635507 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.635500    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:21:11.635507 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.635510    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9lpb\" (UniqueName: \"kubernetes.io/projected/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-kube-api-access-w9lpb\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:21:11.635507 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.635520    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fc60c82-a658-4c5d-9dd4-a4eae3af4846-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:21:11.783658 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.783622    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"]
Oct 30 13:21:11.795667 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:11.795621    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v"]
Oct 30 13:21:12.596009 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:12.595971    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:13.087523 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:13.087491    4544 scope.go:122] "RemoveContainer" containerID="73c3209fcc1f2274c34495595d4d68935c81b1a81c120141307ffe01b22b49f0"
Oct 30 13:21:13.090342 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:13.090315    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fc60c82-a658-4c5d-9dd4-a4eae3af4846" path="/var/lib/kubelet/pods/4fc60c82-a658-4c5d-9dd4-a4eae3af4846/volumes"
Oct 30 13:21:14.151955 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:14.151910    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"]
Oct 30 13:21:14.152488 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:14.152208    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v" podUID="1e40797f-db7b-4502-b06c-bb8b419e52b9" containerName="istio-proxy" containerID="cri-o://17c44f4538ec3162998590472db2bfc1c4d35bd29829331aeab2cfe6768df920" gracePeriod=30
Oct 30 13:21:14.712070 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:14.709083    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:14.857735 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:14.857700    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w"
Oct 30 13:21:14.910630 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:14.910595    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dfe63f2-8812-4b0b-a4e2-e0684d7ba985-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dfe63f2-8812-4b0b-a4e2-e0684d7ba985-plugins-config-volume\") pod \"8dfe63f2-8812-4b0b-a4e2-e0684d7ba985\" (UID: \"8dfe63f2-8812-4b0b-a4e2-e0684d7ba985\") "
Oct 30 13:21:14.910853 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:14.910826    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dfe63f2-8812-4b0b-a4e2-e0684d7ba985-kube-api-access-5jgkg\" (UniqueName: \"kubernetes.io/projected/8dfe63f2-8812-4b0b-a4e2-e0684d7ba985-kube-api-access-5jgkg\") pod \"8dfe63f2-8812-4b0b-a4e2-e0684d7ba985\" (UID: \"8dfe63f2-8812-4b0b-a4e2-e0684d7ba985\") "
Oct 30 13:21:14.911047 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:14.911020    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dfe63f2-8812-4b0b-a4e2-e0684d7ba985-plugins-config-volume" pod "8dfe63f2-8812-4b0b-a4e2-e0684d7ba985" (UID: "8dfe63f2-8812-4b0b-a4e2-e0684d7ba985"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:21:14.911498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:14.911478    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dfe63f2-8812-4b0b-a4e2-e0684d7ba985-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:21:14.913056 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:14.913023    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dfe63f2-8812-4b0b-a4e2-e0684d7ba985-kube-api-access-5jgkg" pod "8dfe63f2-8812-4b0b-a4e2-e0684d7ba985" (UID: "8dfe63f2-8812-4b0b-a4e2-e0684d7ba985"). InnerVolumeSpecName "kube-api-access-5jgkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:21:15.012843 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:15.012804    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jgkg\" (UniqueName: \"kubernetes.io/projected/8dfe63f2-8812-4b0b-a4e2-e0684d7ba985-kube-api-access-5jgkg\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:21:15.479755 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:15.479661    4544 generic.go:356] "Generic (PLEG): container finished" podID="8dfe63f2-8812-4b0b-a4e2-e0684d7ba985" containerID="e0f9c9172ea33a197ff4e79bbc6e634ae2ebaaa34a8d8e6e119c9f7bdf6c0e66" exitCode=0
Oct 30 13:21:15.479755 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:15.479712    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w"
Oct 30 13:21:15.480496 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:15.479744    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w" event={"ID":"8dfe63f2-8812-4b0b-a4e2-e0684d7ba985","Type":"ContainerDied","Data":"e0f9c9172ea33a197ff4e79bbc6e634ae2ebaaa34a8d8e6e119c9f7bdf6c0e66"}
Oct 30 13:21:15.480496 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:15.479820    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w" event={"ID":"8dfe63f2-8812-4b0b-a4e2-e0684d7ba985","Type":"ContainerDied","Data":"c127bab9c40ea13b9fdf4e0c973adca03b9bb7b87e6f45ef0cb0238d808da41c"}
Oct 30 13:21:15.480496 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:15.479855    4544 scope.go:122] "RemoveContainer" containerID="e0f9c9172ea33a197ff4e79bbc6e634ae2ebaaa34a8d8e6e119c9f7bdf6c0e66"
Oct 30 13:21:15.491770 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:15.491747    4544 scope.go:122] "RemoveContainer" containerID="e0f9c9172ea33a197ff4e79bbc6e634ae2ebaaa34a8d8e6e119c9f7bdf6c0e66"
Oct 30 13:21:15.492132 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:21:15.492112    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0f9c9172ea33a197ff4e79bbc6e634ae2ebaaa34a8d8e6e119c9f7bdf6c0e66\": container with ID starting with e0f9c9172ea33a197ff4e79bbc6e634ae2ebaaa34a8d8e6e119c9f7bdf6c0e66 not found: ID does not exist" containerID="e0f9c9172ea33a197ff4e79bbc6e634ae2ebaaa34a8d8e6e119c9f7bdf6c0e66"
Oct 30 13:21:15.492232 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:15.492140    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0f9c9172ea33a197ff4e79bbc6e634ae2ebaaa34a8d8e6e119c9f7bdf6c0e66"} err="failed to get container status \"e0f9c9172ea33a197ff4e79bbc6e634ae2ebaaa34a8d8e6e119c9f7bdf6c0e66\": rpc error: code = NotFound desc = could not find container \"e0f9c9172ea33a197ff4e79bbc6e634ae2ebaaa34a8d8e6e119c9f7bdf6c0e66\": container with ID starting with e0f9c9172ea33a197ff4e79bbc6e634ae2ebaaa34a8d8e6e119c9f7bdf6c0e66 not found: ID does not exist"
Oct 30 13:21:15.577869 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:15.577827    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w"]
Oct 30 13:21:15.589827 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:15.589786    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w"]
Oct 30 13:21:16.889599 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:16.889564    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:17.089171 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:17.089137    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dfe63f2-8812-4b0b-a4e2-e0684d7ba985" path="/var/lib/kubelet/pods/8dfe63f2-8812-4b0b-a4e2-e0684d7ba985/volumes"
Oct 30 13:21:17.208072 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:17.207988    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk"]
Oct 30 13:21:17.364803 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:17.364766    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:17.524639 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:17.524548    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:18.780687 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:18.780650    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:19.454882 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.454840    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 13:21:19.519680 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.519640    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-workload-certs\") pod \"1e40797f-db7b-4502-b06c-bb8b419e52b9\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") "
Oct 30 13:21:19.519916 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.519709    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-credential-socket\") pod \"1e40797f-db7b-4502-b06c-bb8b419e52b9\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") "
Oct 30 13:21:19.519916 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.519870    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e40797f-db7b-4502-b06c-bb8b419e52b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e40797f-db7b-4502-b06c-bb8b419e52b9-istiod-ca-cert\") pod \"1e40797f-db7b-4502-b06c-bb8b419e52b9\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") "
Oct 30 13:21:19.520040 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.519965    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-workload-socket\") pod \"1e40797f-db7b-4502-b06c-bb8b419e52b9\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") "
Oct 30 13:21:19.520040 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.519962    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-workload-certs" pod "1e40797f-db7b-4502-b06c-bb8b419e52b9" (UID: "1e40797f-db7b-4502-b06c-bb8b419e52b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:21:19.520145 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.520049    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-credential-socket" pod "1e40797f-db7b-4502-b06c-bb8b419e52b9" (UID: "1e40797f-db7b-4502-b06c-bb8b419e52b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:21:19.520145 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.520109    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e40797f-db7b-4502-b06c-bb8b419e52b9-kube-api-access-t6lvs\" (UniqueName: \"kubernetes.io/projected/1e40797f-db7b-4502-b06c-bb8b419e52b9-kube-api-access-t6lvs\") pod \"1e40797f-db7b-4502-b06c-bb8b419e52b9\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") "
Oct 30 13:21:19.520375 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.520148    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-istio-envoy\") pod \"1e40797f-db7b-4502-b06c-bb8b419e52b9\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") "
Oct 30 13:21:19.520375 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.520214    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-workload-socket" pod "1e40797f-db7b-4502-b06c-bb8b419e52b9" (UID: "1e40797f-db7b-4502-b06c-bb8b419e52b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:21:19.520375 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.520214    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e40797f-db7b-4502-b06c-bb8b419e52b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e40797f-db7b-4502-b06c-bb8b419e52b9-istio-podinfo\") pod \"1e40797f-db7b-4502-b06c-bb8b419e52b9\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") "
Oct 30 13:21:19.520375 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.520274    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e40797f-db7b-4502-b06c-bb8b419e52b9-istiod-ca-cert" pod "1e40797f-db7b-4502-b06c-bb8b419e52b9" (UID: "1e40797f-db7b-4502-b06c-bb8b419e52b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:21:19.520375 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.520314    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-istio-data\") pod \"1e40797f-db7b-4502-b06c-bb8b419e52b9\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") "
Oct 30 13:21:19.520625 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.520377    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e40797f-db7b-4502-b06c-bb8b419e52b9-istio-token\" (UniqueName: \"kubernetes.io/projected/1e40797f-db7b-4502-b06c-bb8b419e52b9-istio-token\") pod \"1e40797f-db7b-4502-b06c-bb8b419e52b9\" (UID: \"1e40797f-db7b-4502-b06c-bb8b419e52b9\") "
Oct 30 13:21:19.520625 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.520552    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-istio-data" pod "1e40797f-db7b-4502-b06c-bb8b419e52b9" (UID: "1e40797f-db7b-4502-b06c-bb8b419e52b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:21:19.521012 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.520993    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:21:19.521117 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.521018    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:21:19.521117 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.521031    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:21:19.521117 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.521044    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e40797f-db7b-4502-b06c-bb8b419e52b9-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:21:19.521117 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.521058    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:21:19.522710 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.522621    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e40797f-db7b-4502-b06c-bb8b419e52b9-kube-api-access-t6lvs" pod "1e40797f-db7b-4502-b06c-bb8b419e52b9" (UID: "1e40797f-db7b-4502-b06c-bb8b419e52b9"). InnerVolumeSpecName "kube-api-access-t6lvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:21:19.522710 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.522699    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e40797f-db7b-4502-b06c-bb8b419e52b9-istio-token" pod "1e40797f-db7b-4502-b06c-bb8b419e52b9" (UID: "1e40797f-db7b-4502-b06c-bb8b419e52b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:21:19.522845 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.522769    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e40797f-db7b-4502-b06c-bb8b419e52b9-istio-podinfo" pod "1e40797f-db7b-4502-b06c-bb8b419e52b9" (UID: "1e40797f-db7b-4502-b06c-bb8b419e52b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:21:19.522886 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.522848    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-istio-envoy" pod "1e40797f-db7b-4502-b06c-bb8b419e52b9" (UID: "1e40797f-db7b-4502-b06c-bb8b419e52b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:21:19.535725 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.535694    4544 generic.go:356] "Generic (PLEG): container finished" podID="1e40797f-db7b-4502-b06c-bb8b419e52b9" containerID="17c44f4538ec3162998590472db2bfc1c4d35bd29829331aeab2cfe6768df920" exitCode=0
Oct 30 13:21:19.535910 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.535775    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"
Oct 30 13:21:19.535910 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.535785    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v" event={"ID":"1e40797f-db7b-4502-b06c-bb8b419e52b9","Type":"ContainerDied","Data":"17c44f4538ec3162998590472db2bfc1c4d35bd29829331aeab2cfe6768df920"}
Oct 30 13:21:19.535910 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.535839    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v" event={"ID":"1e40797f-db7b-4502-b06c-bb8b419e52b9","Type":"ContainerDied","Data":"8df7c8d982968c65d2668f56c0ecd8d5097aa10e4be9a11bf39e36ce6009c0ac"}
Oct 30 13:21:19.535910 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.535868    4544 scope.go:122] "RemoveContainer" containerID="17c44f4538ec3162998590472db2bfc1c4d35bd29829331aeab2cfe6768df920"
Oct 30 13:21:19.565479 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.565444    4544 scope.go:122] "RemoveContainer" containerID="ceeaed7d2a60a6a4a239bfbe002009ebb873cc0df96aaa62c30588e4636c020f"
Oct 30 13:21:19.584671 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.584635    4544 scope.go:122] "RemoveContainer" containerID="17c44f4538ec3162998590472db2bfc1c4d35bd29829331aeab2cfe6768df920"
Oct 30 13:21:19.594505 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:21:19.587492    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17c44f4538ec3162998590472db2bfc1c4d35bd29829331aeab2cfe6768df920\": container with ID starting with 17c44f4538ec3162998590472db2bfc1c4d35bd29829331aeab2cfe6768df920 not found: ID does not exist" containerID="17c44f4538ec3162998590472db2bfc1c4d35bd29829331aeab2cfe6768df920"
Oct 30 13:21:19.594505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.587541    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17c44f4538ec3162998590472db2bfc1c4d35bd29829331aeab2cfe6768df920"} err="failed to get container status \"17c44f4538ec3162998590472db2bfc1c4d35bd29829331aeab2cfe6768df920\": rpc error: code = NotFound desc = could not find container \"17c44f4538ec3162998590472db2bfc1c4d35bd29829331aeab2cfe6768df920\": container with ID starting with 17c44f4538ec3162998590472db2bfc1c4d35bd29829331aeab2cfe6768df920 not found: ID does not exist"
Oct 30 13:21:19.594505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.587586    4544 scope.go:122] "RemoveContainer" containerID="ceeaed7d2a60a6a4a239bfbe002009ebb873cc0df96aaa62c30588e4636c020f"
Oct 30 13:21:19.594505 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:21:19.588144    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceeaed7d2a60a6a4a239bfbe002009ebb873cc0df96aaa62c30588e4636c020f\": container with ID starting with ceeaed7d2a60a6a4a239bfbe002009ebb873cc0df96aaa62c30588e4636c020f not found: ID does not exist" containerID="ceeaed7d2a60a6a4a239bfbe002009ebb873cc0df96aaa62c30588e4636c020f"
Oct 30 13:21:19.594505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.588179    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceeaed7d2a60a6a4a239bfbe002009ebb873cc0df96aaa62c30588e4636c020f"} err="failed to get container status \"ceeaed7d2a60a6a4a239bfbe002009ebb873cc0df96aaa62c30588e4636c020f\": rpc error: code = NotFound desc = could not find container \"ceeaed7d2a60a6a4a239bfbe002009ebb873cc0df96aaa62c30588e4636c020f\": container with ID starting with ceeaed7d2a60a6a4a239bfbe002009ebb873cc0df96aaa62c30588e4636c020f not found: ID does not exist"
Oct 30 13:21:19.603194 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.603158    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"]
Oct 30 13:21:19.608014 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.607975    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v"]
Oct 30 13:21:19.622068 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.622029    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6lvs\" (UniqueName: \"kubernetes.io/projected/1e40797f-db7b-4502-b06c-bb8b419e52b9-kube-api-access-t6lvs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:21:19.622068 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.622064    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e40797f-db7b-4502-b06c-bb8b419e52b9-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:21:19.622068 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.622074    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e40797f-db7b-4502-b06c-bb8b419e52b9-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:21:19.622361 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:19.622082    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e40797f-db7b-4502-b06c-bb8b419e52b9-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:21:20.623255 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:20.619681    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:21.089039 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:21.089002    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e40797f-db7b-4502-b06c-bb8b419e52b9" path="/var/lib/kubelet/pods/1e40797f-db7b-4502-b06c-bb8b419e52b9/volumes"
Oct 30 13:21:22.209209 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:22.209143    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk" podUID="9564d09c-f00e-48e9-bd57-a96f0bb6efc3" containerName="payload-pre-processing" containerID="cri-o://789c1d6bc39f0376e06f2d93f69ad888743d546b10bdf7d662998b8a626fc798" gracePeriod=25
Oct 30 13:21:22.438937 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:22.438890    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:24.154091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:24.154055    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:25.828811 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:25.828765    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:27.743652 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:27.743611    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47"]
Oct 30 13:21:27.744478 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:27.744459    4544 cpu_manager.go:414] "containerMap: removing container" podUID="4fc60c82-a658-4c5d-9dd4-a4eae3af4846" containerName="istio-proxy"
Oct 30 13:21:27.744543 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:27.744480    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fc60c82-a658-4c5d-9dd4-a4eae3af4846" containerName="istio-proxy"
Oct 30 13:21:27.744543 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:27.744508    4544 cpu_manager.go:414] "containerMap: removing container" podUID="8dfe63f2-8812-4b0b-a4e2-e0684d7ba985" containerName="payload-pre-processing"
Oct 30 13:21:27.744543 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:27.744514    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dfe63f2-8812-4b0b-a4e2-e0684d7ba985" containerName="payload-pre-processing"
Oct 30 13:21:27.744543 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:27.744526    4544 cpu_manager.go:414] "containerMap: removing container" podUID="1e40797f-db7b-4502-b06c-bb8b419e52b9" containerName="istio-proxy"
Oct 30 13:21:27.744543 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:27.744530    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e40797f-db7b-4502-b06c-bb8b419e52b9" containerName="istio-proxy"
Oct 30 13:21:27.744742 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:27.744552    4544 cpu_manager.go:414] "containerMap: removing container" podUID="c07719a1-a141-463d-9191-c060d4ca35c1" containerName="revoke-keys"
Oct 30 13:21:27.744742 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:27.744557    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c07719a1-a141-463d-9191-c060d4ca35c1" containerName="revoke-keys"
Oct 30 13:21:27.744742 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:27.744682    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e40797f-db7b-4502-b06c-bb8b419e52b9" containerName="istio-proxy"
Oct 30 13:21:27.744742 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:27.744695    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fc60c82-a658-4c5d-9dd4-a4eae3af4846" containerName="istio-proxy"
Oct 30 13:21:27.744742 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:27.744701    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="c07719a1-a141-463d-9191-c060d4ca35c1" containerName="revoke-keys"
Oct 30 13:21:27.744742 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:27.744712    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fc60c82-a658-4c5d-9dd4-a4eae3af4846" containerName="istio-proxy"
Oct 30 13:21:27.744742 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:27.744723    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dfe63f2-8812-4b0b-a4e2-e0684d7ba985" containerName="payload-pre-processing"
Oct 30 13:21:27.750634 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:27.750608    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47"
Oct 30 13:21:27.768092 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:27.768058    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-f3338e-serving-cert\""
Oct 30 13:21:27.771765 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:27.771732    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47"]
Oct 30 13:21:27.861513 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:27.861468    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6705f8d7-884a-4bf2-921b-e4656dfae77d-metrics-certs\") pod \"maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47\" (UID: \"6705f8d7-884a-4bf2-921b-e4656dfae77d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47"
Oct 30 13:21:27.861513 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:27.861515    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6705f8d7-884a-4bf2-921b-e4656dfae77d-maas-api-tls\") pod \"maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47\" (UID: \"6705f8d7-884a-4bf2-921b-e4656dfae77d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47"
Oct 30 13:21:27.861771 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:27.861721    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6705f8d7-884a-4bf2-921b-e4656dfae77d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47\" (UID: \"6705f8d7-884a-4bf2-921b-e4656dfae77d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47"
Oct 30 13:21:27.861924 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:27.861894    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr8vc\" (UniqueName: \"kubernetes.io/projected/6705f8d7-884a-4bf2-921b-e4656dfae77d-kube-api-access-jr8vc\") pod \"maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47\" (UID: \"6705f8d7-884a-4bf2-921b-e4656dfae77d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47"
Oct 30 13:21:27.870264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:27.870212    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:27.972567 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:27.972497    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr8vc\" (UniqueName: \"kubernetes.io/projected/6705f8d7-884a-4bf2-921b-e4656dfae77d-kube-api-access-jr8vc\") pod \"maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47\" (UID: \"6705f8d7-884a-4bf2-921b-e4656dfae77d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47"
Oct 30 13:21:28.009532 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:28.009424    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6705f8d7-884a-4bf2-921b-e4656dfae77d-metrics-certs\") pod \"maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47\" (UID: \"6705f8d7-884a-4bf2-921b-e4656dfae77d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47"
Oct 30 13:21:28.009824 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:28.009801    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6705f8d7-884a-4bf2-921b-e4656dfae77d-maas-api-tls\") pod \"maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47\" (UID: \"6705f8d7-884a-4bf2-921b-e4656dfae77d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47"
Oct 30 13:21:28.010203 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:28.010173    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6705f8d7-884a-4bf2-921b-e4656dfae77d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47\" (UID: \"6705f8d7-884a-4bf2-921b-e4656dfae77d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47"
Oct 30 13:21:28.019317 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:28.011299    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6705f8d7-884a-4bf2-921b-e4656dfae77d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47\" (UID: \"6705f8d7-884a-4bf2-921b-e4656dfae77d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47"
Oct 30 13:21:28.049252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:28.038171    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:28.049252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:28.039425    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr8vc\" (UniqueName: \"kubernetes.io/projected/6705f8d7-884a-4bf2-921b-e4656dfae77d-kube-api-access-jr8vc\") pod \"maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47\" (UID: \"6705f8d7-884a-4bf2-921b-e4656dfae77d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47"
Oct 30 13:21:28.049252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:28.042679    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6705f8d7-884a-4bf2-921b-e4656dfae77d-maas-api-tls\") pod \"maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47\" (UID: \"6705f8d7-884a-4bf2-921b-e4656dfae77d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47"
Oct 30 13:21:28.049252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:28.043439    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6705f8d7-884a-4bf2-921b-e4656dfae77d-metrics-certs\") pod \"maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47\" (UID: \"6705f8d7-884a-4bf2-921b-e4656dfae77d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47"
Oct 30 13:21:28.066034 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:28.065989    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47"
Oct 30 13:21:28.415178 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:28.415139    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47"]
Oct 30 13:21:28.416836 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:21:28.416807    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6705f8d7_884a_4bf2_921b_e4656dfae77d.slice/crio-926c4a5f6754175914773d39b5708d87d8347b969e053406b14d4417e97da24f WatchSource:0}: Error finding container 926c4a5f6754175914773d39b5708d87d8347b969e053406b14d4417e97da24f: Status 404 returned error can't find the container with id 926c4a5f6754175914773d39b5708d87d8347b969e053406b14d4417e97da24f
Oct 30 13:21:28.616885 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:28.616844    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47" event={"ID":"6705f8d7-884a-4bf2-921b-e4656dfae77d","Type":"ContainerStarted","Data":"926c4a5f6754175914773d39b5708d87d8347b969e053406b14d4417e97da24f"}
Oct 30 13:21:28.770617 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:28.770570    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk" podUID="9564d09c-f00e-48e9-bd57-a96f0bb6efc3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:21:29.010068 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:21:29.009977    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:21:29.627577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:29.627537    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47" event={"ID":"6705f8d7-884a-4bf2-921b-e4656dfae77d","Type":"ContainerStarted","Data":"0aa5195bbce1b8ad4de5e90ef43c784f95a9b23770b6edcb1853ef7af94bd7fa"}
Oct 30 13:21:29.627807 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:29.627667    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47"
Oct 30 13:21:29.685073 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:29.685011    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47" podStartSLOduration=2.372406624 podStartE2EDuration="2.684987919s" totalImagesPullingTime="312.581295ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:29.684723443 +0000 UTC m=+2184.731857397" watchObservedRunningTime="2026-10-30 13:21:29.684987919 +0000 UTC m=+2184.732121872"
Oct 30 13:21:29.980799 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:29.980710    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:31.262066 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:31.262027    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:32.105450 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:32.105305    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:34.305640 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:34.305598    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:35.640878 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:35.640847    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47"
Oct 30 13:21:35.977978 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:35.977879    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:37.784541 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:37.784507    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:38.768416 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:38.768371    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk" podUID="9564d09c-f00e-48e9-bd57-a96f0bb6efc3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:21:39.946681 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:39.946644    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:40.538523 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:40.538486    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn"]
Oct 30 13:21:40.539454 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:40.539433    4544 cpu_manager.go:414] "containerMap: removing container" podUID="1e40797f-db7b-4502-b06c-bb8b419e52b9" containerName="istio-proxy"
Oct 30 13:21:40.539454 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:40.539453    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e40797f-db7b-4502-b06c-bb8b419e52b9" containerName="istio-proxy"
Oct 30 13:21:40.539601 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:40.539498    4544 cpu_manager.go:414] "containerMap: removing container" podUID="4fc60c82-a658-4c5d-9dd4-a4eae3af4846" containerName="istio-proxy"
Oct 30 13:21:40.539601 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:40.539505    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fc60c82-a658-4c5d-9dd4-a4eae3af4846" containerName="istio-proxy"
Oct 30 13:21:40.539671 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:40.539642    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e40797f-db7b-4502-b06c-bb8b419e52b9" containerName="istio-proxy"
Oct 30 13:21:40.543472 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:40.543448    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn"
Oct 30 13:21:40.573345 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:40.573306    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-619b9b\"/\"openshift-service-ca.crt\""
Oct 30 13:21:40.573542 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:40.573449    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-619b9b\"/\"default-dockercfg-zbqzg\""
Oct 30 13:21:40.573597 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:40.573561    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-619b9b\"/\"test-model-619b9b-kserve-self-signed-certs\""
Oct 30 13:21:40.573597 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:40.573306    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-619b9b\"/\"kube-root-ca.crt\""
Oct 30 13:21:40.578737 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:40.578701    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn"]
Oct 30 13:21:40.609176 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:40.609136    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:40.667997 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:40.667961    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb9846d-c5b3-4d4d-8243-4169400da4ca-tls-certs\") pod \"test-model-619b9b-kserve-86b9bbfc-kl9pn\" (UID: \"ecb9846d-c5b3-4d4d-8243-4169400da4ca\") " pod="ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn"
Oct 30 13:21:40.668187 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:40.668152    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb9846d-c5b3-4d4d-8243-4169400da4ca-home\") pod \"test-model-619b9b-kserve-86b9bbfc-kl9pn\" (UID: \"ecb9846d-c5b3-4d4d-8243-4169400da4ca\") " pod="ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn"
Oct 30 13:21:40.668306 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:40.668260    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5zc2\" (UniqueName: \"kubernetes.io/projected/ecb9846d-c5b3-4d4d-8243-4169400da4ca-kube-api-access-w5zc2\") pod \"test-model-619b9b-kserve-86b9bbfc-kl9pn\" (UID: \"ecb9846d-c5b3-4d4d-8243-4169400da4ca\") " pod="ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn"
Oct 30 13:21:40.668372 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:40.668318    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb9846d-c5b3-4d4d-8243-4169400da4ca-dshm\") pod \"test-model-619b9b-kserve-86b9bbfc-kl9pn\" (UID: \"ecb9846d-c5b3-4d4d-8243-4169400da4ca\") " pod="ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn"
Oct 30 13:21:40.668439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:40.668378    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb9846d-c5b3-4d4d-8243-4169400da4ca-tmp-dir\") pod \"test-model-619b9b-kserve-86b9bbfc-kl9pn\" (UID: \"ecb9846d-c5b3-4d4d-8243-4169400da4ca\") " pod="ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn"
Oct 30 13:21:40.668488 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:40.668451    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb9846d-c5b3-4d4d-8243-4169400da4ca-model-cache\") pod \"test-model-619b9b-kserve-86b9bbfc-kl9pn\" (UID: \"ecb9846d-c5b3-4d4d-8243-4169400da4ca\") " pod="ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn"
Oct 30 13:21:40.769689 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:40.769654    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb9846d-c5b3-4d4d-8243-4169400da4ca-tmp-dir\") pod \"test-model-619b9b-kserve-86b9bbfc-kl9pn\" (UID: \"ecb9846d-c5b3-4d4d-8243-4169400da4ca\") " pod="ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn"
Oct 30 13:21:40.769891 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:40.769707    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb9846d-c5b3-4d4d-8243-4169400da4ca-model-cache\") pod \"test-model-619b9b-kserve-86b9bbfc-kl9pn\" (UID: \"ecb9846d-c5b3-4d4d-8243-4169400da4ca\") " pod="ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn"
Oct 30 13:21:40.769891 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:40.769794    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb9846d-c5b3-4d4d-8243-4169400da4ca-tls-certs\") pod \"test-model-619b9b-kserve-86b9bbfc-kl9pn\" (UID: \"ecb9846d-c5b3-4d4d-8243-4169400da4ca\") " pod="ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn"
Oct 30 13:21:40.769891 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:40.769881    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb9846d-c5b3-4d4d-8243-4169400da4ca-home\") pod \"test-model-619b9b-kserve-86b9bbfc-kl9pn\" (UID: \"ecb9846d-c5b3-4d4d-8243-4169400da4ca\") " pod="ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn"
Oct 30 13:21:40.770042 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:40.769930    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5zc2\" (UniqueName: \"kubernetes.io/projected/ecb9846d-c5b3-4d4d-8243-4169400da4ca-kube-api-access-w5zc2\") pod \"test-model-619b9b-kserve-86b9bbfc-kl9pn\" (UID: \"ecb9846d-c5b3-4d4d-8243-4169400da4ca\") " pod="ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn"
Oct 30 13:21:40.770042 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:40.769960    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb9846d-c5b3-4d4d-8243-4169400da4ca-dshm\") pod \"test-model-619b9b-kserve-86b9bbfc-kl9pn\" (UID: \"ecb9846d-c5b3-4d4d-8243-4169400da4ca\") " pod="ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn"
Oct 30 13:21:40.770189 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:40.770162    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb9846d-c5b3-4d4d-8243-4169400da4ca-tmp-dir\") pod \"test-model-619b9b-kserve-86b9bbfc-kl9pn\" (UID: \"ecb9846d-c5b3-4d4d-8243-4169400da4ca\") " pod="ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn"
Oct 30 13:21:40.770276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:40.770208    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb9846d-c5b3-4d4d-8243-4169400da4ca-model-cache\") pod \"test-model-619b9b-kserve-86b9bbfc-kl9pn\" (UID: \"ecb9846d-c5b3-4d4d-8243-4169400da4ca\") " pod="ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn"
Oct 30 13:21:40.770351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:40.770336    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb9846d-c5b3-4d4d-8243-4169400da4ca-home\") pod \"test-model-619b9b-kserve-86b9bbfc-kl9pn\" (UID: \"ecb9846d-c5b3-4d4d-8243-4169400da4ca\") " pod="ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn"
Oct 30 13:21:40.772396 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:40.772374    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb9846d-c5b3-4d4d-8243-4169400da4ca-dshm\") pod \"test-model-619b9b-kserve-86b9bbfc-kl9pn\" (UID: \"ecb9846d-c5b3-4d4d-8243-4169400da4ca\") " pod="ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn"
Oct 30 13:21:40.772619 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:40.772602    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb9846d-c5b3-4d4d-8243-4169400da4ca-tls-certs\") pod \"test-model-619b9b-kserve-86b9bbfc-kl9pn\" (UID: \"ecb9846d-c5b3-4d4d-8243-4169400da4ca\") " pod="ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn"
Oct 30 13:21:40.841246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:40.833571    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5zc2\" (UniqueName: \"kubernetes.io/projected/ecb9846d-c5b3-4d4d-8243-4169400da4ca-kube-api-access-w5zc2\") pod \"test-model-619b9b-kserve-86b9bbfc-kl9pn\" (UID: \"ecb9846d-c5b3-4d4d-8243-4169400da4ca\") " pod="ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn"
Oct 30 13:21:40.861232 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:40.860039    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn"
Oct 30 13:21:41.248184 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:41.248144    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn"]
Oct 30 13:21:41.251207 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:21:41.251175    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecb9846d_c5b3_4d4d_8243_4169400da4ca.slice/crio-69730359c0402e5b28d77a76f16a9285680aec0bfb44ae567f88df14d543b7e9 WatchSource:0}: Error finding container 69730359c0402e5b28d77a76f16a9285680aec0bfb44ae567f88df14d543b7e9: Status 404 returned error can't find the container with id 69730359c0402e5b28d77a76f16a9285680aec0bfb44ae567f88df14d543b7e9
Oct 30 13:21:41.326636 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:41.326599    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:41.869248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:41.869085    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn" event={"ID":"ecb9846d-c5b3-4d4d-8243-4169400da4ca","Type":"ContainerStarted","Data":"3697c8615d103a05fcf3bab13349506c6610f6483aab3161c85e11a1eb7d38c5"}
Oct 30 13:21:41.869248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:41.869145    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn" event={"ID":"ecb9846d-c5b3-4d4d-8243-4169400da4ca","Type":"ContainerStarted","Data":"69730359c0402e5b28d77a76f16a9285680aec0bfb44ae567f88df14d543b7e9"}
Oct 30 13:21:41.936269 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:41.935405    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn" podStartSLOduration=1.9353814969999998 podStartE2EDuration="1.935381497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:41.931664214 +0000 UTC m=+2196.978798169" watchObservedRunningTime="2026-10-30 13:21:41.935381497 +0000 UTC m=+2196.982515451"
Oct 30 13:21:42.099904 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:42.099865    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:42.180086 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:42.179990    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-649303-8kg5n"]
Oct 30 13:21:42.184328 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:42.184297    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-649303-8kg5n"]
Oct 30 13:21:42.242349 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:42.242316    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:42.540454 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:42.540416    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk"
Oct 30 13:21:42.646717 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:42.646680    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9564d09c-f00e-48e9-bd57-a96f0bb6efc3-kube-api-access-gs2pv\" (UniqueName: \"kubernetes.io/projected/9564d09c-f00e-48e9-bd57-a96f0bb6efc3-kube-api-access-gs2pv\") pod \"9564d09c-f00e-48e9-bd57-a96f0bb6efc3\" (UID: \"9564d09c-f00e-48e9-bd57-a96f0bb6efc3\") "
Oct 30 13:21:42.646946 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:42.646757    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9564d09c-f00e-48e9-bd57-a96f0bb6efc3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9564d09c-f00e-48e9-bd57-a96f0bb6efc3-plugins-config-volume\") pod \"9564d09c-f00e-48e9-bd57-a96f0bb6efc3\" (UID: \"9564d09c-f00e-48e9-bd57-a96f0bb6efc3\") "
Oct 30 13:21:42.647166 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:42.647138    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9564d09c-f00e-48e9-bd57-a96f0bb6efc3-plugins-config-volume" pod "9564d09c-f00e-48e9-bd57-a96f0bb6efc3" (UID: "9564d09c-f00e-48e9-bd57-a96f0bb6efc3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:21:42.647332 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:42.647314    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9564d09c-f00e-48e9-bd57-a96f0bb6efc3-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:21:42.649322 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:42.649295    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9564d09c-f00e-48e9-bd57-a96f0bb6efc3-kube-api-access-gs2pv" pod "9564d09c-f00e-48e9-bd57-a96f0bb6efc3" (UID: "9564d09c-f00e-48e9-bd57-a96f0bb6efc3"). InnerVolumeSpecName "kube-api-access-gs2pv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:21:42.741785 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:42.741729    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:42.748578 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:42.748541    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs2pv\" (UniqueName: \"kubernetes.io/projected/9564d09c-f00e-48e9-bd57-a96f0bb6efc3-kube-api-access-gs2pv\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:21:42.860603 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:42.860565    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn"
Oct 30 13:21:42.879082 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:42.879044    4544 generic.go:356] "Generic (PLEG): container finished" podID="9564d09c-f00e-48e9-bd57-a96f0bb6efc3" containerID="789c1d6bc39f0376e06f2d93f69ad888743d546b10bdf7d662998b8a626fc798" exitCode=0
Oct 30 13:21:42.879295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:42.879092    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk"
Oct 30 13:21:42.879295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:42.879156    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk" event={"ID":"9564d09c-f00e-48e9-bd57-a96f0bb6efc3","Type":"ContainerDied","Data":"789c1d6bc39f0376e06f2d93f69ad888743d546b10bdf7d662998b8a626fc798"}
Oct 30 13:21:42.879295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:42.879213    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk" event={"ID":"9564d09c-f00e-48e9-bd57-a96f0bb6efc3","Type":"ContainerDied","Data":"862454dff47b6538f309151fec2880ed877a54dd8dcd8447bdd0d2319caa0211"}
Oct 30 13:21:42.879295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:42.879268    4544 scope.go:122] "RemoveContainer" containerID="789c1d6bc39f0376e06f2d93f69ad888743d546b10bdf7d662998b8a626fc798"
Oct 30 13:21:42.892921 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:42.892895    4544 scope.go:122] "RemoveContainer" containerID="789c1d6bc39f0376e06f2d93f69ad888743d546b10bdf7d662998b8a626fc798"
Oct 30 13:21:42.893333 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:21:42.893305    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"789c1d6bc39f0376e06f2d93f69ad888743d546b10bdf7d662998b8a626fc798\": container with ID starting with 789c1d6bc39f0376e06f2d93f69ad888743d546b10bdf7d662998b8a626fc798 not found: ID does not exist" containerID="789c1d6bc39f0376e06f2d93f69ad888743d546b10bdf7d662998b8a626fc798"
Oct 30 13:21:42.893461 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:42.893340    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"789c1d6bc39f0376e06f2d93f69ad888743d546b10bdf7d662998b8a626fc798"} err="failed to get container status \"789c1d6bc39f0376e06f2d93f69ad888743d546b10bdf7d662998b8a626fc798\": rpc error: code = NotFound desc = could not find container \"789c1d6bc39f0376e06f2d93f69ad888743d546b10bdf7d662998b8a626fc798\": container with ID starting with 789c1d6bc39f0376e06f2d93f69ad888743d546b10bdf7d662998b8a626fc798 not found: ID does not exist"
Oct 30 13:21:42.934984 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:42.934944    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk"]
Oct 30 13:21:42.946934 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:42.946900    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk"]
Oct 30 13:21:43.088938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:43.088902    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="056cc0ce-a210-4506-a6dc-7f43b733324d" path="/var/lib/kubelet/pods/056cc0ce-a210-4506-a6dc-7f43b733324d/volumes"
Oct 30 13:21:43.089482 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:43.089464    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9564d09c-f00e-48e9-bd57-a96f0bb6efc3" path="/var/lib/kubelet/pods/9564d09c-f00e-48e9-bd57-a96f0bb6efc3/volumes"
Oct 30 13:21:44.332887 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:44.332852    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:44.459249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:44.452961    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:46.479916 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:46.479882    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:48.419051 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:48.418953    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:49.472741 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:49.472705    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:49.641675 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:49.641638    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:49.858757 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:49.858691    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:50.537326 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:50.537269    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:50.861501 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:50.861462    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn"
Oct 30 13:21:50.875456 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:50.875425    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn"
Oct 30 13:21:51.038595 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:51.038562    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn"
Oct 30 13:21:52.501698 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:52.501646    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:53.287180 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:53.287144    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:54.500672 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:54.500636    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:56.023805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:56.023769    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:56.152420 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:56.152384    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:56.263197 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:56.263158    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:57.194980 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:57.194947    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-578524af-ht57r"]
Oct 30 13:21:57.195976 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:57.195951    4544 cpu_manager.go:414] "containerMap: removing container" podUID="9564d09c-f00e-48e9-bd57-a96f0bb6efc3" containerName="payload-pre-processing"
Oct 30 13:21:57.196114 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:57.195976    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9564d09c-f00e-48e9-bd57-a96f0bb6efc3" containerName="payload-pre-processing"
Oct 30 13:21:57.196244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:57.196214    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="9564d09c-f00e-48e9-bd57-a96f0bb6efc3" containerName="payload-pre-processing"
Oct 30 13:21:57.201613 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:57.201592    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-578524af-ht57r"
Oct 30 13:21:57.220942 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:57.220908    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-578524af-ht57r"]
Oct 30 13:21:57.227702 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:57.227128    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:21:57.227702 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:57.227174    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:21:57.353145 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:57.353104    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de9588c5-4d8b-48b1-a56b-f7d8b2a15463-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-578524af-ht57r\" (UID: \"de9588c5-4d8b-48b1-a56b-f7d8b2a15463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578524af-ht57r"
Oct 30 13:21:57.353431 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:57.353413    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de9588c5-4d8b-48b1-a56b-f7d8b2a15463-maas-api-xks-ca\") pod \"maas-api-revoke-sub-578524af-ht57r\" (UID: \"de9588c5-4d8b-48b1-a56b-f7d8b2a15463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578524af-ht57r"
Oct 30 13:21:57.353481 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:57.353447    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbwcl\" (UniqueName: \"kubernetes.io/projected/de9588c5-4d8b-48b1-a56b-f7d8b2a15463-kube-api-access-cbwcl\") pod \"maas-api-revoke-sub-578524af-ht57r\" (UID: \"de9588c5-4d8b-48b1-a56b-f7d8b2a15463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578524af-ht57r"
Oct 30 13:21:57.454859 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:57.454767    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de9588c5-4d8b-48b1-a56b-f7d8b2a15463-maas-api-xks-ca\") pod \"maas-api-revoke-sub-578524af-ht57r\" (UID: \"de9588c5-4d8b-48b1-a56b-f7d8b2a15463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578524af-ht57r"
Oct 30 13:21:57.454859 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:57.454811    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbwcl\" (UniqueName: \"kubernetes.io/projected/de9588c5-4d8b-48b1-a56b-f7d8b2a15463-kube-api-access-cbwcl\") pod \"maas-api-revoke-sub-578524af-ht57r\" (UID: \"de9588c5-4d8b-48b1-a56b-f7d8b2a15463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578524af-ht57r"
Oct 30 13:21:57.455071 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:57.454923    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de9588c5-4d8b-48b1-a56b-f7d8b2a15463-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-578524af-ht57r\" (UID: \"de9588c5-4d8b-48b1-a56b-f7d8b2a15463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578524af-ht57r"
Oct 30 13:21:57.455801 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:57.455748    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de9588c5-4d8b-48b1-a56b-f7d8b2a15463-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-578524af-ht57r\" (UID: \"de9588c5-4d8b-48b1-a56b-f7d8b2a15463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578524af-ht57r"
Oct 30 13:21:57.457459 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:57.457433    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de9588c5-4d8b-48b1-a56b-f7d8b2a15463-maas-api-xks-ca\") pod \"maas-api-revoke-sub-578524af-ht57r\" (UID: \"de9588c5-4d8b-48b1-a56b-f7d8b2a15463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578524af-ht57r"
Oct 30 13:21:57.478748 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:57.478712    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbwcl\" (UniqueName: \"kubernetes.io/projected/de9588c5-4d8b-48b1-a56b-f7d8b2a15463-kube-api-access-cbwcl\") pod \"maas-api-revoke-sub-578524af-ht57r\" (UID: \"de9588c5-4d8b-48b1-a56b-f7d8b2a15463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578524af-ht57r"
Oct 30 13:21:57.517562 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:57.517524    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-578524af-ht57r"
Oct 30 13:21:57.697340 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:57.697254    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6"]
Oct 30 13:21:57.697938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:57.697883    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6" podUID="40c596cb-37ac-468d-828a-65708ed129f1" containerName="maas-api" containerID="cri-o://41f3b93a49e85a993611ba1dcea23d74022356084a07c5ab2e5a97f4925e3dcd" gracePeriod=30
Oct 30 13:21:57.746384 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:57.746349    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:57.827666 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:57.827637    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-578524af-ht57r"]
Oct 30 13:21:57.829058 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:21:57.829028    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde9588c5_4d8b_48b1_a56b_f7d8b2a15463.slice/crio-6bc3dea82bc56799d5a0b4adbd2db9b300b08f7e468ea7628187885c7bc41c56 WatchSource:0}: Error finding container 6bc3dea82bc56799d5a0b4adbd2db9b300b08f7e468ea7628187885c7bc41c56: Status 404 returned error can't find the container with id 6bc3dea82bc56799d5a0b4adbd2db9b300b08f7e468ea7628187885c7bc41c56
Oct 30 13:21:58.190244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:58.189658    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6"
Oct 30 13:21:58.190437 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:58.190417    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-578524af-ht57r" event={"ID":"de9588c5-4d8b-48b1-a56b-f7d8b2a15463","Type":"ContainerStarted","Data":"6bc3dea82bc56799d5a0b4adbd2db9b300b08f7e468ea7628187885c7bc41c56"}
Oct 30 13:21:58.244675 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:58.244642    4544 generic.go:356] "Generic (PLEG): container finished" podID="40c596cb-37ac-468d-828a-65708ed129f1" containerID="41f3b93a49e85a993611ba1dcea23d74022356084a07c5ab2e5a97f4925e3dcd" exitCode=0
Oct 30 13:21:58.245303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:58.245275    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6" event={"ID":"40c596cb-37ac-468d-828a-65708ed129f1","Type":"ContainerDied","Data":"41f3b93a49e85a993611ba1dcea23d74022356084a07c5ab2e5a97f4925e3dcd"}
Oct 30 13:21:58.245481 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:58.245466    4544 scope.go:122] "RemoveContainer" containerID="41f3b93a49e85a993611ba1dcea23d74022356084a07c5ab2e5a97f4925e3dcd"
Oct 30 13:21:58.245753 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:58.245730    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6"
Oct 30 13:21:58.333584 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:58.333543    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40c596cb-37ac-468d-828a-65708ed129f1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40c596cb-37ac-468d-828a-65708ed129f1-openshift-service-ca\") pod \"40c596cb-37ac-468d-828a-65708ed129f1\" (UID: \"40c596cb-37ac-468d-828a-65708ed129f1\") "
Oct 30 13:21:58.364614 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:58.364530    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40c596cb-37ac-468d-828a-65708ed129f1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/40c596cb-37ac-468d-828a-65708ed129f1-metrics-certs\") pod \"40c596cb-37ac-468d-828a-65708ed129f1\" (UID: \"40c596cb-37ac-468d-828a-65708ed129f1\") "
Oct 30 13:21:58.364878 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:58.364621    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40c596cb-37ac-468d-828a-65708ed129f1-kube-api-access-9x9hp\" (UniqueName: \"kubernetes.io/projected/40c596cb-37ac-468d-828a-65708ed129f1-kube-api-access-9x9hp\") pod \"40c596cb-37ac-468d-828a-65708ed129f1\" (UID: \"40c596cb-37ac-468d-828a-65708ed129f1\") "
Oct 30 13:21:58.364878 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:58.364701    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40c596cb-37ac-468d-828a-65708ed129f1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40c596cb-37ac-468d-828a-65708ed129f1-maas-api-tls\") pod \"40c596cb-37ac-468d-828a-65708ed129f1\" (UID: \"40c596cb-37ac-468d-828a-65708ed129f1\") "
Oct 30 13:21:58.373335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:58.337700    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c596cb-37ac-468d-828a-65708ed129f1-openshift-service-ca" pod "40c596cb-37ac-468d-828a-65708ed129f1" (UID: "40c596cb-37ac-468d-828a-65708ed129f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:21:58.384168 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:58.382640    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c596cb-37ac-468d-828a-65708ed129f1-kube-api-access-9x9hp" pod "40c596cb-37ac-468d-828a-65708ed129f1" (UID: "40c596cb-37ac-468d-828a-65708ed129f1"). InnerVolumeSpecName "kube-api-access-9x9hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:21:58.388086 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:58.387391    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40c596cb-37ac-468d-828a-65708ed129f1-metrics-certs" pod "40c596cb-37ac-468d-828a-65708ed129f1" (UID: "40c596cb-37ac-468d-828a-65708ed129f1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:21:58.388757 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:58.388490    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40c596cb-37ac-468d-828a-65708ed129f1-maas-api-tls" pod "40c596cb-37ac-468d-828a-65708ed129f1" (UID: "40c596cb-37ac-468d-828a-65708ed129f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:21:58.467576 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:58.467537    4544 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40c596cb-37ac-468d-828a-65708ed129f1-metrics-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:21:58.467576 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:58.467574    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x9hp\" (UniqueName: \"kubernetes.io/projected/40c596cb-37ac-468d-828a-65708ed129f1-kube-api-access-9x9hp\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:21:58.467818 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:58.467586    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40c596cb-37ac-468d-828a-65708ed129f1-maas-api-tls\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:21:58.467818 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:58.467597    4544 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40c596cb-37ac-468d-828a-65708ed129f1-openshift-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:21:58.595934 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:58.595837    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6"]
Oct 30 13:21:58.599963 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:58.599935    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6f9071-6464d4dd6c-dpdq6"]
Oct 30 13:21:58.601379 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:58.601352    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:58.846504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:58.846409    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:58.867024 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:58.866981    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f9071-qxtqd"]
Oct 30 13:21:58.871337 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:58.871305    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f9071-qxtqd"]
Oct 30 13:21:58.989929 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:58.989888    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:21:59.029723 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:21:59.029691    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:21:59.089270 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:59.089232    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40c596cb-37ac-468d-828a-65708ed129f1" path="/var/lib/kubelet/pods/40c596cb-37ac-468d-828a-65708ed129f1/volumes"
Oct 30 13:21:59.089744 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:59.089730    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c07719a1-a141-463d-9191-c060d4ca35c1" path="/var/lib/kubelet/pods/c07719a1-a141-463d-9191-c060d4ca35c1/volumes"
Oct 30 13:21:59.256642 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:59.256551    4544 generic.go:356] "Generic (PLEG): container finished" podID="de9588c5-4d8b-48b1-a56b-f7d8b2a15463" containerID="0182b2d67ba8785b0e530622213cbd0c6b600f2acd52433c11c29076b3cfcc23" exitCode=0
Oct 30 13:21:59.257029 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:59.256650    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-578524af-ht57r" event={"ID":"de9588c5-4d8b-48b1-a56b-f7d8b2a15463","Type":"ContainerDied","Data":"0182b2d67ba8785b0e530622213cbd0c6b600f2acd52433c11c29076b3cfcc23"}
Oct 30 13:21:59.421919 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:21:59.421862    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:00.448932 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:00.448907    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-578524af-ht57r"
Oct 30 13:22:00.499689 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:00.499653    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de9588c5-4d8b-48b1-a56b-f7d8b2a15463-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de9588c5-4d8b-48b1-a56b-f7d8b2a15463-maas-api-xks-ca\") pod \"de9588c5-4d8b-48b1-a56b-f7d8b2a15463\" (UID: \"de9588c5-4d8b-48b1-a56b-f7d8b2a15463\") "
Oct 30 13:22:00.499868 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:00.499856    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de9588c5-4d8b-48b1-a56b-f7d8b2a15463-kube-api-access-cbwcl\" (UniqueName: \"kubernetes.io/projected/de9588c5-4d8b-48b1-a56b-f7d8b2a15463-kube-api-access-cbwcl\") pod \"de9588c5-4d8b-48b1-a56b-f7d8b2a15463\" (UID: \"de9588c5-4d8b-48b1-a56b-f7d8b2a15463\") "
Oct 30 13:22:00.499915 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:00.499894    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de9588c5-4d8b-48b1-a56b-f7d8b2a15463-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de9588c5-4d8b-48b1-a56b-f7d8b2a15463-maas-api-openshift-ca\") pod \"de9588c5-4d8b-48b1-a56b-f7d8b2a15463\" (UID: \"de9588c5-4d8b-48b1-a56b-f7d8b2a15463\") "
Oct 30 13:22:00.500345 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:00.500309    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de9588c5-4d8b-48b1-a56b-f7d8b2a15463-maas-api-openshift-ca" pod "de9588c5-4d8b-48b1-a56b-f7d8b2a15463" (UID: "de9588c5-4d8b-48b1-a56b-f7d8b2a15463"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:22:00.500473 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:00.500406    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de9588c5-4d8b-48b1-a56b-f7d8b2a15463-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:22:00.501900 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:00.501870    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de9588c5-4d8b-48b1-a56b-f7d8b2a15463-maas-api-xks-ca" pod "de9588c5-4d8b-48b1-a56b-f7d8b2a15463" (UID: "de9588c5-4d8b-48b1-a56b-f7d8b2a15463"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:22:00.502141 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:00.502116    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de9588c5-4d8b-48b1-a56b-f7d8b2a15463-kube-api-access-cbwcl" pod "de9588c5-4d8b-48b1-a56b-f7d8b2a15463" (UID: "de9588c5-4d8b-48b1-a56b-f7d8b2a15463"). InnerVolumeSpecName "kube-api-access-cbwcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:22:00.601991 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:00.601954    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbwcl\" (UniqueName: \"kubernetes.io/projected/de9588c5-4d8b-48b1-a56b-f7d8b2a15463-kube-api-access-cbwcl\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:22:00.601991 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:00.601988    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de9588c5-4d8b-48b1-a56b-f7d8b2a15463-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:22:00.866289 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:00.866165    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:01.277162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:01.277134    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-578524af-ht57r"
Oct 30 13:22:01.277162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:01.277134    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-578524af-ht57r" event={"ID":"de9588c5-4d8b-48b1-a56b-f7d8b2a15463","Type":"ContainerDied","Data":"6bc3dea82bc56799d5a0b4adbd2db9b300b08f7e468ea7628187885c7bc41c56"}
Oct 30 13:22:01.277420 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:01.277202    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bc3dea82bc56799d5a0b4adbd2db9b300b08f7e468ea7628187885c7bc41c56"
Oct 30 13:22:02.869683 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:02.869646    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:04.685860 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:04.685813    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:06.141670 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:06.141638    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:06.252919 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:06.252887    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:06.397982 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:06.397892    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:06.486274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:06.486234    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"]
Oct 30 13:22:06.486548 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:06.486511    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj" podUID="1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff" containerName="istio-proxy" containerID="cri-o://fbcba18ac868587aa34f549c3c9acb5f52895a6da4de09944ec74e4398b89bbd" gracePeriod=30
Oct 30 13:22:06.528899 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:06.528864    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:08.325285 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:08.325243    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:09.277900 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:09.277862    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r"]
Oct 30 13:22:09.332825 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:09.332782    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:09.843946 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:09.843908    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5b82fd8-trfr8"]
Oct 30 13:22:09.845604 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:09.845565    4544 cpu_manager.go:414] "containerMap: removing container" podUID="40c596cb-37ac-468d-828a-65708ed129f1" containerName="maas-api"
Oct 30 13:22:09.845604 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:09.845599    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40c596cb-37ac-468d-828a-65708ed129f1" containerName="maas-api"
Oct 30 13:22:09.845831 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:09.845619    4544 cpu_manager.go:414] "containerMap: removing container" podUID="de9588c5-4d8b-48b1-a56b-f7d8b2a15463" containerName="revoke-keys"
Oct 30 13:22:09.845831 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:09.845627    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de9588c5-4d8b-48b1-a56b-f7d8b2a15463" containerName="revoke-keys"
Oct 30 13:22:09.845943 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:09.845916    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="40c596cb-37ac-468d-828a-65708ed129f1" containerName="maas-api"
Oct 30 13:22:09.845943 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:09.845934    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="de9588c5-4d8b-48b1-a56b-f7d8b2a15463" containerName="revoke-keys"
Oct 30 13:22:09.850385 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:09.850363    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5b82fd8-trfr8"
Oct 30 13:22:09.867158 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:09.867121    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:22:09.873623 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:09.873588    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5b82fd8-trfr8"]
Oct 30 13:22:09.874037 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:09.874011    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:22:09.930676 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:09.930629    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c6ks\" (UniqueName: \"kubernetes.io/projected/1b68f691-8b13-4422-9f40-4461ee58128d-kube-api-access-6c6ks\") pod \"maas-api-revoke-sub-d5b82fd8-trfr8\" (UID: \"1b68f691-8b13-4422-9f40-4461ee58128d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5b82fd8-trfr8"
Oct 30 13:22:09.931489 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:09.930772    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b68f691-8b13-4422-9f40-4461ee58128d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5b82fd8-trfr8\" (UID: \"1b68f691-8b13-4422-9f40-4461ee58128d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5b82fd8-trfr8"
Oct 30 13:22:09.940139 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:09.940095    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b68f691-8b13-4422-9f40-4461ee58128d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5b82fd8-trfr8\" (UID: \"1b68f691-8b13-4422-9f40-4461ee58128d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5b82fd8-trfr8"
Oct 30 13:22:10.041644 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:10.041590    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c6ks\" (UniqueName: \"kubernetes.io/projected/1b68f691-8b13-4422-9f40-4461ee58128d-kube-api-access-6c6ks\") pod \"maas-api-revoke-sub-d5b82fd8-trfr8\" (UID: \"1b68f691-8b13-4422-9f40-4461ee58128d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5b82fd8-trfr8"
Oct 30 13:22:10.041836 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:10.041735    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b68f691-8b13-4422-9f40-4461ee58128d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5b82fd8-trfr8\" (UID: \"1b68f691-8b13-4422-9f40-4461ee58128d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5b82fd8-trfr8"
Oct 30 13:22:10.041916 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:10.041856    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b68f691-8b13-4422-9f40-4461ee58128d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5b82fd8-trfr8\" (UID: \"1b68f691-8b13-4422-9f40-4461ee58128d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5b82fd8-trfr8"
Oct 30 13:22:10.042444 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:10.042416    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b68f691-8b13-4422-9f40-4461ee58128d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5b82fd8-trfr8\" (UID: \"1b68f691-8b13-4422-9f40-4461ee58128d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5b82fd8-trfr8"
Oct 30 13:22:10.044397 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:10.044371    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b68f691-8b13-4422-9f40-4461ee58128d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5b82fd8-trfr8\" (UID: \"1b68f691-8b13-4422-9f40-4461ee58128d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5b82fd8-trfr8"
Oct 30 13:22:10.066337 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:10.066306    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c6ks\" (UniqueName: \"kubernetes.io/projected/1b68f691-8b13-4422-9f40-4461ee58128d-kube-api-access-6c6ks\") pod \"maas-api-revoke-sub-d5b82fd8-trfr8\" (UID: \"1b68f691-8b13-4422-9f40-4461ee58128d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5b82fd8-trfr8"
Oct 30 13:22:10.099955 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:10.099879    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:10.173243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:10.172821    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5b82fd8-trfr8"
Oct 30 13:22:10.510423 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:10.510391    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5b82fd8-trfr8"]
Oct 30 13:22:10.512379 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:22:10.512353    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b68f691_8b13_4422_9f40_4461ee58128d.slice/crio-ca0a8b27e7e6b2fe7e374ccde0b19165fccd9cb0d72b61e2f956ba19972be1d6 WatchSource:0}: Error finding container ca0a8b27e7e6b2fe7e374ccde0b19165fccd9cb0d72b61e2f956ba19972be1d6: Status 404 returned error can't find the container with id ca0a8b27e7e6b2fe7e374ccde0b19165fccd9cb0d72b61e2f956ba19972be1d6
Oct 30 13:22:11.368752 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:11.368719    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5b82fd8-trfr8" event={"ID":"1b68f691-8b13-4422-9f40-4461ee58128d","Type":"ContainerStarted","Data":"b99220279d32c63d51786be6dc6fe616fc1b19b92f67a72ad56dfbe0f284d7f7"}
Oct 30 13:22:11.368961 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:11.368777    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5b82fd8-trfr8" event={"ID":"1b68f691-8b13-4422-9f40-4461ee58128d","Type":"ContainerStarted","Data":"ca0a8b27e7e6b2fe7e374ccde0b19165fccd9cb0d72b61e2f956ba19972be1d6"}
Oct 30 13:22:11.412145 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:11.412089    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5b82fd8-trfr8" podStartSLOduration=2.412075336 podStartE2EDuration="2.412075336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:11.409046324 +0000 UTC m=+2226.456180280" watchObservedRunningTime="2026-10-30 13:22:11.412075336 +0000 UTC m=+2226.459209289"
Oct 30 13:22:11.771341 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:11.771309    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:22:11.859814 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:11.859781    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:11.877282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:11.877213    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istiod-ca-cert\") pod \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") "
Oct 30 13:22:11.877461 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:11.877314    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-kube-api-access-rmjzb\" (UniqueName: \"kubernetes.io/projected/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-kube-api-access-rmjzb\") pod \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") "
Oct 30 13:22:11.877461 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:11.877381    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istio-podinfo\") pod \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") "
Oct 30 13:22:11.877594 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:11.877493    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-workload-certs\") pod \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") "
Oct 30 13:22:11.877594 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:11.877574    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-credential-socket\") pod \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") "
Oct 30 13:22:11.877709 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:11.877642    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istiod-ca-cert" pod "1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff" (UID: "1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:22:11.877835 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:11.877814    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istio-token\" (UniqueName: \"kubernetes.io/projected/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istio-token\") pod \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") "
Oct 30 13:22:11.877901 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:11.877854    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istio-data\") pod \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") "
Oct 30 13:22:11.877901 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:11.877882    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-workload-socket\") pod \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") "
Oct 30 13:22:11.878019 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:11.877936    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istio-envoy\") pod \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\" (UID: \"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff\") "
Oct 30 13:22:11.878019 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:11.877954    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-workload-certs" pod "1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff" (UID: "1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:22:11.880800 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:11.878455    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-credential-socket" pod "1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff" (UID: "1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:22:11.880800 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:11.878629    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-workload-socket" pod "1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff" (UID: "1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:22:11.880800 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:11.878642    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:22:11.880800 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:11.878667    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:22:11.880800 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:11.878681    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:22:11.880800 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:11.878805    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istio-data" pod "1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff" (UID: "1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:22:11.880800 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:11.880115    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-kube-api-access-rmjzb" pod "1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff" (UID: "1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff"). InnerVolumeSpecName "kube-api-access-rmjzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:22:11.880800 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:11.880490    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istio-podinfo" pod "1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff" (UID: "1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:22:11.881172 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:11.880907    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istio-envoy" pod "1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff" (UID: "1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:22:11.881172 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:11.880933    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istio-token" pod "1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff" (UID: "1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:22:11.980682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:11.980602    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:22:11.980682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:11.980630    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:22:11.980682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:11.980639    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmjzb\" (UniqueName: \"kubernetes.io/projected/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-kube-api-access-rmjzb\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:22:11.980682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:11.980650    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:22:11.980682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:11.980660    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:22:11.980682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:11.980667    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:22:12.378856 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:12.378826    4544 generic.go:356] "Generic (PLEG): container finished" podID="1b68f691-8b13-4422-9f40-4461ee58128d" containerID="b99220279d32c63d51786be6dc6fe616fc1b19b92f67a72ad56dfbe0f284d7f7" exitCode=0
Oct 30 13:22:12.379057 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:12.378915    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5b82fd8-trfr8" event={"ID":"1b68f691-8b13-4422-9f40-4461ee58128d","Type":"ContainerDied","Data":"b99220279d32c63d51786be6dc6fe616fc1b19b92f67a72ad56dfbe0f284d7f7"}
Oct 30 13:22:12.380706 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:12.380681    4544 generic.go:356] "Generic (PLEG): container finished" podID="1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff" containerID="fbcba18ac868587aa34f549c3c9acb5f52895a6da4de09944ec74e4398b89bbd" exitCode=0
Oct 30 13:22:12.380785 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:12.380732    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"
Oct 30 13:22:12.380785 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:12.380759    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj" event={"ID":"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff","Type":"ContainerDied","Data":"fbcba18ac868587aa34f549c3c9acb5f52895a6da4de09944ec74e4398b89bbd"}
Oct 30 13:22:12.380850 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:12.380802    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj" event={"ID":"1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff","Type":"ContainerDied","Data":"92858d1aa2f8579ae45405cf9ab499ae79e94188744076ec1390f74675b71668"}
Oct 30 13:22:12.380850 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:12.380824    4544 scope.go:122] "RemoveContainer" containerID="fbcba18ac868587aa34f549c3c9acb5f52895a6da4de09944ec74e4398b89bbd"
Oct 30 13:22:12.393561 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:12.393546    4544 scope.go:122] "RemoveContainer" containerID="0e84581c5e263c206940181471dcfde276bbb741244437eeab53f8d17892d386"
Oct 30 13:22:12.403385 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:12.403369    4544 scope.go:122] "RemoveContainer" containerID="fbcba18ac868587aa34f549c3c9acb5f52895a6da4de09944ec74e4398b89bbd"
Oct 30 13:22:12.403646 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:22:12.403623    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbcba18ac868587aa34f549c3c9acb5f52895a6da4de09944ec74e4398b89bbd\": container with ID starting with fbcba18ac868587aa34f549c3c9acb5f52895a6da4de09944ec74e4398b89bbd not found: ID does not exist" containerID="fbcba18ac868587aa34f549c3c9acb5f52895a6da4de09944ec74e4398b89bbd"
Oct 30 13:22:12.403696 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:12.403655    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbcba18ac868587aa34f549c3c9acb5f52895a6da4de09944ec74e4398b89bbd"} err="failed to get container status \"fbcba18ac868587aa34f549c3c9acb5f52895a6da4de09944ec74e4398b89bbd\": rpc error: code = NotFound desc = could not find container \"fbcba18ac868587aa34f549c3c9acb5f52895a6da4de09944ec74e4398b89bbd\": container with ID starting with fbcba18ac868587aa34f549c3c9acb5f52895a6da4de09944ec74e4398b89bbd not found: ID does not exist"
Oct 30 13:22:12.403696 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:12.403670    4544 scope.go:122] "RemoveContainer" containerID="0e84581c5e263c206940181471dcfde276bbb741244437eeab53f8d17892d386"
Oct 30 13:22:12.403919 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:22:12.403893    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e84581c5e263c206940181471dcfde276bbb741244437eeab53f8d17892d386\": container with ID starting with 0e84581c5e263c206940181471dcfde276bbb741244437eeab53f8d17892d386 not found: ID does not exist" containerID="0e84581c5e263c206940181471dcfde276bbb741244437eeab53f8d17892d386"
Oct 30 13:22:12.403962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:12.403924    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e84581c5e263c206940181471dcfde276bbb741244437eeab53f8d17892d386"} err="failed to get container status \"0e84581c5e263c206940181471dcfde276bbb741244437eeab53f8d17892d386\": rpc error: code = NotFound desc = could not find container \"0e84581c5e263c206940181471dcfde276bbb741244437eeab53f8d17892d386\": container with ID starting with 0e84581c5e263c206940181471dcfde276bbb741244437eeab53f8d17892d386 not found: ID does not exist"
Oct 30 13:22:12.463147 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:12.463118    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"]
Oct 30 13:22:12.469260 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:12.469232    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj"]
Oct 30 13:22:13.088680 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:13.088650    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff" path="/var/lib/kubelet/pods/1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff/volumes"
Oct 30 13:22:13.559741 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:13.559713    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5b82fd8-trfr8"
Oct 30 13:22:13.611296 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:13.611262    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b68f691-8b13-4422-9f40-4461ee58128d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b68f691-8b13-4422-9f40-4461ee58128d-maas-api-xks-ca\") pod \"1b68f691-8b13-4422-9f40-4461ee58128d\" (UID: \"1b68f691-8b13-4422-9f40-4461ee58128d\") "
Oct 30 13:22:13.611457 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:13.611377    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b68f691-8b13-4422-9f40-4461ee58128d-kube-api-access-6c6ks\" (UniqueName: \"kubernetes.io/projected/1b68f691-8b13-4422-9f40-4461ee58128d-kube-api-access-6c6ks\") pod \"1b68f691-8b13-4422-9f40-4461ee58128d\" (UID: \"1b68f691-8b13-4422-9f40-4461ee58128d\") "
Oct 30 13:22:13.611640 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:13.611622    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b68f691-8b13-4422-9f40-4461ee58128d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b68f691-8b13-4422-9f40-4461ee58128d-maas-api-openshift-ca\") pod \"1b68f691-8b13-4422-9f40-4461ee58128d\" (UID: \"1b68f691-8b13-4422-9f40-4461ee58128d\") "
Oct 30 13:22:13.611947 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:13.611924    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b68f691-8b13-4422-9f40-4461ee58128d-maas-api-openshift-ca" pod "1b68f691-8b13-4422-9f40-4461ee58128d" (UID: "1b68f691-8b13-4422-9f40-4461ee58128d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:22:13.612252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:13.612235    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b68f691-8b13-4422-9f40-4461ee58128d-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:22:13.613480 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:13.613458    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b68f691-8b13-4422-9f40-4461ee58128d-kube-api-access-6c6ks" pod "1b68f691-8b13-4422-9f40-4461ee58128d" (UID: "1b68f691-8b13-4422-9f40-4461ee58128d"). InnerVolumeSpecName "kube-api-access-6c6ks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:22:13.613742 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:13.613726    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b68f691-8b13-4422-9f40-4461ee58128d-maas-api-xks-ca" pod "1b68f691-8b13-4422-9f40-4461ee58128d" (UID: "1b68f691-8b13-4422-9f40-4461ee58128d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:22:13.714128 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:13.714047    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b68f691-8b13-4422-9f40-4461ee58128d-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:22:13.714128 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:13.714091    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c6ks\" (UniqueName: \"kubernetes.io/projected/1b68f691-8b13-4422-9f40-4461ee58128d-kube-api-access-6c6ks\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:22:13.979001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:13.978903    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:14.400032 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:14.399999    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5b82fd8-trfr8"
Oct 30 13:22:14.400032 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:14.400010    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5b82fd8-trfr8" event={"ID":"1b68f691-8b13-4422-9f40-4461ee58128d","Type":"ContainerDied","Data":"ca0a8b27e7e6b2fe7e374ccde0b19165fccd9cb0d72b61e2f956ba19972be1d6"}
Oct 30 13:22:14.400513 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:14.400064    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca0a8b27e7e6b2fe7e374ccde0b19165fccd9cb0d72b61e2f956ba19972be1d6"
Oct 30 13:22:16.090202 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:16.090169    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:17.989043 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:17.989005    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:20.133245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:20.131689    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:21.937311 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:21.937264    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:24.035243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:24.032924    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:24.300362 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:24.300276    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r" podUID="0a673362-df63-4d76-81fa-b35ade9305ef" containerName="main" containerID="cri-o://5ea3acfa3f7ca3b951f59aed5cde0d5658ebfc7c63dc5714fa6ac7ada4a3fa5e" gracePeriod=45
Oct 30 13:22:24.514813 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:24.514774    4544 generic.go:356] "Generic (PLEG): container finished" podID="0a673362-df63-4d76-81fa-b35ade9305ef" containerID="5ea3acfa3f7ca3b951f59aed5cde0d5658ebfc7c63dc5714fa6ac7ada4a3fa5e" exitCode=0
Oct 30 13:22:24.515005 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:24.514901    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r" event={"ID":"0a673362-df63-4d76-81fa-b35ade9305ef","Type":"ContainerDied","Data":"5ea3acfa3f7ca3b951f59aed5cde0d5658ebfc7c63dc5714fa6ac7ada4a3fa5e"}
Oct 30 13:22:24.596967 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:24.596933    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r"
Oct 30 13:22:24.640411 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:24.640376    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a673362-df63-4d76-81fa-b35ade9305ef-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a673362-df63-4d76-81fa-b35ade9305ef-dshm\") pod \"0a673362-df63-4d76-81fa-b35ade9305ef\" (UID: \"0a673362-df63-4d76-81fa-b35ade9305ef\") "
Oct 30 13:22:24.640598 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:24.640445    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a673362-df63-4d76-81fa-b35ade9305ef-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a673362-df63-4d76-81fa-b35ade9305ef-tmp-dir\") pod \"0a673362-df63-4d76-81fa-b35ade9305ef\" (UID: \"0a673362-df63-4d76-81fa-b35ade9305ef\") "
Oct 30 13:22:24.640598 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:24.640558    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a673362-df63-4d76-81fa-b35ade9305ef-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a673362-df63-4d76-81fa-b35ade9305ef-model-cache\") pod \"0a673362-df63-4d76-81fa-b35ade9305ef\" (UID: \"0a673362-df63-4d76-81fa-b35ade9305ef\") "
Oct 30 13:22:24.640598 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:24.640597    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a673362-df63-4d76-81fa-b35ade9305ef-home\" (UniqueName: \"kubernetes.io/empty-dir/0a673362-df63-4d76-81fa-b35ade9305ef-home\") pod \"0a673362-df63-4d76-81fa-b35ade9305ef\" (UID: \"0a673362-df63-4d76-81fa-b35ade9305ef\") "
Oct 30 13:22:24.640787 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:24.640646    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a673362-df63-4d76-81fa-b35ade9305ef-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a673362-df63-4d76-81fa-b35ade9305ef-tls-certs\") pod \"0a673362-df63-4d76-81fa-b35ade9305ef\" (UID: \"0a673362-df63-4d76-81fa-b35ade9305ef\") "
Oct 30 13:22:24.640787 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:24.640680    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a673362-df63-4d76-81fa-b35ade9305ef-kube-api-access-tzb99\" (UniqueName: \"kubernetes.io/projected/0a673362-df63-4d76-81fa-b35ade9305ef-kube-api-access-tzb99\") pod \"0a673362-df63-4d76-81fa-b35ade9305ef\" (UID: \"0a673362-df63-4d76-81fa-b35ade9305ef\") "
Oct 30 13:22:24.640908 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:24.640776    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a673362-df63-4d76-81fa-b35ade9305ef-tmp-dir" pod "0a673362-df63-4d76-81fa-b35ade9305ef" (UID: "0a673362-df63-4d76-81fa-b35ade9305ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:22:24.640908 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:24.640815    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a673362-df63-4d76-81fa-b35ade9305ef-model-cache" pod "0a673362-df63-4d76-81fa-b35ade9305ef" (UID: "0a673362-df63-4d76-81fa-b35ade9305ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:22:24.641014 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:24.640918    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a673362-df63-4d76-81fa-b35ade9305ef-home" pod "0a673362-df63-4d76-81fa-b35ade9305ef" (UID: "0a673362-df63-4d76-81fa-b35ade9305ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:22:24.641607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:24.641585    4544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a673362-df63-4d76-81fa-b35ade9305ef-tmp-dir\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:22:24.641607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:24.641605    4544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a673362-df63-4d76-81fa-b35ade9305ef-model-cache\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:22:24.641607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:24.641614    4544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a673362-df63-4d76-81fa-b35ade9305ef-home\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:22:24.645605 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:24.643019    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a673362-df63-4d76-81fa-b35ade9305ef-tls-certs" pod "0a673362-df63-4d76-81fa-b35ade9305ef" (UID: "0a673362-df63-4d76-81fa-b35ade9305ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:22:24.645605 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:24.643025    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a673362-df63-4d76-81fa-b35ade9305ef-dshm" pod "0a673362-df63-4d76-81fa-b35ade9305ef" (UID: "0a673362-df63-4d76-81fa-b35ade9305ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:22:24.645605 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:24.643116    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a673362-df63-4d76-81fa-b35ade9305ef-kube-api-access-tzb99" pod "0a673362-df63-4d76-81fa-b35ade9305ef" (UID: "0a673362-df63-4d76-81fa-b35ade9305ef"). InnerVolumeSpecName "kube-api-access-tzb99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:22:24.742766 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:24.742730    4544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a673362-df63-4d76-81fa-b35ade9305ef-dshm\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:22:24.742766 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:24.742759    4544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a673362-df63-4d76-81fa-b35ade9305ef-tls-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:22:24.742766 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:24.742769    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzb99\" (UniqueName: \"kubernetes.io/projected/0a673362-df63-4d76-81fa-b35ade9305ef-kube-api-access-tzb99\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:22:25.524593 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:25.524516    4544 generic.go:356] "Generic (PLEG): container finished" podID="bd4c038f-d0d4-4fd5-ad51-2dbae210cb26" containerID="8065e62d66aaf050576444ef5dd78cae779752f4f4e2c60b2f551ced1d07dd5d" exitCode=0
Oct 30 13:22:25.525063 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:25.524583    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl" event={"ID":"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26","Type":"ContainerDied","Data":"8065e62d66aaf050576444ef5dd78cae779752f4f4e2c60b2f551ced1d07dd5d"}
Oct 30 13:22:25.525063 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:25.525059    4544 scope.go:122] "RemoveContainer" containerID="8065e62d66aaf050576444ef5dd78cae779752f4f4e2c60b2f551ced1d07dd5d"
Oct 30 13:22:25.526171 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:25.526149    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r"
Oct 30 13:22:25.526252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:25.526202    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r" event={"ID":"0a673362-df63-4d76-81fa-b35ade9305ef","Type":"ContainerDied","Data":"e8ed30d39d7c81492c91efb98d8e77b79ce2d61c4e3031c6555b98a3f22b4577"}
Oct 30 13:22:25.526306 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:25.526251    4544 scope.go:122] "RemoveContainer" containerID="5ea3acfa3f7ca3b951f59aed5cde0d5658ebfc7c63dc5714fa6ac7ada4a3fa5e"
Oct 30 13:22:25.527242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:25.527189    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:22:25.527454 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:25.527300    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:22:25.527454 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:25.527353    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:22:25.620252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:25.620207    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r"]
Oct 30 13:22:25.627787 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:25.627759    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-6af611/test-model-6af611-kserve-5b6bbd8ddf-8ds2r"]
Oct 30 13:22:25.656892 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:25.656856    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:26.544299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:26.544260    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl" event={"ID":"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26","Type":"ContainerStarted","Data":"ad78e25267fb73c05e7e65d6e5d7fb1a098a6d5988b0134d3255e6b807b0d621"}
Oct 30 13:22:26.589963 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:26.589932    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:22:26.590167 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:26.590142    4544 patch_prober.go:29] interesting pod/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 13:22:26.590232 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:26.590205    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl" podUID="bd4c038f-d0d4-4fd5-ad51-2dbae210cb26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 13:22:27.089727 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.089692    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a673362-df63-4d76-81fa-b35ade9305ef" path="/var/lib/kubelet/pods/0a673362-df63-4d76-81fa-b35ade9305ef/volumes"
Oct 30 13:22:27.268884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.268850    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj"]
Oct 30 13:22:27.270060 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.270037    4544 cpu_manager.go:414] "containerMap: removing container" podUID="1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff" containerName="istio-proxy"
Oct 30 13:22:27.270168 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.270062    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff" containerName="istio-proxy"
Oct 30 13:22:27.270168 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.270088    4544 cpu_manager.go:414] "containerMap: removing container" podUID="1b68f691-8b13-4422-9f40-4461ee58128d" containerName="revoke-keys"
Oct 30 13:22:27.270168 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.270096    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b68f691-8b13-4422-9f40-4461ee58128d" containerName="revoke-keys"
Oct 30 13:22:27.270168 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.270153    4544 cpu_manager.go:414] "containerMap: removing container" podUID="0a673362-df63-4d76-81fa-b35ade9305ef" containerName="main"
Oct 30 13:22:27.270168 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.270161    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a673362-df63-4d76-81fa-b35ade9305ef" containerName="main"
Oct 30 13:22:27.270398 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.270367    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a673362-df63-4d76-81fa-b35ade9305ef" containerName="main"
Oct 30 13:22:27.270398 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.270382    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff" containerName="istio-proxy"
Oct 30 13:22:27.270491 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.270398    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff" containerName="istio-proxy"
Oct 30 13:22:27.270491 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.270413    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b68f691-8b13-4422-9f40-4461ee58128d" containerName="revoke-keys"
Oct 30 13:22:27.284590 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.284552    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj"
Oct 30 13:22:27.294191 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.294161    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj"]
Oct 30 13:22:27.335505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.334342    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-bcd76c\"/\"openshift-service-ca.crt\""
Oct 30 13:22:27.335505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.334457    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-bcd76c\"/\"kube-root-ca.crt\""
Oct 30 13:22:27.346569 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.346459    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-bcd76c\"/\"auth-test-model-bcd76c-kserve-self-signed-certs\""
Oct 30 13:22:27.346874 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.346591    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-bcd76c\"/\"default-dockercfg-p4hx7\""
Oct 30 13:22:27.358915 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.358877    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:27.395981 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.395955    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21ffd071-46da-4940-879a-2877eca7d1fe-dshm\") pod \"auth-test-model-bcd76c-kserve-665b758b77-bppbj\" (UID: \"21ffd071-46da-4940-879a-2877eca7d1fe\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj"
Oct 30 13:22:27.396133 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.395993    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21ffd071-46da-4940-879a-2877eca7d1fe-tls-certs\") pod \"auth-test-model-bcd76c-kserve-665b758b77-bppbj\" (UID: \"21ffd071-46da-4940-879a-2877eca7d1fe\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj"
Oct 30 13:22:27.396179 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.396135    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21ffd071-46da-4940-879a-2877eca7d1fe-home\") pod \"auth-test-model-bcd76c-kserve-665b758b77-bppbj\" (UID: \"21ffd071-46da-4940-879a-2877eca7d1fe\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj"
Oct 30 13:22:27.396312 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.396208    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ffd071-46da-4940-879a-2877eca7d1fe-tmp-dir\") pod \"auth-test-model-bcd76c-kserve-665b758b77-bppbj\" (UID: \"21ffd071-46da-4940-879a-2877eca7d1fe\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj"
Oct 30 13:22:27.396378 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.396349    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ffd071-46da-4940-879a-2877eca7d1fe-model-cache\") pod \"auth-test-model-bcd76c-kserve-665b758b77-bppbj\" (UID: \"21ffd071-46da-4940-879a-2877eca7d1fe\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj"
Oct 30 13:22:27.396483 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.396466    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjg44\" (UniqueName: \"kubernetes.io/projected/21ffd071-46da-4940-879a-2877eca7d1fe-kube-api-access-hjg44\") pod \"auth-test-model-bcd76c-kserve-665b758b77-bppbj\" (UID: \"21ffd071-46da-4940-879a-2877eca7d1fe\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj"
Oct 30 13:22:27.474280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.474210    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:27.497695 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.497669    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ffd071-46da-4940-879a-2877eca7d1fe-tmp-dir\") pod \"auth-test-model-bcd76c-kserve-665b758b77-bppbj\" (UID: \"21ffd071-46da-4940-879a-2877eca7d1fe\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj"
Oct 30 13:22:27.497815 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.497739    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ffd071-46da-4940-879a-2877eca7d1fe-model-cache\") pod \"auth-test-model-bcd76c-kserve-665b758b77-bppbj\" (UID: \"21ffd071-46da-4940-879a-2877eca7d1fe\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj"
Oct 30 13:22:27.497861 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.497814    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjg44\" (UniqueName: \"kubernetes.io/projected/21ffd071-46da-4940-879a-2877eca7d1fe-kube-api-access-hjg44\") pod \"auth-test-model-bcd76c-kserve-665b758b77-bppbj\" (UID: \"21ffd071-46da-4940-879a-2877eca7d1fe\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj"
Oct 30 13:22:27.497912 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.497882    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21ffd071-46da-4940-879a-2877eca7d1fe-dshm\") pod \"auth-test-model-bcd76c-kserve-665b758b77-bppbj\" (UID: \"21ffd071-46da-4940-879a-2877eca7d1fe\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj"
Oct 30 13:22:27.497912 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.497902    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21ffd071-46da-4940-879a-2877eca7d1fe-tls-certs\") pod \"auth-test-model-bcd76c-kserve-665b758b77-bppbj\" (UID: \"21ffd071-46da-4940-879a-2877eca7d1fe\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj"
Oct 30 13:22:27.498012 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.497975    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21ffd071-46da-4940-879a-2877eca7d1fe-home\") pod \"auth-test-model-bcd76c-kserve-665b758b77-bppbj\" (UID: \"21ffd071-46da-4940-879a-2877eca7d1fe\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj"
Oct 30 13:22:27.498117 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.498092    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ffd071-46da-4940-879a-2877eca7d1fe-tmp-dir\") pod \"auth-test-model-bcd76c-kserve-665b758b77-bppbj\" (UID: \"21ffd071-46da-4940-879a-2877eca7d1fe\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj"
Oct 30 13:22:27.498209 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.498125    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ffd071-46da-4940-879a-2877eca7d1fe-model-cache\") pod \"auth-test-model-bcd76c-kserve-665b758b77-bppbj\" (UID: \"21ffd071-46da-4940-879a-2877eca7d1fe\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj"
Oct 30 13:22:27.498342 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.498307    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21ffd071-46da-4940-879a-2877eca7d1fe-home\") pod \"auth-test-model-bcd76c-kserve-665b758b77-bppbj\" (UID: \"21ffd071-46da-4940-879a-2877eca7d1fe\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj"
Oct 30 13:22:27.500317 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.500293    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21ffd071-46da-4940-879a-2877eca7d1fe-dshm\") pod \"auth-test-model-bcd76c-kserve-665b758b77-bppbj\" (UID: \"21ffd071-46da-4940-879a-2877eca7d1fe\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj"
Oct 30 13:22:27.500669 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.500623    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21ffd071-46da-4940-879a-2877eca7d1fe-tls-certs\") pod \"auth-test-model-bcd76c-kserve-665b758b77-bppbj\" (UID: \"21ffd071-46da-4940-879a-2877eca7d1fe\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj"
Oct 30 13:22:27.561790 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.536982    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjg44\" (UniqueName: \"kubernetes.io/projected/21ffd071-46da-4940-879a-2877eca7d1fe-kube-api-access-hjg44\") pod \"auth-test-model-bcd76c-kserve-665b758b77-bppbj\" (UID: \"21ffd071-46da-4940-879a-2877eca7d1fe\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj"
Oct 30 13:22:27.591361 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.590907    4544 patch_prober.go:29] interesting pod/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 13:22:27.591361 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.590966    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl" podUID="bd4c038f-d0d4-4fd5-ad51-2dbae210cb26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 13:22:27.609518 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.609139    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj"
Oct 30 13:22:27.880429 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:27.880355    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:28.053158 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:28.053130    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj"]
Oct 30 13:22:28.067761 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:22:28.063196    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21ffd071_46da_4940_879a_2877eca7d1fe.slice/crio-3c88c84be01dc41f0f1d9db40f59e14d1f0200237b9a153086df54c07f67e94e WatchSource:0}: Error finding container 3c88c84be01dc41f0f1d9db40f59e14d1f0200237b9a153086df54c07f67e94e: Status 404 returned error can't find the container with id 3c88c84be01dc41f0f1d9db40f59e14d1f0200237b9a153086df54c07f67e94e
Oct 30 13:22:28.589551 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:28.589515    4544 patch_prober.go:29] interesting pod/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 13:22:28.589961 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:28.589564    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl" podUID="bd4c038f-d0d4-4fd5-ad51-2dbae210cb26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 13:22:28.604442 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:28.604402    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj" event={"ID":"21ffd071-46da-4940-879a-2877eca7d1fe","Type":"ContainerStarted","Data":"358222b95bc3c2e17638279ed3172de130bab87c95c03cd8b32a785e788c0cfe"}
Oct 30 13:22:28.604442 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:28.604449    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj" event={"ID":"21ffd071-46da-4940-879a-2877eca7d1fe","Type":"ContainerStarted","Data":"3c88c84be01dc41f0f1d9db40f59e14d1f0200237b9a153086df54c07f67e94e"}
Oct 30 13:22:28.610738 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:28.610714    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj"
Oct 30 13:22:28.663180 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:28.663124    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj" podStartSLOduration=1.663108615 podStartE2EDuration="1.663108615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:28.660939396 +0000 UTC m=+2243.708073349" watchObservedRunningTime="2026-10-30 13:22:28.663108615 +0000 UTC m=+2243.710242568"
Oct 30 13:22:29.009898 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:22:29.009807    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:22:29.589664 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:29.589625    4544 patch_prober.go:29] interesting pod/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 13:22:29.590094 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:29.589680    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl" podUID="bd4c038f-d0d4-4fd5-ad51-2dbae210cb26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 13:22:29.934320 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:29.934205    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:30.063100 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:30.063064    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:30.589955 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:30.589920    4544 patch_prober.go:29] interesting pod/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 13:22:30.590390 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:30.589983    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl" podUID="bd4c038f-d0d4-4fd5-ad51-2dbae210cb26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 13:22:31.589907 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:31.589870    4544 patch_prober.go:29] interesting pod/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 13:22:31.590098 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:31.589944    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl" podUID="bd4c038f-d0d4-4fd5-ad51-2dbae210cb26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 13:22:32.173384 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:32.173338    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:32.594505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:32.594474    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:22:32.594994 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:32.594975    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:22:32.595630 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:32.595612    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:22:33.304852 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:33.302985    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:33.527986 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:33.527946    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6af611-qhx72"]
Oct 30 13:22:33.529129 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:33.529099    4544 cpu_manager.go:414] "containerMap: removing container" podUID="1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff" containerName="istio-proxy"
Oct 30 13:22:33.529129 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:33.529125    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1253c4f1-9b9a-4a12-b1d2-fc61d9e219ff" containerName="istio-proxy"
Oct 30 13:22:33.533933 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:33.533909    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6af611-qhx72"
Oct 30 13:22:33.557467 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:33.557399    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6af611-qhx72"]
Oct 30 13:22:33.566350 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:33.564507    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:22:33.566350 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:33.564587    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:22:33.678710 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:33.678679    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3a14fde-d5d6-4769-9462-0363207909f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6af611-qhx72\" (UID: \"d3a14fde-d5d6-4769-9462-0363207909f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6af611-qhx72"
Oct 30 13:22:33.679098 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:33.678877    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwnqh\" (UniqueName: \"kubernetes.io/projected/d3a14fde-d5d6-4769-9462-0363207909f4-kube-api-access-zwnqh\") pod \"maas-api-revoke-keys-e2e-inf-a-6af611-qhx72\" (UID: \"d3a14fde-d5d6-4769-9462-0363207909f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6af611-qhx72"
Oct 30 13:22:33.679098 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:33.678992    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3a14fde-d5d6-4769-9462-0363207909f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6af611-qhx72\" (UID: \"d3a14fde-d5d6-4769-9462-0363207909f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6af611-qhx72"
Oct 30 13:22:33.782534 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:33.780717    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwnqh\" (UniqueName: \"kubernetes.io/projected/d3a14fde-d5d6-4769-9462-0363207909f4-kube-api-access-zwnqh\") pod \"maas-api-revoke-keys-e2e-inf-a-6af611-qhx72\" (UID: \"d3a14fde-d5d6-4769-9462-0363207909f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6af611-qhx72"
Oct 30 13:22:33.782534 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:33.780833    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3a14fde-d5d6-4769-9462-0363207909f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6af611-qhx72\" (UID: \"d3a14fde-d5d6-4769-9462-0363207909f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6af611-qhx72"
Oct 30 13:22:33.782534 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:33.780939    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3a14fde-d5d6-4769-9462-0363207909f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6af611-qhx72\" (UID: \"d3a14fde-d5d6-4769-9462-0363207909f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6af611-qhx72"
Oct 30 13:22:33.782534 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:33.782365    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3a14fde-d5d6-4769-9462-0363207909f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6af611-qhx72\" (UID: \"d3a14fde-d5d6-4769-9462-0363207909f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6af611-qhx72"
Oct 30 13:22:33.784880 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:33.784846    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3a14fde-d5d6-4769-9462-0363207909f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6af611-qhx72\" (UID: \"d3a14fde-d5d6-4769-9462-0363207909f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6af611-qhx72"
Oct 30 13:22:33.806718 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:33.806680    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwnqh\" (UniqueName: \"kubernetes.io/projected/d3a14fde-d5d6-4769-9462-0363207909f4-kube-api-access-zwnqh\") pod \"maas-api-revoke-keys-e2e-inf-a-6af611-qhx72\" (UID: \"d3a14fde-d5d6-4769-9462-0363207909f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6af611-qhx72"
Oct 30 13:22:33.853872 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:33.853833    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6af611-qhx72"
Oct 30 13:22:34.169900 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:34.169870    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6af611-qhx72"]
Oct 30 13:22:34.171691 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:22:34.171662    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a14fde_d5d6_4769_9462_0363207909f4.slice/crio-2626e0cc275e104730797d34210e03199f2d16215cae2d43e34a7e6ed53d8d7f WatchSource:0}: Error finding container 2626e0cc275e104730797d34210e03199f2d16215cae2d43e34a7e6ed53d8d7f: Status 404 returned error can't find the container with id 2626e0cc275e104730797d34210e03199f2d16215cae2d43e34a7e6ed53d8d7f
Oct 30 13:22:34.190092 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:34.190062    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:34.676013 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:34.675965    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6af611-qhx72" event={"ID":"d3a14fde-d5d6-4769-9462-0363207909f4","Type":"ContainerStarted","Data":"e45b706db4fb5d8c7b23fba6a6030c0d31ff3a15e2c7f55d23323d901c7c2051"}
Oct 30 13:22:34.676264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:34.676050    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6af611-qhx72" event={"ID":"d3a14fde-d5d6-4769-9462-0363207909f4","Type":"ContainerStarted","Data":"2626e0cc275e104730797d34210e03199f2d16215cae2d43e34a7e6ed53d8d7f"}
Oct 30 13:22:34.721160 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:34.721097    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6af611-qhx72" podStartSLOduration=1.721080398 podStartE2EDuration="1.721080398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:34.716142013 +0000 UTC m=+2249.763275965" watchObservedRunningTime="2026-10-30 13:22:34.721080398 +0000 UTC m=+2249.768214354"
Oct 30 13:22:35.685697 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:35.685663    4544 generic.go:356] "Generic (PLEG): container finished" podID="d3a14fde-d5d6-4769-9462-0363207909f4" containerID="e45b706db4fb5d8c7b23fba6a6030c0d31ff3a15e2c7f55d23323d901c7c2051" exitCode=0
Oct 30 13:22:35.685875 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:35.685728    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6af611-qhx72" event={"ID":"d3a14fde-d5d6-4769-9462-0363207909f4","Type":"ContainerDied","Data":"e45b706db4fb5d8c7b23fba6a6030c0d31ff3a15e2c7f55d23323d901c7c2051"}
Oct 30 13:22:36.165909 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:36.165874    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:36.882166 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:36.882138    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6af611-qhx72"
Oct 30 13:22:36.933931 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:36.933898    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3a14fde-d5d6-4769-9462-0363207909f4-kube-api-access-zwnqh\" (UniqueName: \"kubernetes.io/projected/d3a14fde-d5d6-4769-9462-0363207909f4-kube-api-access-zwnqh\") pod \"d3a14fde-d5d6-4769-9462-0363207909f4\" (UID: \"d3a14fde-d5d6-4769-9462-0363207909f4\") "
Oct 30 13:22:36.934123 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:36.933968    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3a14fde-d5d6-4769-9462-0363207909f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3a14fde-d5d6-4769-9462-0363207909f4-maas-api-xks-ca\") pod \"d3a14fde-d5d6-4769-9462-0363207909f4\" (UID: \"d3a14fde-d5d6-4769-9462-0363207909f4\") "
Oct 30 13:22:36.934186 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:36.934166    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3a14fde-d5d6-4769-9462-0363207909f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3a14fde-d5d6-4769-9462-0363207909f4-maas-api-openshift-ca\") pod \"d3a14fde-d5d6-4769-9462-0363207909f4\" (UID: \"d3a14fde-d5d6-4769-9462-0363207909f4\") "
Oct 30 13:22:36.934697 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:36.934665    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3a14fde-d5d6-4769-9462-0363207909f4-maas-api-openshift-ca" pod "d3a14fde-d5d6-4769-9462-0363207909f4" (UID: "d3a14fde-d5d6-4769-9462-0363207909f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:22:36.936128 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:36.936099    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3a14fde-d5d6-4769-9462-0363207909f4-maas-api-xks-ca" pod "d3a14fde-d5d6-4769-9462-0363207909f4" (UID: "d3a14fde-d5d6-4769-9462-0363207909f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:22:36.936266 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:36.936210    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3a14fde-d5d6-4769-9462-0363207909f4-kube-api-access-zwnqh" pod "d3a14fde-d5d6-4769-9462-0363207909f4" (UID: "d3a14fde-d5d6-4769-9462-0363207909f4"). InnerVolumeSpecName "kube-api-access-zwnqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:22:37.035794 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:37.035759    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwnqh\" (UniqueName: \"kubernetes.io/projected/d3a14fde-d5d6-4769-9462-0363207909f4-kube-api-access-zwnqh\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:22:37.035794 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:37.035788    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3a14fde-d5d6-4769-9462-0363207909f4-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:22:37.035794 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:37.035798    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3a14fde-d5d6-4769-9462-0363207909f4-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:22:37.610524 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:37.610485    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj"
Oct 30 13:22:37.624096 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:37.624062    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj"
Oct 30 13:22:37.872981 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:37.872034    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6af611-qhx72"
Oct 30 13:22:37.872981 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:37.872279    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6af611-qhx72" event={"ID":"d3a14fde-d5d6-4769-9462-0363207909f4","Type":"ContainerDied","Data":"2626e0cc275e104730797d34210e03199f2d16215cae2d43e34a7e6ed53d8d7f"}
Oct 30 13:22:37.872981 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:37.872381    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2626e0cc275e104730797d34210e03199f2d16215cae2d43e34a7e6ed53d8d7f"
Oct 30 13:22:37.889492 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:37.889451    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj"
Oct 30 13:22:38.021666 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:38.021632    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:39.577170 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:39.577134    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29501dc6-fbmrr"]
Oct 30 13:22:39.577957 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:39.577941    4544 cpu_manager.go:414] "containerMap: removing container" podUID="d3a14fde-d5d6-4769-9462-0363207909f4" containerName="revoke-keys"
Oct 30 13:22:39.578006 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:39.577958    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3a14fde-d5d6-4769-9462-0363207909f4" containerName="revoke-keys"
Oct 30 13:22:39.578079 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:39.578070    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3a14fde-d5d6-4769-9462-0363207909f4" containerName="revoke-keys"
Oct 30 13:22:39.581895 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:39.581878    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29501dc6-fbmrr"
Oct 30 13:22:39.598019 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:39.597992    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:22:39.610203 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:39.610158    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:22:39.616700 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:39.616667    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29501dc6-fbmrr"]
Oct 30 13:22:39.669835 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:39.669795    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f132dc3a-c383-44dc-a5c0-39d18dbea7bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29501dc6-fbmrr\" (UID: \"f132dc3a-c383-44dc-a5c0-39d18dbea7bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29501dc6-fbmrr"
Oct 30 13:22:39.669999 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:39.669960    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f132dc3a-c383-44dc-a5c0-39d18dbea7bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29501dc6-fbmrr\" (UID: \"f132dc3a-c383-44dc-a5c0-39d18dbea7bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29501dc6-fbmrr"
Oct 30 13:22:39.670122 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:39.670105    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9l8t\" (UniqueName: \"kubernetes.io/projected/f132dc3a-c383-44dc-a5c0-39d18dbea7bf-kube-api-access-v9l8t\") pod \"maas-api-revoke-sub-29501dc6-fbmrr\" (UID: \"f132dc3a-c383-44dc-a5c0-39d18dbea7bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29501dc6-fbmrr"
Oct 30 13:22:39.772026 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:39.771989    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9l8t\" (UniqueName: \"kubernetes.io/projected/f132dc3a-c383-44dc-a5c0-39d18dbea7bf-kube-api-access-v9l8t\") pod \"maas-api-revoke-sub-29501dc6-fbmrr\" (UID: \"f132dc3a-c383-44dc-a5c0-39d18dbea7bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29501dc6-fbmrr"
Oct 30 13:22:39.772208 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:39.772100    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f132dc3a-c383-44dc-a5c0-39d18dbea7bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29501dc6-fbmrr\" (UID: \"f132dc3a-c383-44dc-a5c0-39d18dbea7bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29501dc6-fbmrr"
Oct 30 13:22:39.772337 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:39.772208    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f132dc3a-c383-44dc-a5c0-39d18dbea7bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29501dc6-fbmrr\" (UID: \"f132dc3a-c383-44dc-a5c0-39d18dbea7bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29501dc6-fbmrr"
Oct 30 13:22:39.772839 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:39.772813    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f132dc3a-c383-44dc-a5c0-39d18dbea7bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29501dc6-fbmrr\" (UID: \"f132dc3a-c383-44dc-a5c0-39d18dbea7bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29501dc6-fbmrr"
Oct 30 13:22:39.774611 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:39.774588    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f132dc3a-c383-44dc-a5c0-39d18dbea7bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29501dc6-fbmrr\" (UID: \"f132dc3a-c383-44dc-a5c0-39d18dbea7bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29501dc6-fbmrr"
Oct 30 13:22:39.795705 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:39.795670    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9l8t\" (UniqueName: \"kubernetes.io/projected/f132dc3a-c383-44dc-a5c0-39d18dbea7bf-kube-api-access-v9l8t\") pod \"maas-api-revoke-sub-29501dc6-fbmrr\" (UID: \"f132dc3a-c383-44dc-a5c0-39d18dbea7bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29501dc6-fbmrr"
Oct 30 13:22:39.896916 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:39.896824    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29501dc6-fbmrr"
Oct 30 13:22:40.172259 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:40.172162    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:40.250698 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:40.250602    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29501dc6-fbmrr"]
Oct 30 13:22:40.263037 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:22:40.262984    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf132dc3a_c383_44dc_a5c0_39d18dbea7bf.slice/crio-e28377fc270ebfde84cdffa1ac5c4b57270effcef3ecefa48fa942565c3b09a2 WatchSource:0}: Error finding container e28377fc270ebfde84cdffa1ac5c4b57270effcef3ecefa48fa942565c3b09a2: Status 404 returned error can't find the container with id e28377fc270ebfde84cdffa1ac5c4b57270effcef3ecefa48fa942565c3b09a2
Oct 30 13:22:40.896579 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:40.896544    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29501dc6-fbmrr" event={"ID":"f132dc3a-c383-44dc-a5c0-39d18dbea7bf","Type":"ContainerStarted","Data":"6a788f8c95b3baba1affc198e9443c9273445a04e4ab3ff795099857fb60d511"}
Oct 30 13:22:40.896983 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:40.896598    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29501dc6-fbmrr" event={"ID":"f132dc3a-c383-44dc-a5c0-39d18dbea7bf","Type":"ContainerStarted","Data":"e28377fc270ebfde84cdffa1ac5c4b57270effcef3ecefa48fa942565c3b09a2"}
Oct 30 13:22:40.937682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:40.937616    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29501dc6-fbmrr" podStartSLOduration=1.937593948 podStartE2EDuration="1.937593948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:40.936143403 +0000 UTC m=+2255.983277356" watchObservedRunningTime="2026-10-30 13:22:40.937593948 +0000 UTC m=+2255.984727901"
Oct 30 13:22:41.906328 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:41.906294    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:41.907067 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:41.907040    4544 generic.go:356] "Generic (PLEG): container finished" podID="f132dc3a-c383-44dc-a5c0-39d18dbea7bf" containerID="6a788f8c95b3baba1affc198e9443c9273445a04e4ab3ff795099857fb60d511" exitCode=0
Oct 30 13:22:41.907184 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:41.907126    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29501dc6-fbmrr" event={"ID":"f132dc3a-c383-44dc-a5c0-39d18dbea7bf","Type":"ContainerDied","Data":"6a788f8c95b3baba1affc198e9443c9273445a04e4ab3ff795099857fb60d511"}
Oct 30 13:22:42.766209 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:42.766170    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:43.016519 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.016418    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq"]
Oct 30 13:22:43.033561 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.033525    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq"
Oct 30 13:22:43.072036 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.071978    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-60a9a6\"/\"openshift-service-ca.crt\""
Oct 30 13:22:43.073656 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.073330    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-60a9a6\"/\"kube-root-ca.crt\""
Oct 30 13:22:43.073656 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.073591    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-60a9a6\"/\"test-model-60a9a6-kserve-self-signed-certs\""
Oct 30 13:22:43.073938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.073657    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-60a9a6\"/\"default-dockercfg-bgjtr\""
Oct 30 13:22:43.107130 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.107095    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq"]
Oct 30 13:22:43.170347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.166085    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-tls-certs\") pod \"test-model-60a9a6-kserve-7db4fcd49-6m4bq\" (UID: \"0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf\") " pod="ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq"
Oct 30 13:22:43.170347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.166157    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-dshm\") pod \"test-model-60a9a6-kserve-7db4fcd49-6m4bq\" (UID: \"0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf\") " pod="ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq"
Oct 30 13:22:43.170347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.166428    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz676\" (UniqueName: \"kubernetes.io/projected/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-kube-api-access-kz676\") pod \"test-model-60a9a6-kserve-7db4fcd49-6m4bq\" (UID: \"0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf\") " pod="ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq"
Oct 30 13:22:43.170347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.166487    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-home\") pod \"test-model-60a9a6-kserve-7db4fcd49-6m4bq\" (UID: \"0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf\") " pod="ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq"
Oct 30 13:22:43.170347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.166560    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-tmp-dir\") pod \"test-model-60a9a6-kserve-7db4fcd49-6m4bq\" (UID: \"0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf\") " pod="ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq"
Oct 30 13:22:43.170347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.166671    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-model-cache\") pod \"test-model-60a9a6-kserve-7db4fcd49-6m4bq\" (UID: \"0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf\") " pod="ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq"
Oct 30 13:22:43.200537 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.200507    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29501dc6-fbmrr"
Oct 30 13:22:43.262188 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.262155    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:43.267875 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.267817    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz676\" (UniqueName: \"kubernetes.io/projected/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-kube-api-access-kz676\") pod \"test-model-60a9a6-kserve-7db4fcd49-6m4bq\" (UID: \"0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf\") " pod="ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq"
Oct 30 13:22:43.267969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.267879    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-home\") pod \"test-model-60a9a6-kserve-7db4fcd49-6m4bq\" (UID: \"0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf\") " pod="ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq"
Oct 30 13:22:43.267969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.267959    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-tmp-dir\") pod \"test-model-60a9a6-kserve-7db4fcd49-6m4bq\" (UID: \"0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf\") " pod="ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq"
Oct 30 13:22:43.268110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.268084    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-model-cache\") pod \"test-model-60a9a6-kserve-7db4fcd49-6m4bq\" (UID: \"0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf\") " pod="ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq"
Oct 30 13:22:43.268372 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.268349    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-home\") pod \"test-model-60a9a6-kserve-7db4fcd49-6m4bq\" (UID: \"0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf\") " pod="ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq"
Oct 30 13:22:43.268542 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.268417    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-tmp-dir\") pod \"test-model-60a9a6-kserve-7db4fcd49-6m4bq\" (UID: \"0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf\") " pod="ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq"
Oct 30 13:22:43.268542 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.268436    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-tls-certs\") pod \"test-model-60a9a6-kserve-7db4fcd49-6m4bq\" (UID: \"0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf\") " pod="ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq"
Oct 30 13:22:43.268542 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.268498    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-dshm\") pod \"test-model-60a9a6-kserve-7db4fcd49-6m4bq\" (UID: \"0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf\") " pod="ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq"
Oct 30 13:22:43.268690 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.268499    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-model-cache\") pod \"test-model-60a9a6-kserve-7db4fcd49-6m4bq\" (UID: \"0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf\") " pod="ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq"
Oct 30 13:22:43.270753 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.270730    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-dshm\") pod \"test-model-60a9a6-kserve-7db4fcd49-6m4bq\" (UID: \"0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf\") " pod="ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq"
Oct 30 13:22:43.271353 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.271094    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-tls-certs\") pod \"test-model-60a9a6-kserve-7db4fcd49-6m4bq\" (UID: \"0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf\") " pod="ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq"
Oct 30 13:22:43.294995 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.294948    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz676\" (UniqueName: \"kubernetes.io/projected/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-kube-api-access-kz676\") pod \"test-model-60a9a6-kserve-7db4fcd49-6m4bq\" (UID: \"0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf\") " pod="ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq"
Oct 30 13:22:43.357552 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.357505    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq"
Oct 30 13:22:43.396021 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.380676    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f132dc3a-c383-44dc-a5c0-39d18dbea7bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f132dc3a-c383-44dc-a5c0-39d18dbea7bf-maas-api-xks-ca\") pod \"f132dc3a-c383-44dc-a5c0-39d18dbea7bf\" (UID: \"f132dc3a-c383-44dc-a5c0-39d18dbea7bf\") "
Oct 30 13:22:43.396021 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.380888    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f132dc3a-c383-44dc-a5c0-39d18dbea7bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f132dc3a-c383-44dc-a5c0-39d18dbea7bf-maas-api-openshift-ca\") pod \"f132dc3a-c383-44dc-a5c0-39d18dbea7bf\" (UID: \"f132dc3a-c383-44dc-a5c0-39d18dbea7bf\") "
Oct 30 13:22:43.396021 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.380989    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f132dc3a-c383-44dc-a5c0-39d18dbea7bf-kube-api-access-v9l8t\" (UniqueName: \"kubernetes.io/projected/f132dc3a-c383-44dc-a5c0-39d18dbea7bf-kube-api-access-v9l8t\") pod \"f132dc3a-c383-44dc-a5c0-39d18dbea7bf\" (UID: \"f132dc3a-c383-44dc-a5c0-39d18dbea7bf\") "
Oct 30 13:22:43.396021 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.389093    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f132dc3a-c383-44dc-a5c0-39d18dbea7bf-maas-api-openshift-ca" pod "f132dc3a-c383-44dc-a5c0-39d18dbea7bf" (UID: "f132dc3a-c383-44dc-a5c0-39d18dbea7bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:22:43.401524 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.400824    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f132dc3a-c383-44dc-a5c0-39d18dbea7bf-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:22:43.421299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.416885    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f132dc3a-c383-44dc-a5c0-39d18dbea7bf-kube-api-access-v9l8t" pod "f132dc3a-c383-44dc-a5c0-39d18dbea7bf" (UID: "f132dc3a-c383-44dc-a5c0-39d18dbea7bf"). InnerVolumeSpecName "kube-api-access-v9l8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:22:43.425419 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.425363    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f132dc3a-c383-44dc-a5c0-39d18dbea7bf-maas-api-xks-ca" pod "f132dc3a-c383-44dc-a5c0-39d18dbea7bf" (UID: "f132dc3a-c383-44dc-a5c0-39d18dbea7bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:22:43.502531 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.502094    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9l8t\" (UniqueName: \"kubernetes.io/projected/f132dc3a-c383-44dc-a5c0-39d18dbea7bf-kube-api-access-v9l8t\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:22:43.502531 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.502130    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f132dc3a-c383-44dc-a5c0-39d18dbea7bf-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:22:43.811061 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.810978    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq"]
Oct 30 13:22:43.816526 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:22:43.816495    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0bbf1f7a_a241_4a1d_9cc7_f6fa136979bf.slice/crio-afde71b914de7f6bf0aa0d8f75737d2f778280d4cfb565fc38dcdba80ac8d414 WatchSource:0}: Error finding container afde71b914de7f6bf0aa0d8f75737d2f778280d4cfb565fc38dcdba80ac8d414: Status 404 returned error can't find the container with id afde71b914de7f6bf0aa0d8f75737d2f778280d4cfb565fc38dcdba80ac8d414
Oct 30 13:22:43.933595 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.933546    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:43.934860 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.934835    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq" event={"ID":"0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf","Type":"ContainerStarted","Data":"09585a9abbaec9c381ba9c05a5fb6f5fcdf57084ee65a4af4bcad94cc6759f60"}
Oct 30 13:22:43.935041 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.935025    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq" event={"ID":"0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf","Type":"ContainerStarted","Data":"afde71b914de7f6bf0aa0d8f75737d2f778280d4cfb565fc38dcdba80ac8d414"}
Oct 30 13:22:43.937275 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.937251    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29501dc6-fbmrr" event={"ID":"f132dc3a-c383-44dc-a5c0-39d18dbea7bf","Type":"ContainerDied","Data":"e28377fc270ebfde84cdffa1ac5c4b57270effcef3ecefa48fa942565c3b09a2"}
Oct 30 13:22:43.937413 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.937285    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29501dc6-fbmrr"
Oct 30 13:22:43.937413 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.937301    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e28377fc270ebfde84cdffa1ac5c4b57270effcef3ecefa48fa942565c3b09a2"
Oct 30 13:22:43.989659 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:43.985296    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq" podStartSLOduration=1.985276239 podStartE2EDuration="1.985276239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:43.977275136 +0000 UTC m=+2259.024409090" watchObservedRunningTime="2026-10-30 13:22:43.985276239 +0000 UTC m=+2259.032410193"
Oct 30 13:22:44.075915 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:44.075877    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:44.196024 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:44.195987    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:44.359118 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:44.359010    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq"
Oct 30 13:22:45.978998 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:45.978951    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:46.881558 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:46.881497    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:47.792781 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:47.792699    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:49.555893 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:49.555841    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:51.503893 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:51.503856    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:53.252381 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:53.252344    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:53.358475 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:53.358436    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq"
Oct 30 13:22:53.371781 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:53.371748    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq"
Oct 30 13:22:54.032209 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:54.032169    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq"
Oct 30 13:22:55.223285 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:55.220265    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:56.189891 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:56.189813    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:57.245962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:57.245925    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:22:59.009456 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:22:59.009417    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:22:59.082395 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:22:59.082357    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:01.001063 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:01.001018    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:02.598203 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:02.598168    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:03.236732 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:03.236651    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:03.461271 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:03.461238    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:03.504291 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:03.504128    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6af611-qhx72"]
Oct 30 13:23:03.542280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:03.542243    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6af611-qhx72"]
Oct 30 13:23:03.605859 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:03.605118    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:04.014972 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:04.014937    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:05.088779 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:05.088744    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3a14fde-d5d6-4769-9462-0363207909f4" path="/var/lib/kubelet/pods/d3a14fde-d5d6-4769-9462-0363207909f4/volumes"
Oct 30 13:23:05.699910 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:05.699867    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:06.732865 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:06.732830    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg"]
Oct 30 13:23:06.733927 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:06.733903    4544 cpu_manager.go:414] "containerMap: removing container" podUID="f132dc3a-c383-44dc-a5c0-39d18dbea7bf" containerName="revoke-keys"
Oct 30 13:23:06.733927 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:06.733928    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f132dc3a-c383-44dc-a5c0-39d18dbea7bf" containerName="revoke-keys"
Oct 30 13:23:06.734189 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:06.734175    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="f132dc3a-c383-44dc-a5c0-39d18dbea7bf" containerName="revoke-keys"
Oct 30 13:23:06.738446 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:06.738422    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg"
Oct 30 13:23:06.758570 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:06.758542    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f3338e\"/\"openshift-service-ca.crt\""
Oct 30 13:23:06.758745 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:06.758659    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f3338e\"/\"kube-root-ca.crt\""
Oct 30 13:23:06.770748 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:06.770715    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f3338e\"/\"test-model-f3338e-kserve-self-signed-certs\""
Oct 30 13:23:06.774931 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:06.774896    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f3338e\"/\"default-dockercfg-v4sqj\""
Oct 30 13:23:06.775383 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:06.775352    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg"]
Oct 30 13:23:06.817648 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:06.817613    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6998926-73ac-4828-91b7-d453b5ebc33e-dshm\") pod \"test-model-f3338e-kserve-86cff7f499-7lldg\" (UID: \"c6998926-73ac-4828-91b7-d453b5ebc33e\") " pod="ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg"
Oct 30 13:23:06.817889 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:06.817677    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6998926-73ac-4828-91b7-d453b5ebc33e-tmp-dir\") pod \"test-model-f3338e-kserve-86cff7f499-7lldg\" (UID: \"c6998926-73ac-4828-91b7-d453b5ebc33e\") " pod="ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg"
Oct 30 13:23:06.817889 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:06.817852    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6998926-73ac-4828-91b7-d453b5ebc33e-home\") pod \"test-model-f3338e-kserve-86cff7f499-7lldg\" (UID: \"c6998926-73ac-4828-91b7-d453b5ebc33e\") " pod="ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg"
Oct 30 13:23:06.818046 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:06.818022    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6998926-73ac-4828-91b7-d453b5ebc33e-model-cache\") pod \"test-model-f3338e-kserve-86cff7f499-7lldg\" (UID: \"c6998926-73ac-4828-91b7-d453b5ebc33e\") " pod="ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg"
Oct 30 13:23:06.818200 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:06.818181    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6998926-73ac-4828-91b7-d453b5ebc33e-tls-certs\") pod \"test-model-f3338e-kserve-86cff7f499-7lldg\" (UID: \"c6998926-73ac-4828-91b7-d453b5ebc33e\") " pod="ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg"
Oct 30 13:23:06.818301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:06.818215    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghtpj\" (UniqueName: \"kubernetes.io/projected/c6998926-73ac-4828-91b7-d453b5ebc33e-kube-api-access-ghtpj\") pod \"test-model-f3338e-kserve-86cff7f499-7lldg\" (UID: \"c6998926-73ac-4828-91b7-d453b5ebc33e\") " pod="ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg"
Oct 30 13:23:06.867420 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:06.867389    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:06.919684 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:06.919643    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6998926-73ac-4828-91b7-d453b5ebc33e-tls-certs\") pod \"test-model-f3338e-kserve-86cff7f499-7lldg\" (UID: \"c6998926-73ac-4828-91b7-d453b5ebc33e\") " pod="ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg"
Oct 30 13:23:06.919930 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:06.919699    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghtpj\" (UniqueName: \"kubernetes.io/projected/c6998926-73ac-4828-91b7-d453b5ebc33e-kube-api-access-ghtpj\") pod \"test-model-f3338e-kserve-86cff7f499-7lldg\" (UID: \"c6998926-73ac-4828-91b7-d453b5ebc33e\") " pod="ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg"
Oct 30 13:23:06.919930 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:06.919821    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6998926-73ac-4828-91b7-d453b5ebc33e-dshm\") pod \"test-model-f3338e-kserve-86cff7f499-7lldg\" (UID: \"c6998926-73ac-4828-91b7-d453b5ebc33e\") " pod="ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg"
Oct 30 13:23:06.919930 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:06.919877    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6998926-73ac-4828-91b7-d453b5ebc33e-tmp-dir\") pod \"test-model-f3338e-kserve-86cff7f499-7lldg\" (UID: \"c6998926-73ac-4828-91b7-d453b5ebc33e\") " pod="ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg"
Oct 30 13:23:06.920097 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:06.919957    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6998926-73ac-4828-91b7-d453b5ebc33e-home\") pod \"test-model-f3338e-kserve-86cff7f499-7lldg\" (UID: \"c6998926-73ac-4828-91b7-d453b5ebc33e\") " pod="ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg"
Oct 30 13:23:06.920156 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:06.920100    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6998926-73ac-4828-91b7-d453b5ebc33e-model-cache\") pod \"test-model-f3338e-kserve-86cff7f499-7lldg\" (UID: \"c6998926-73ac-4828-91b7-d453b5ebc33e\") " pod="ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg"
Oct 30 13:23:06.920410 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:06.920389    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6998926-73ac-4828-91b7-d453b5ebc33e-tmp-dir\") pod \"test-model-f3338e-kserve-86cff7f499-7lldg\" (UID: \"c6998926-73ac-4828-91b7-d453b5ebc33e\") " pod="ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg"
Oct 30 13:23:06.920519 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:06.920444    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6998926-73ac-4828-91b7-d453b5ebc33e-home\") pod \"test-model-f3338e-kserve-86cff7f499-7lldg\" (UID: \"c6998926-73ac-4828-91b7-d453b5ebc33e\") " pod="ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg"
Oct 30 13:23:06.920519 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:06.920481    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6998926-73ac-4828-91b7-d453b5ebc33e-model-cache\") pod \"test-model-f3338e-kserve-86cff7f499-7lldg\" (UID: \"c6998926-73ac-4828-91b7-d453b5ebc33e\") " pod="ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg"
Oct 30 13:23:06.922338 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:06.922311    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6998926-73ac-4828-91b7-d453b5ebc33e-dshm\") pod \"test-model-f3338e-kserve-86cff7f499-7lldg\" (UID: \"c6998926-73ac-4828-91b7-d453b5ebc33e\") " pod="ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg"
Oct 30 13:23:06.922877 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:06.922855    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6998926-73ac-4828-91b7-d453b5ebc33e-tls-certs\") pod \"test-model-f3338e-kserve-86cff7f499-7lldg\" (UID: \"c6998926-73ac-4828-91b7-d453b5ebc33e\") " pod="ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg"
Oct 30 13:23:06.951885 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:06.951854    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghtpj\" (UniqueName: \"kubernetes.io/projected/c6998926-73ac-4828-91b7-d453b5ebc33e-kube-api-access-ghtpj\") pod \"test-model-f3338e-kserve-86cff7f499-7lldg\" (UID: \"c6998926-73ac-4828-91b7-d453b5ebc33e\") " pod="ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg"
Oct 30 13:23:07.083030 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:07.076529    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg"
Oct 30 13:23:07.472245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:07.472201    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg"]
Oct 30 13:23:07.473759 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:23:07.473730    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6998926_73ac_4828_91b7_d453b5ebc33e.slice/crio-05ca08ca5f50632937b3fa96cc57250bffe6d3c220f544e3424e209cf64881c7 WatchSource:0}: Error finding container 05ca08ca5f50632937b3fa96cc57250bffe6d3c220f544e3424e209cf64881c7: Status 404 returned error can't find the container with id 05ca08ca5f50632937b3fa96cc57250bffe6d3c220f544e3424e209cf64881c7
Oct 30 13:23:07.602799 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:07.602763    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:08.247008 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:08.246967    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg" event={"ID":"c6998926-73ac-4828-91b7-d453b5ebc33e","Type":"ContainerStarted","Data":"6f2a250ed923317b1f0789e2c6550d0dfdfff9c56695f2628bff4e0a8dff05fe"}
Oct 30 13:23:08.247430 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:08.247020    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg" event={"ID":"c6998926-73ac-4828-91b7-d453b5ebc33e","Type":"ContainerStarted","Data":"05ca08ca5f50632937b3fa96cc57250bffe6d3c220f544e3424e209cf64881c7"}
Oct 30 13:23:08.290981 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:08.290922    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg" podStartSLOduration=2.290906829 podStartE2EDuration="2.290906829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:08.289166447 +0000 UTC m=+2283.336300411" watchObservedRunningTime="2026-10-30 13:23:08.290906829 +0000 UTC m=+2283.338040782"
Oct 30 13:23:09.088060 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:09.088026    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg"
Oct 30 13:23:09.373315 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:09.373213    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"]
Oct 30 13:23:09.378331 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:09.378306    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:23:09.403304 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:09.403269    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-618c72-openshift-default-dockercfg-f98p9\""
Oct 30 13:23:09.420232 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:09.420179    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"]
Oct 30 13:23:09.430608 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:09.430547    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:09.485114 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:09.485062    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-workload-socket\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:23:09.485358 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:09.485138    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-istio-data\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:23:09.485358 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:09.485252    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-istio-envoy\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:23:09.485515 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:09.485491    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c84b2e67-2800-4616-a136-a0912e553b55-istiod-ca-cert\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:23:09.485583 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:09.485559    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4bbc\" (UniqueName: \"kubernetes.io/projected/c84b2e67-2800-4616-a136-a0912e553b55-kube-api-access-n4bbc\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:23:09.485664 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:09.485648    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-credential-socket\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:23:09.485727 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:09.485696    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-workload-certs\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:23:09.485905 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:09.485887    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c84b2e67-2800-4616-a136-a0912e553b55-istio-podinfo\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:23:09.486040 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:09.485938    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c84b2e67-2800-4616-a136-a0912e553b55-istio-token\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:23:09.587780 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:09.587742    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c84b2e67-2800-4616-a136-a0912e553b55-istio-podinfo\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:23:09.587972 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:09.587793    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c84b2e67-2800-4616-a136-a0912e553b55-istio-token\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:23:09.587972 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:09.587874    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-workload-socket\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:23:09.587972 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:09.587917    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-istio-data\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:23:09.588715 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:09.588676    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-istio-envoy\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:23:09.588848 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:09.588765    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c84b2e67-2800-4616-a136-a0912e553b55-istiod-ca-cert\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:23:09.588848 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:09.588823    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4bbc\" (UniqueName: \"kubernetes.io/projected/c84b2e67-2800-4616-a136-a0912e553b55-kube-api-access-n4bbc\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:23:09.588974 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:09.588874    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-workload-socket\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:23:09.588974 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:09.588907    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-credential-socket\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:23:09.588974 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:09.588966    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-workload-certs\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:23:09.589152 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:09.588993    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-istio-data\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:23:09.589321 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:09.589301    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-workload-certs\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:23:09.589399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:09.589323    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-credential-socket\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:23:09.589513 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:09.589496    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c84b2e67-2800-4616-a136-a0912e553b55-istiod-ca-cert\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:23:09.590598 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:09.590577    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c84b2e67-2800-4616-a136-a0912e553b55-istio-podinfo\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:23:09.590890 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:09.590873    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-istio-envoy\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:23:09.611681 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:09.611616    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c84b2e67-2800-4616-a136-a0912e553b55-istio-token\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:23:09.616527 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:09.616497    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4bbc\" (UniqueName: \"kubernetes.io/projected/c84b2e67-2800-4616-a136-a0912e553b55-kube-api-access-n4bbc\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:23:09.695011 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:09.694920    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:23:10.028015 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:23:10.027986    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc84b2e67_2800_4616_a136_a0912e553b55.slice/crio-23aa5d295dc4da2b00fc07e2bd07363ea1da8bf0e1304994067c06534b34007c WatchSource:0}: Error finding container 23aa5d295dc4da2b00fc07e2bd07363ea1da8bf0e1304994067c06534b34007c: Status 404 returned error can't find the container with id 23aa5d295dc4da2b00fc07e2bd07363ea1da8bf0e1304994067c06534b34007c
Oct 30 13:23:10.031415 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:10.031364    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:23:10.031554 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:10.031468    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:23:10.031554 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:10.031522    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:23:10.032946 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:10.032923    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"]
Oct 30 13:23:10.265093 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:10.265055    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv" event={"ID":"c84b2e67-2800-4616-a136-a0912e553b55","Type":"ContainerStarted","Data":"61a94be2ddf37f7258e9fe8fdcdc0ae7f03feb95f7757778113d05cf9ae5b6d8"}
Oct 30 13:23:10.265265 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:10.265121    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv" event={"ID":"c84b2e67-2800-4616-a136-a0912e553b55","Type":"ContainerStarted","Data":"23aa5d295dc4da2b00fc07e2bd07363ea1da8bf0e1304994067c06534b34007c"}
Oct 30 13:23:10.327444 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:10.327375    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv" podStartSLOduration=1.327355035 podStartE2EDuration="1.327355035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:10.322857537 +0000 UTC m=+2285.369991493" watchObservedRunningTime="2026-10-30 13:23:10.327355035 +0000 UTC m=+2285.374488990"
Oct 30 13:23:10.695715 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:10.695617    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:23:10.893430 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:10.893394    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:11.037309 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:11.037232    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:11.274578 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:11.274526    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"]
Oct 30 13:23:11.274861 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:11.274829    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648" podUID="da3e4fe2-4f62-4385-9fc3-102dbda6c781" containerName="istio-proxy" containerID="cri-o://c67196c086b43443840eac544778cbbac903096f6637eaffa683a814a2fb3d98" gracePeriod=30
Oct 30 13:23:11.309563 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:11.309483    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:11.696509 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:11.694392    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:11.713263 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:11.697411    4544 patch_prober.go:29] interesting pod/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Oct 30 13:23:11.713263 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:11.697470    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv" podUID="c84b2e67-2800-4616-a136-a0912e553b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Oct 30 13:23:11.872291 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:11.870312    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:11.875115 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:11.873279    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cd2d2e-jk6vl"]
Oct 30 13:23:12.074203 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:12.049734    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cd2d2e-jk6vl"
Oct 30 13:23:12.076423 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:12.076382    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cd2d2e-jk6vl"]
Oct 30 13:23:12.158928 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:12.158897    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:23:12.159080 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:12.158999    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:23:12.257617 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:12.257579    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38b8eead-3d07-4df3-a422-49c83e202231-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cd2d2e-jk6vl\" (UID: \"38b8eead-3d07-4df3-a422-49c83e202231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cd2d2e-jk6vl"
Oct 30 13:23:12.257843 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:12.257640    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38b8eead-3d07-4df3-a422-49c83e202231-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cd2d2e-jk6vl\" (UID: \"38b8eead-3d07-4df3-a422-49c83e202231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cd2d2e-jk6vl"
Oct 30 13:23:12.257843 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:12.257725    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmjtc\" (UniqueName: \"kubernetes.io/projected/38b8eead-3d07-4df3-a422-49c83e202231-kube-api-access-vmjtc\") pod \"maas-api-revoke-keys-e2e-inf-b-cd2d2e-jk6vl\" (UID: \"38b8eead-3d07-4df3-a422-49c83e202231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cd2d2e-jk6vl"
Oct 30 13:23:12.358750 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:12.358659    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38b8eead-3d07-4df3-a422-49c83e202231-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cd2d2e-jk6vl\" (UID: \"38b8eead-3d07-4df3-a422-49c83e202231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cd2d2e-jk6vl"
Oct 30 13:23:12.358750 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:12.358726    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38b8eead-3d07-4df3-a422-49c83e202231-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cd2d2e-jk6vl\" (UID: \"38b8eead-3d07-4df3-a422-49c83e202231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cd2d2e-jk6vl"
Oct 30 13:23:12.359009 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:12.358762    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmjtc\" (UniqueName: \"kubernetes.io/projected/38b8eead-3d07-4df3-a422-49c83e202231-kube-api-access-vmjtc\") pod \"maas-api-revoke-keys-e2e-inf-b-cd2d2e-jk6vl\" (UID: \"38b8eead-3d07-4df3-a422-49c83e202231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cd2d2e-jk6vl"
Oct 30 13:23:12.359466 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:12.359441    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38b8eead-3d07-4df3-a422-49c83e202231-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cd2d2e-jk6vl\" (UID: \"38b8eead-3d07-4df3-a422-49c83e202231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cd2d2e-jk6vl"
Oct 30 13:23:12.361344 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:12.361323    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38b8eead-3d07-4df3-a422-49c83e202231-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cd2d2e-jk6vl\" (UID: \"38b8eead-3d07-4df3-a422-49c83e202231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cd2d2e-jk6vl"
Oct 30 13:23:12.384597 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:12.384566    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmjtc\" (UniqueName: \"kubernetes.io/projected/38b8eead-3d07-4df3-a422-49c83e202231-kube-api-access-vmjtc\") pod \"maas-api-revoke-keys-e2e-inf-b-cd2d2e-jk6vl\" (UID: \"38b8eead-3d07-4df3-a422-49c83e202231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cd2d2e-jk6vl"
Oct 30 13:23:12.434245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:12.427485    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cd2d2e-jk6vl"
Oct 30 13:23:12.479421 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:12.479364    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7db9402-5nkgf"]
Oct 30 13:23:12.487873 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:12.487828    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7db9402-5nkgf"
Oct 30 13:23:12.565424 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:12.564300    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7db9402-5nkgf"]
Oct 30 13:23:12.668452 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:12.667288    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chndw\" (UniqueName: \"kubernetes.io/projected/806e56d4-e04c-4b18-b18f-da7f5b44189c-kube-api-access-chndw\") pod \"maas-api-revoke-sub-c7db9402-5nkgf\" (UID: \"806e56d4-e04c-4b18-b18f-da7f5b44189c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7db9402-5nkgf"
Oct 30 13:23:12.668452 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:12.667355    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806e56d4-e04c-4b18-b18f-da7f5b44189c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7db9402-5nkgf\" (UID: \"806e56d4-e04c-4b18-b18f-da7f5b44189c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7db9402-5nkgf"
Oct 30 13:23:12.668452 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:12.667393    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806e56d4-e04c-4b18-b18f-da7f5b44189c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7db9402-5nkgf\" (UID: \"806e56d4-e04c-4b18-b18f-da7f5b44189c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7db9402-5nkgf"
Oct 30 13:23:12.696543 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:12.696504    4544 patch_prober.go:29] interesting pod/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Oct 30 13:23:12.696945 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:12.696577    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv" podUID="c84b2e67-2800-4616-a136-a0912e553b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Oct 30 13:23:12.768667 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:12.768626    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chndw\" (UniqueName: \"kubernetes.io/projected/806e56d4-e04c-4b18-b18f-da7f5b44189c-kube-api-access-chndw\") pod \"maas-api-revoke-sub-c7db9402-5nkgf\" (UID: \"806e56d4-e04c-4b18-b18f-da7f5b44189c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7db9402-5nkgf"
Oct 30 13:23:12.768840 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:12.768699    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806e56d4-e04c-4b18-b18f-da7f5b44189c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7db9402-5nkgf\" (UID: \"806e56d4-e04c-4b18-b18f-da7f5b44189c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7db9402-5nkgf"
Oct 30 13:23:12.768840 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:12.768736    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806e56d4-e04c-4b18-b18f-da7f5b44189c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7db9402-5nkgf\" (UID: \"806e56d4-e04c-4b18-b18f-da7f5b44189c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7db9402-5nkgf"
Oct 30 13:23:12.769564 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:12.769542    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806e56d4-e04c-4b18-b18f-da7f5b44189c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7db9402-5nkgf\" (UID: \"806e56d4-e04c-4b18-b18f-da7f5b44189c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7db9402-5nkgf"
Oct 30 13:23:12.771305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:12.771276    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806e56d4-e04c-4b18-b18f-da7f5b44189c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7db9402-5nkgf\" (UID: \"806e56d4-e04c-4b18-b18f-da7f5b44189c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7db9402-5nkgf"
Oct 30 13:23:12.783119 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:12.783066    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cd2d2e-jk6vl"]
Oct 30 13:23:12.786360 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:23:12.785001    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38b8eead_3d07_4df3_a422_49c83e202231.slice/crio-32a7422255ebe77d51287c4d4663a92f8a0ed4323f5a3532105223f4f564f907 WatchSource:0}: Error finding container 32a7422255ebe77d51287c4d4663a92f8a0ed4323f5a3532105223f4f564f907: Status 404 returned error can't find the container with id 32a7422255ebe77d51287c4d4663a92f8a0ed4323f5a3532105223f4f564f907
Oct 30 13:23:12.798748 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:12.798720    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chndw\" (UniqueName: \"kubernetes.io/projected/806e56d4-e04c-4b18-b18f-da7f5b44189c-kube-api-access-chndw\") pod \"maas-api-revoke-sub-c7db9402-5nkgf\" (UID: \"806e56d4-e04c-4b18-b18f-da7f5b44189c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7db9402-5nkgf"
Oct 30 13:23:12.809982 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:12.809952    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7db9402-5nkgf"
Oct 30 13:23:13.150011 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:13.149978    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7db9402-5nkgf"]
Oct 30 13:23:13.153847 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:23:13.153817    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod806e56d4_e04c_4b18_b18f_da7f5b44189c.slice/crio-03377a44af8e23f3389c776068aac15fd39eb4c1d00681d1a5415744e9457970 WatchSource:0}: Error finding container 03377a44af8e23f3389c776068aac15fd39eb4c1d00681d1a5415744e9457970: Status 404 returned error can't find the container with id 03377a44af8e23f3389c776068aac15fd39eb4c1d00681d1a5415744e9457970
Oct 30 13:23:13.302207 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:13.302167    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cd2d2e-jk6vl" event={"ID":"38b8eead-3d07-4df3-a422-49c83e202231","Type":"ContainerStarted","Data":"32a7422255ebe77d51287c4d4663a92f8a0ed4323f5a3532105223f4f564f907"}
Oct 30 13:23:13.303547 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:13.303521    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7db9402-5nkgf" event={"ID":"806e56d4-e04c-4b18-b18f-da7f5b44189c","Type":"ContainerStarted","Data":"03377a44af8e23f3389c776068aac15fd39eb4c1d00681d1a5415744e9457970"}
Oct 30 13:23:13.695840 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:13.695806    4544 patch_prober.go:29] interesting pod/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Oct 30 13:23:13.696033 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:13.695858    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv" podUID="c84b2e67-2800-4616-a136-a0912e553b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Oct 30 13:23:14.358036 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:14.357998    4544 generic.go:356] "Generic (PLEG): container finished" podID="38b8eead-3d07-4df3-a422-49c83e202231" containerID="a4fe84d90cb5fbb0c1ea3e625479f530b7214a0948353b81ec20bf213122ba10" exitCode=0
Oct 30 13:23:14.358447 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:14.358290    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cd2d2e-jk6vl" event={"ID":"38b8eead-3d07-4df3-a422-49c83e202231","Type":"ContainerDied","Data":"a4fe84d90cb5fbb0c1ea3e625479f530b7214a0948353b81ec20bf213122ba10"}
Oct 30 13:23:14.360048 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:14.360021    4544 generic.go:356] "Generic (PLEG): container finished" podID="806e56d4-e04c-4b18-b18f-da7f5b44189c" containerID="8cc767c52b53d99484cf704e793c8d70899ac29782298a414bb938de35546d61" exitCode=0
Oct 30 13:23:14.360177 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:14.360155    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7db9402-5nkgf" event={"ID":"806e56d4-e04c-4b18-b18f-da7f5b44189c","Type":"ContainerDied","Data":"8cc767c52b53d99484cf704e793c8d70899ac29782298a414bb938de35546d61"}
Oct 30 13:23:14.440314 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:14.440045    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:14.672341 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:14.672306    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:14.695560 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:14.695523    4544 patch_prober.go:29] interesting pod/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Oct 30 13:23:14.695763 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:14.695582    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv" podUID="c84b2e67-2800-4616-a136-a0912e553b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Oct 30 13:23:15.597518 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:15.597487    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cd2d2e-jk6vl"
Oct 30 13:23:15.649873 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:15.649838    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38b8eead-3d07-4df3-a422-49c83e202231-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38b8eead-3d07-4df3-a422-49c83e202231-maas-api-openshift-ca\") pod \"38b8eead-3d07-4df3-a422-49c83e202231\" (UID: \"38b8eead-3d07-4df3-a422-49c83e202231\") "
Oct 30 13:23:15.650055 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:15.649907    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38b8eead-3d07-4df3-a422-49c83e202231-kube-api-access-vmjtc\" (UniqueName: \"kubernetes.io/projected/38b8eead-3d07-4df3-a422-49c83e202231-kube-api-access-vmjtc\") pod \"38b8eead-3d07-4df3-a422-49c83e202231\" (UID: \"38b8eead-3d07-4df3-a422-49c83e202231\") "
Oct 30 13:23:15.650055 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:15.649951    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38b8eead-3d07-4df3-a422-49c83e202231-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38b8eead-3d07-4df3-a422-49c83e202231-maas-api-xks-ca\") pod \"38b8eead-3d07-4df3-a422-49c83e202231\" (UID: \"38b8eead-3d07-4df3-a422-49c83e202231\") "
Oct 30 13:23:15.651080 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:15.651060    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38b8eead-3d07-4df3-a422-49c83e202231-maas-api-openshift-ca" pod "38b8eead-3d07-4df3-a422-49c83e202231" (UID: "38b8eead-3d07-4df3-a422-49c83e202231"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:23:15.656051 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:15.656023    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38b8eead-3d07-4df3-a422-49c83e202231-maas-api-xks-ca" pod "38b8eead-3d07-4df3-a422-49c83e202231" (UID: "38b8eead-3d07-4df3-a422-49c83e202231"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:23:15.656177 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:15.656160    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38b8eead-3d07-4df3-a422-49c83e202231-kube-api-access-vmjtc" pod "38b8eead-3d07-4df3-a422-49c83e202231" (UID: "38b8eead-3d07-4df3-a422-49c83e202231"). InnerVolumeSpecName "kube-api-access-vmjtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:23:15.684394 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:15.684373    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7db9402-5nkgf"
Oct 30 13:23:15.696352 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:15.696321    4544 patch_prober.go:29] interesting pod/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Oct 30 13:23:15.696473 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:15.696376    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv" podUID="c84b2e67-2800-4616-a136-a0912e553b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Oct 30 13:23:15.751629 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:15.751592    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/806e56d4-e04c-4b18-b18f-da7f5b44189c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806e56d4-e04c-4b18-b18f-da7f5b44189c-maas-api-openshift-ca\") pod \"806e56d4-e04c-4b18-b18f-da7f5b44189c\" (UID: \"806e56d4-e04c-4b18-b18f-da7f5b44189c\") "
Oct 30 13:23:15.751824 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:15.751646    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/806e56d4-e04c-4b18-b18f-da7f5b44189c-kube-api-access-chndw\" (UniqueName: \"kubernetes.io/projected/806e56d4-e04c-4b18-b18f-da7f5b44189c-kube-api-access-chndw\") pod \"806e56d4-e04c-4b18-b18f-da7f5b44189c\" (UID: \"806e56d4-e04c-4b18-b18f-da7f5b44189c\") "
Oct 30 13:23:15.751824 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:15.751743    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/806e56d4-e04c-4b18-b18f-da7f5b44189c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806e56d4-e04c-4b18-b18f-da7f5b44189c-maas-api-xks-ca\") pod \"806e56d4-e04c-4b18-b18f-da7f5b44189c\" (UID: \"806e56d4-e04c-4b18-b18f-da7f5b44189c\") "
Oct 30 13:23:15.752001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:15.751976    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/806e56d4-e04c-4b18-b18f-da7f5b44189c-maas-api-openshift-ca" pod "806e56d4-e04c-4b18-b18f-da7f5b44189c" (UID: "806e56d4-e04c-4b18-b18f-da7f5b44189c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:23:15.752551 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:15.752526    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38b8eead-3d07-4df3-a422-49c83e202231-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:23:15.752551 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:15.752547    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmjtc\" (UniqueName: \"kubernetes.io/projected/38b8eead-3d07-4df3-a422-49c83e202231-kube-api-access-vmjtc\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:23:15.752739 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:15.752557    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38b8eead-3d07-4df3-a422-49c83e202231-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:23:15.752739 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:15.752566    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806e56d4-e04c-4b18-b18f-da7f5b44189c-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:23:15.753830 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:15.753802    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/806e56d4-e04c-4b18-b18f-da7f5b44189c-kube-api-access-chndw" pod "806e56d4-e04c-4b18-b18f-da7f5b44189c" (UID: "806e56d4-e04c-4b18-b18f-da7f5b44189c"). InnerVolumeSpecName "kube-api-access-chndw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:23:15.753948 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:15.753932    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/806e56d4-e04c-4b18-b18f-da7f5b44189c-maas-api-xks-ca" pod "806e56d4-e04c-4b18-b18f-da7f5b44189c" (UID: "806e56d4-e04c-4b18-b18f-da7f5b44189c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:23:15.858461 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:15.856026    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chndw\" (UniqueName: \"kubernetes.io/projected/806e56d4-e04c-4b18-b18f-da7f5b44189c-kube-api-access-chndw\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:23:15.858461 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:15.856068    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806e56d4-e04c-4b18-b18f-da7f5b44189c-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:23:16.376796 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.376771    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cd2d2e-jk6vl"
Oct 30 13:23:16.376796 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.376788    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cd2d2e-jk6vl" event={"ID":"38b8eead-3d07-4df3-a422-49c83e202231","Type":"ContainerDied","Data":"32a7422255ebe77d51287c4d4663a92f8a0ed4323f5a3532105223f4f564f907"}
Oct 30 13:23:16.377030 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.376842    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32a7422255ebe77d51287c4d4663a92f8a0ed4323f5a3532105223f4f564f907"
Oct 30 13:23:16.378258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.378236    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7db9402-5nkgf"
Oct 30 13:23:16.378258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.378249    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7db9402-5nkgf" event={"ID":"806e56d4-e04c-4b18-b18f-da7f5b44189c","Type":"ContainerDied","Data":"03377a44af8e23f3389c776068aac15fd39eb4c1d00681d1a5415744e9457970"}
Oct 30 13:23:16.378481 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.378291    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03377a44af8e23f3389c776068aac15fd39eb4c1d00681d1a5415744e9457970"
Oct 30 13:23:16.515253 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.512165    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:16.695482 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.695444    4544 patch_prober.go:29] interesting pod/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Oct 30 13:23:16.695958 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.695510    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv" podUID="c84b2e67-2800-4616-a136-a0912e553b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Oct 30 13:23:16.737445 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.737412    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:23:16.879628 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.879544    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istiod-ca-cert\") pod \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") "
Oct 30 13:23:16.879796 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.879660    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istio-podinfo\") pod \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") "
Oct 30 13:23:16.879796 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.879758    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da3e4fe2-4f62-4385-9fc3-102dbda6c781-kube-api-access-vjtbk\" (UniqueName: \"kubernetes.io/projected/da3e4fe2-4f62-4385-9fc3-102dbda6c781-kube-api-access-vjtbk\") pod \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") "
Oct 30 13:23:16.879908 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.879869    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istio-token\" (UniqueName: \"kubernetes.io/projected/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istio-token\") pod \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") "
Oct 30 13:23:16.879991 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.879948    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-workload-socket\") pod \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") "
Oct 30 13:23:16.880058 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.880010    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istiod-ca-cert" pod "da3e4fe2-4f62-4385-9fc3-102dbda6c781" (UID: "da3e4fe2-4f62-4385-9fc3-102dbda6c781"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:23:16.880058 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.880046    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istio-envoy\") pod \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") "
Oct 30 13:23:16.880174 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.880094    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-credential-socket\") pod \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") "
Oct 30 13:23:16.880174 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.880152    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istio-data\") pod \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") "
Oct 30 13:23:16.880321 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.880197    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-workload-certs\") pod \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\" (UID: \"da3e4fe2-4f62-4385-9fc3-102dbda6c781\") "
Oct 30 13:23:16.880434 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.880407    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istio-data" pod "da3e4fe2-4f62-4385-9fc3-102dbda6c781" (UID: "da3e4fe2-4f62-4385-9fc3-102dbda6c781"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:23:16.880513 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.880416    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-workload-socket" pod "da3e4fe2-4f62-4385-9fc3-102dbda6c781" (UID: "da3e4fe2-4f62-4385-9fc3-102dbda6c781"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:23:16.880581 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.880556    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-workload-certs" pod "da3e4fe2-4f62-4385-9fc3-102dbda6c781" (UID: "da3e4fe2-4f62-4385-9fc3-102dbda6c781"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:23:16.881031 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.880965    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:23:16.881031 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.880990    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:23:16.881031 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.881004    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:23:16.881031 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.881017    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:23:16.881448 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.881311    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-credential-socket" pod "da3e4fe2-4f62-4385-9fc3-102dbda6c781" (UID: "da3e4fe2-4f62-4385-9fc3-102dbda6c781"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:23:16.882507 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.882481    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istio-envoy" pod "da3e4fe2-4f62-4385-9fc3-102dbda6c781" (UID: "da3e4fe2-4f62-4385-9fc3-102dbda6c781"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:23:16.882628 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.882549    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3e4fe2-4f62-4385-9fc3-102dbda6c781-kube-api-access-vjtbk" pod "da3e4fe2-4f62-4385-9fc3-102dbda6c781" (UID: "da3e4fe2-4f62-4385-9fc3-102dbda6c781"). InnerVolumeSpecName "kube-api-access-vjtbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:23:16.882708 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.882691    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istio-token" pod "da3e4fe2-4f62-4385-9fc3-102dbda6c781" (UID: "da3e4fe2-4f62-4385-9fc3-102dbda6c781"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:23:16.882779 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.882756    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istio-podinfo" pod "da3e4fe2-4f62-4385-9fc3-102dbda6c781" (UID: "da3e4fe2-4f62-4385-9fc3-102dbda6c781"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:23:16.982578 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.982543    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:23:16.982578 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.982571    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3e4fe2-4f62-4385-9fc3-102dbda6c781-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:23:16.982578 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.982580    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:23:16.982875 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.982590    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjtbk\" (UniqueName: \"kubernetes.io/projected/da3e4fe2-4f62-4385-9fc3-102dbda6c781-kube-api-access-vjtbk\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:23:16.982875 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:16.982599    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da3e4fe2-4f62-4385-9fc3-102dbda6c781-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:23:17.094774 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:17.094728    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg"
Oct 30 13:23:17.094957 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:17.094819    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg"
Oct 30 13:23:17.114537 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:17.114508    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg"
Oct 30 13:23:17.388462 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:17.388424    4544 generic.go:356] "Generic (PLEG): container finished" podID="da3e4fe2-4f62-4385-9fc3-102dbda6c781" containerID="c67196c086b43443840eac544778cbbac903096f6637eaffa683a814a2fb3d98" exitCode=0
Oct 30 13:23:17.388645 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:17.388486    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"
Oct 30 13:23:17.388645 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:17.388511    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648" event={"ID":"da3e4fe2-4f62-4385-9fc3-102dbda6c781","Type":"ContainerDied","Data":"c67196c086b43443840eac544778cbbac903096f6637eaffa683a814a2fb3d98"}
Oct 30 13:23:17.388645 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:17.388565    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648" event={"ID":"da3e4fe2-4f62-4385-9fc3-102dbda6c781","Type":"ContainerDied","Data":"d987c9e8106a2e0b529768ef0999ca1961226977ea5cfede472967bdb8222416"}
Oct 30 13:23:17.388645 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:17.388586    4544 scope.go:122] "RemoveContainer" containerID="c67196c086b43443840eac544778cbbac903096f6637eaffa683a814a2fb3d98"
Oct 30 13:23:17.399786 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:17.399765    4544 scope.go:122] "RemoveContainer" containerID="bc2ef4441794765bc7ae114875739124ce0b10ba8092dee461d869ff0616b0fa"
Oct 30 13:23:17.409650 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:17.409631    4544 scope.go:122] "RemoveContainer" containerID="c67196c086b43443840eac544778cbbac903096f6637eaffa683a814a2fb3d98"
Oct 30 13:23:17.409939 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:23:17.409920    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c67196c086b43443840eac544778cbbac903096f6637eaffa683a814a2fb3d98\": container with ID starting with c67196c086b43443840eac544778cbbac903096f6637eaffa683a814a2fb3d98 not found: ID does not exist" containerID="c67196c086b43443840eac544778cbbac903096f6637eaffa683a814a2fb3d98"
Oct 30 13:23:17.409991 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:17.409947    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c67196c086b43443840eac544778cbbac903096f6637eaffa683a814a2fb3d98"} err="failed to get container status \"c67196c086b43443840eac544778cbbac903096f6637eaffa683a814a2fb3d98\": rpc error: code = NotFound desc = could not find container \"c67196c086b43443840eac544778cbbac903096f6637eaffa683a814a2fb3d98\": container with ID starting with c67196c086b43443840eac544778cbbac903096f6637eaffa683a814a2fb3d98 not found: ID does not exist"
Oct 30 13:23:17.409991 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:17.409963    4544 scope.go:122] "RemoveContainer" containerID="bc2ef4441794765bc7ae114875739124ce0b10ba8092dee461d869ff0616b0fa"
Oct 30 13:23:17.410252 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:23:17.410200    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc2ef4441794765bc7ae114875739124ce0b10ba8092dee461d869ff0616b0fa\": container with ID starting with bc2ef4441794765bc7ae114875739124ce0b10ba8092dee461d869ff0616b0fa not found: ID does not exist" containerID="bc2ef4441794765bc7ae114875739124ce0b10ba8092dee461d869ff0616b0fa"
Oct 30 13:23:17.410311 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:17.410252    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc2ef4441794765bc7ae114875739124ce0b10ba8092dee461d869ff0616b0fa"} err="failed to get container status \"bc2ef4441794765bc7ae114875739124ce0b10ba8092dee461d869ff0616b0fa\": rpc error: code = NotFound desc = could not find container \"bc2ef4441794765bc7ae114875739124ce0b10ba8092dee461d869ff0616b0fa\": container with ID starting with bc2ef4441794765bc7ae114875739124ce0b10ba8092dee461d869ff0616b0fa not found: ID does not exist"
Oct 30 13:23:17.445307 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:17.445286    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"]
Oct 30 13:23:17.452398 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:17.452365    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648"]
Oct 30 13:23:17.700440 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:17.700369    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:23:17.700866 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:17.700848    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:23:17.701590 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:17.701569    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:23:18.495115 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:18.495084    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:18.655245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:18.653063    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:19.088080 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:19.088048    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da3e4fe2-4f62-4385-9fc3-102dbda6c781" path="/var/lib/kubelet/pods/da3e4fe2-4f62-4385-9fc3-102dbda6c781/volumes"
Oct 30 13:23:20.393649 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:20.393606    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:22.150183 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:22.150151    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:22.485204 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:22.485107    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:22.630559 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:22.630524    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:23.505985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:23.505948    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:24.345244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:24.342679    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:26.622137 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:26.622097    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:27.973391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:27.973352    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv"]
Oct 30 13:23:27.974206 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:27.974188    4544 cpu_manager.go:414] "containerMap: removing container" podUID="806e56d4-e04c-4b18-b18f-da7f5b44189c" containerName="revoke-keys"
Oct 30 13:23:27.974275 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:27.974206    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="806e56d4-e04c-4b18-b18f-da7f5b44189c" containerName="revoke-keys"
Oct 30 13:23:27.974275 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:27.974238    4544 cpu_manager.go:414] "containerMap: removing container" podUID="38b8eead-3d07-4df3-a422-49c83e202231" containerName="revoke-keys"
Oct 30 13:23:27.974275 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:27.974245    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38b8eead-3d07-4df3-a422-49c83e202231" containerName="revoke-keys"
Oct 30 13:23:27.974275 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:27.974258    4544 cpu_manager.go:414] "containerMap: removing container" podUID="da3e4fe2-4f62-4385-9fc3-102dbda6c781" containerName="istio-proxy"
Oct 30 13:23:27.974275 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:27.974264    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da3e4fe2-4f62-4385-9fc3-102dbda6c781" containerName="istio-proxy"
Oct 30 13:23:27.974444 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:27.974278    4544 cpu_manager.go:414] "containerMap: removing container" podUID="da3e4fe2-4f62-4385-9fc3-102dbda6c781" containerName="istio-proxy"
Oct 30 13:23:27.974444 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:27.974283    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da3e4fe2-4f62-4385-9fc3-102dbda6c781" containerName="istio-proxy"
Oct 30 13:23:27.974444 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:27.974396    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="806e56d4-e04c-4b18-b18f-da7f5b44189c" containerName="revoke-keys"
Oct 30 13:23:27.974444 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:27.974407    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="38b8eead-3d07-4df3-a422-49c83e202231" containerName="revoke-keys"
Oct 30 13:23:27.974444 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:27.974415    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="da3e4fe2-4f62-4385-9fc3-102dbda6c781" containerName="istio-proxy"
Oct 30 13:23:27.974444 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:27.974423    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="da3e4fe2-4f62-4385-9fc3-102dbda6c781" containerName="istio-proxy"
Oct 30 13:23:27.979783 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:27.979751    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv"
Oct 30 13:23:27.999158 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:27.999127    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-0e874e\"/\"auth-test-model-0e874e-kserve-self-signed-certs\""
Oct 30 13:23:28.011553 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:28.011522    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-0e874e\"/\"kube-root-ca.crt\""
Oct 30 13:23:28.011763 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:28.011522    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-0e874e\"/\"openshift-service-ca.crt\""
Oct 30 13:23:28.022252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:28.019062    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv"]
Oct 30 13:23:28.047345 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:28.047300    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:28.099904 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:28.099867    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-0e874e\"/\"default-dockercfg-tqsdq\""
Oct 30 13:23:28.100833 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:28.100788    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7739d642-eb43-4056-ae8b-ace5f85e9155-tmp-dir\") pod \"auth-test-model-0e874e-kserve-b784968df-2wwfv\" (UID: \"7739d642-eb43-4056-ae8b-ace5f85e9155\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv"
Oct 30 13:23:28.101021 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:28.101003    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7739d642-eb43-4056-ae8b-ace5f85e9155-dshm\") pod \"auth-test-model-0e874e-kserve-b784968df-2wwfv\" (UID: \"7739d642-eb43-4056-ae8b-ace5f85e9155\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv"
Oct 30 13:23:28.101275 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:28.101196    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7739d642-eb43-4056-ae8b-ace5f85e9155-home\") pod \"auth-test-model-0e874e-kserve-b784968df-2wwfv\" (UID: \"7739d642-eb43-4056-ae8b-ace5f85e9155\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv"
Oct 30 13:23:28.101386 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:28.101367    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7739d642-eb43-4056-ae8b-ace5f85e9155-tls-certs\") pod \"auth-test-model-0e874e-kserve-b784968df-2wwfv\" (UID: \"7739d642-eb43-4056-ae8b-ace5f85e9155\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv"
Oct 30 13:23:28.101538 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:28.101517    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7739d642-eb43-4056-ae8b-ace5f85e9155-model-cache\") pod \"auth-test-model-0e874e-kserve-b784968df-2wwfv\" (UID: \"7739d642-eb43-4056-ae8b-ace5f85e9155\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv"
Oct 30 13:23:28.102007 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:28.101979    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dms82\" (UniqueName: \"kubernetes.io/projected/7739d642-eb43-4056-ae8b-ace5f85e9155-kube-api-access-dms82\") pod \"auth-test-model-0e874e-kserve-b784968df-2wwfv\" (UID: \"7739d642-eb43-4056-ae8b-ace5f85e9155\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv"
Oct 30 13:23:28.203397 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:28.203355    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7739d642-eb43-4056-ae8b-ace5f85e9155-dshm\") pod \"auth-test-model-0e874e-kserve-b784968df-2wwfv\" (UID: \"7739d642-eb43-4056-ae8b-ace5f85e9155\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv"
Oct 30 13:23:28.203647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:28.203439    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7739d642-eb43-4056-ae8b-ace5f85e9155-home\") pod \"auth-test-model-0e874e-kserve-b784968df-2wwfv\" (UID: \"7739d642-eb43-4056-ae8b-ace5f85e9155\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv"
Oct 30 13:23:28.203647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:28.203481    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7739d642-eb43-4056-ae8b-ace5f85e9155-tls-certs\") pod \"auth-test-model-0e874e-kserve-b784968df-2wwfv\" (UID: \"7739d642-eb43-4056-ae8b-ace5f85e9155\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv"
Oct 30 13:23:28.203647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:28.203526    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7739d642-eb43-4056-ae8b-ace5f85e9155-model-cache\") pod \"auth-test-model-0e874e-kserve-b784968df-2wwfv\" (UID: \"7739d642-eb43-4056-ae8b-ace5f85e9155\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv"
Oct 30 13:23:28.203647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:28.203613    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dms82\" (UniqueName: \"kubernetes.io/projected/7739d642-eb43-4056-ae8b-ace5f85e9155-kube-api-access-dms82\") pod \"auth-test-model-0e874e-kserve-b784968df-2wwfv\" (UID: \"7739d642-eb43-4056-ae8b-ace5f85e9155\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv"
Oct 30 13:23:28.203898 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:28.203675    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7739d642-eb43-4056-ae8b-ace5f85e9155-tmp-dir\") pod \"auth-test-model-0e874e-kserve-b784968df-2wwfv\" (UID: \"7739d642-eb43-4056-ae8b-ace5f85e9155\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv"
Oct 30 13:23:28.203898 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:28.203861    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7739d642-eb43-4056-ae8b-ace5f85e9155-home\") pod \"auth-test-model-0e874e-kserve-b784968df-2wwfv\" (UID: \"7739d642-eb43-4056-ae8b-ace5f85e9155\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv"
Oct 30 13:23:28.204002 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:28.203921    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7739d642-eb43-4056-ae8b-ace5f85e9155-model-cache\") pod \"auth-test-model-0e874e-kserve-b784968df-2wwfv\" (UID: \"7739d642-eb43-4056-ae8b-ace5f85e9155\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv"
Oct 30 13:23:28.204061 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:28.204048    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7739d642-eb43-4056-ae8b-ace5f85e9155-tmp-dir\") pod \"auth-test-model-0e874e-kserve-b784968df-2wwfv\" (UID: \"7739d642-eb43-4056-ae8b-ace5f85e9155\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv"
Oct 30 13:23:28.205961 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:28.205933    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7739d642-eb43-4056-ae8b-ace5f85e9155-dshm\") pod \"auth-test-model-0e874e-kserve-b784968df-2wwfv\" (UID: \"7739d642-eb43-4056-ae8b-ace5f85e9155\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv"
Oct 30 13:23:28.206112 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:28.206100    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7739d642-eb43-4056-ae8b-ace5f85e9155-tls-certs\") pod \"auth-test-model-0e874e-kserve-b784968df-2wwfv\" (UID: \"7739d642-eb43-4056-ae8b-ace5f85e9155\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv"
Oct 30 13:23:28.227858 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:28.227779    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dms82\" (UniqueName: \"kubernetes.io/projected/7739d642-eb43-4056-ae8b-ace5f85e9155-kube-api-access-dms82\") pod \"auth-test-model-0e874e-kserve-b784968df-2wwfv\" (UID: \"7739d642-eb43-4056-ae8b-ace5f85e9155\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv"
Oct 30 13:23:28.300887 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:28.299533    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv"
Oct 30 13:23:28.386989 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:28.386946    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:28.676383 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:28.676351    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv"]
Oct 30 13:23:28.676913 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:23:28.676887    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7739d642_eb43_4056_ae8b_ace5f85e9155.slice/crio-a6037d5de8c23879c1db1e81f4ec1b41c4a743625c7612b702b280c7a4a24ae0 WatchSource:0}: Error finding container a6037d5de8c23879c1db1e81f4ec1b41c4a743625c7612b702b280c7a4a24ae0: Status 404 returned error can't find the container with id a6037d5de8c23879c1db1e81f4ec1b41c4a743625c7612b702b280c7a4a24ae0
Oct 30 13:23:29.008076 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:23:29.008042    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:23:29.549050 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:29.549014    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv" event={"ID":"7739d642-eb43-4056-ae8b-ace5f85e9155","Type":"ContainerStarted","Data":"4d2ffd282929303e855066378909503f6c1f38695df2efb5c7b34e7b6bdf147b"}
Oct 30 13:23:29.549050 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:29.549056    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv" event={"ID":"7739d642-eb43-4056-ae8b-ace5f85e9155","Type":"ContainerStarted","Data":"a6037d5de8c23879c1db1e81f4ec1b41c4a743625c7612b702b280c7a4a24ae0"}
Oct 30 13:23:29.595374 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:29.595321    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv" podStartSLOduration=2.595306718 podStartE2EDuration="2.595306718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:29.590060462 +0000 UTC m=+2304.637194415" watchObservedRunningTime="2026-10-30 13:23:29.595306718 +0000 UTC m=+2304.642440670"
Oct 30 13:23:30.300347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:30.300315    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv"
Oct 30 13:23:30.399758 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:30.399725    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:31.363783 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:31.363747    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:32.221322 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:32.221287    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:33.971178 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:33.971134    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:35.948242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:35.941388    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:37.759069 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:37.759032    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:38.300305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:38.300272    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv"
Oct 30 13:23:38.313121 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:38.313091    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv"
Oct 30 13:23:38.632783 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:38.632752    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv"
Oct 30 13:23:39.565448 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:39.565404    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:40.343553 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:40.343521    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz"]
Oct 30 13:23:40.383797 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:40.383760    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:40.634487 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:40.634402    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80fbd9-6ncg8"]
Oct 30 13:23:40.641426 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:40.641400    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80fbd9-6ncg8"
Oct 30 13:23:40.681194 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:40.681170    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:23:40.686293 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:40.686267    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80fbd9-6ncg8"]
Oct 30 13:23:40.691541 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:40.691516    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:23:40.741166 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:40.741132    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/710a5656-e79a-4c00-8df1-99bcbd92e400-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-80fbd9-6ncg8\" (UID: \"710a5656-e79a-4c00-8df1-99bcbd92e400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80fbd9-6ncg8"
Oct 30 13:23:40.741441 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:40.741412    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/710a5656-e79a-4c00-8df1-99bcbd92e400-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-80fbd9-6ncg8\" (UID: \"710a5656-e79a-4c00-8df1-99bcbd92e400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80fbd9-6ncg8"
Oct 30 13:23:40.741496 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:40.741458    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5dxt\" (UniqueName: \"kubernetes.io/projected/710a5656-e79a-4c00-8df1-99bcbd92e400-kube-api-access-r5dxt\") pod \"maas-api-revoke-keys-e2e-inf-a-80fbd9-6ncg8\" (UID: \"710a5656-e79a-4c00-8df1-99bcbd92e400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80fbd9-6ncg8"
Oct 30 13:23:40.842920 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:40.842882    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/710a5656-e79a-4c00-8df1-99bcbd92e400-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-80fbd9-6ncg8\" (UID: \"710a5656-e79a-4c00-8df1-99bcbd92e400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80fbd9-6ncg8"
Oct 30 13:23:40.843142 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:40.843097    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/710a5656-e79a-4c00-8df1-99bcbd92e400-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-80fbd9-6ncg8\" (UID: \"710a5656-e79a-4c00-8df1-99bcbd92e400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80fbd9-6ncg8"
Oct 30 13:23:40.843142 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:40.843128    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5dxt\" (UniqueName: \"kubernetes.io/projected/710a5656-e79a-4c00-8df1-99bcbd92e400-kube-api-access-r5dxt\") pod \"maas-api-revoke-keys-e2e-inf-a-80fbd9-6ncg8\" (UID: \"710a5656-e79a-4c00-8df1-99bcbd92e400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80fbd9-6ncg8"
Oct 30 13:23:40.843868 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:40.843842    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/710a5656-e79a-4c00-8df1-99bcbd92e400-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-80fbd9-6ncg8\" (UID: \"710a5656-e79a-4c00-8df1-99bcbd92e400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80fbd9-6ncg8"
Oct 30 13:23:40.845382 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:40.845356    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/710a5656-e79a-4c00-8df1-99bcbd92e400-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-80fbd9-6ncg8\" (UID: \"710a5656-e79a-4c00-8df1-99bcbd92e400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80fbd9-6ncg8"
Oct 30 13:23:40.867881 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:40.867854    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5dxt\" (UniqueName: \"kubernetes.io/projected/710a5656-e79a-4c00-8df1-99bcbd92e400-kube-api-access-r5dxt\") pod \"maas-api-revoke-keys-e2e-inf-a-80fbd9-6ncg8\" (UID: \"710a5656-e79a-4c00-8df1-99bcbd92e400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80fbd9-6ncg8"
Oct 30 13:23:40.957000 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:40.956913    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80fbd9-6ncg8"
Oct 30 13:23:41.281562 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:41.281527    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80fbd9-6ncg8"]
Oct 30 13:23:41.282012 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:23:41.281986    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod710a5656_e79a_4c00_8df1_99bcbd92e400.slice/crio-1828721e437f703a9b9b0b56dd6b31aaadc5e6addcad00f2a21475a2ca736945 WatchSource:0}: Error finding container 1828721e437f703a9b9b0b56dd6b31aaadc5e6addcad00f2a21475a2ca736945: Status 404 returned error can't find the container with id 1828721e437f703a9b9b0b56dd6b31aaadc5e6addcad00f2a21475a2ca736945
Oct 30 13:23:41.576733 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:41.576697    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:41.680288 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:41.680249    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80fbd9-6ncg8" event={"ID":"710a5656-e79a-4c00-8df1-99bcbd92e400","Type":"ContainerStarted","Data":"1828721e437f703a9b9b0b56dd6b31aaadc5e6addcad00f2a21475a2ca736945"}
Oct 30 13:23:42.690886 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:42.690799    4544 generic.go:356] "Generic (PLEG): container finished" podID="710a5656-e79a-4c00-8df1-99bcbd92e400" containerID="74e100dc43c2c66516f42c81f5df8dde200f49202da0bfb7c93d787c50e70102" exitCode=0
Oct 30 13:23:42.690886 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:42.690854    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80fbd9-6ncg8" event={"ID":"710a5656-e79a-4c00-8df1-99bcbd92e400","Type":"ContainerDied","Data":"74e100dc43c2c66516f42c81f5df8dde200f49202da0bfb7c93d787c50e70102"}
Oct 30 13:23:43.406888 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:43.406815    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:43.867535 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:43.867505    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80fbd9-6ncg8"
Oct 30 13:23:43.894854 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:43.894824    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/710a5656-e79a-4c00-8df1-99bcbd92e400-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/710a5656-e79a-4c00-8df1-99bcbd92e400-maas-api-openshift-ca\") pod \"710a5656-e79a-4c00-8df1-99bcbd92e400\" (UID: \"710a5656-e79a-4c00-8df1-99bcbd92e400\") "
Oct 30 13:23:43.895039 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:43.894883    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/710a5656-e79a-4c00-8df1-99bcbd92e400-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/710a5656-e79a-4c00-8df1-99bcbd92e400-maas-api-xks-ca\") pod \"710a5656-e79a-4c00-8df1-99bcbd92e400\" (UID: \"710a5656-e79a-4c00-8df1-99bcbd92e400\") "
Oct 30 13:23:43.895039 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:43.894940    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/710a5656-e79a-4c00-8df1-99bcbd92e400-kube-api-access-r5dxt\" (UniqueName: \"kubernetes.io/projected/710a5656-e79a-4c00-8df1-99bcbd92e400-kube-api-access-r5dxt\") pod \"710a5656-e79a-4c00-8df1-99bcbd92e400\" (UID: \"710a5656-e79a-4c00-8df1-99bcbd92e400\") "
Oct 30 13:23:43.895208 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:43.895181    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/710a5656-e79a-4c00-8df1-99bcbd92e400-maas-api-openshift-ca" pod "710a5656-e79a-4c00-8df1-99bcbd92e400" (UID: "710a5656-e79a-4c00-8df1-99bcbd92e400"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:23:43.895581 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:43.895560    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/710a5656-e79a-4c00-8df1-99bcbd92e400-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:23:43.897043 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:43.897017    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/710a5656-e79a-4c00-8df1-99bcbd92e400-maas-api-xks-ca" pod "710a5656-e79a-4c00-8df1-99bcbd92e400" (UID: "710a5656-e79a-4c00-8df1-99bcbd92e400"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:23:43.897200 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:43.897182    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/710a5656-e79a-4c00-8df1-99bcbd92e400-kube-api-access-r5dxt" pod "710a5656-e79a-4c00-8df1-99bcbd92e400" (UID: "710a5656-e79a-4c00-8df1-99bcbd92e400"). InnerVolumeSpecName "kube-api-access-r5dxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:23:43.997049 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:43.996963    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/710a5656-e79a-4c00-8df1-99bcbd92e400-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:23:43.997049 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:43.996995    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5dxt\" (UniqueName: \"kubernetes.io/projected/710a5656-e79a-4c00-8df1-99bcbd92e400-kube-api-access-r5dxt\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:23:44.709058 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:44.709030    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80fbd9-6ncg8"
Oct 30 13:23:44.709270 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:44.709061    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80fbd9-6ncg8" event={"ID":"710a5656-e79a-4c00-8df1-99bcbd92e400","Type":"ContainerDied","Data":"1828721e437f703a9b9b0b56dd6b31aaadc5e6addcad00f2a21475a2ca736945"}
Oct 30 13:23:44.709270 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:44.709138    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1828721e437f703a9b9b0b56dd6b31aaadc5e6addcad00f2a21475a2ca736945"
Oct 30 13:23:45.132941 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:45.132891    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:47.100079 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:47.100042    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:48.695811 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:48.695774    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-547c0f1d-mq9j4"]
Oct 30 13:23:48.696601 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:48.696584    4544 cpu_manager.go:414] "containerMap: removing container" podUID="710a5656-e79a-4c00-8df1-99bcbd92e400" containerName="revoke-keys"
Oct 30 13:23:48.696675 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:48.696602    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="710a5656-e79a-4c00-8df1-99bcbd92e400" containerName="revoke-keys"
Oct 30 13:23:48.696776 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:48.696765    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="710a5656-e79a-4c00-8df1-99bcbd92e400" containerName="revoke-keys"
Oct 30 13:23:48.701819 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:48.701799    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-547c0f1d-mq9j4"
Oct 30 13:23:48.719572 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:48.719508    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:23:48.726232 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:48.726193    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-547c0f1d-mq9j4"]
Oct 30 13:23:48.731035 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:48.731014    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:23:48.776652 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:48.776618    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8403cb99-bff9-4129-b35b-ff6fd0af6f19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-547c0f1d-mq9j4\" (UID: \"8403cb99-bff9-4129-b35b-ff6fd0af6f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-547c0f1d-mq9j4"
Oct 30 13:23:48.776849 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:48.776811    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8403cb99-bff9-4129-b35b-ff6fd0af6f19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-547c0f1d-mq9j4\" (UID: \"8403cb99-bff9-4129-b35b-ff6fd0af6f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-547c0f1d-mq9j4"
Oct 30 13:23:48.776894 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:48.776856    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrssj\" (UniqueName: \"kubernetes.io/projected/8403cb99-bff9-4129-b35b-ff6fd0af6f19-kube-api-access-xrssj\") pod \"maas-api-revoke-sub-547c0f1d-mq9j4\" (UID: \"8403cb99-bff9-4129-b35b-ff6fd0af6f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-547c0f1d-mq9j4"
Oct 30 13:23:48.879508 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:48.879462    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8403cb99-bff9-4129-b35b-ff6fd0af6f19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-547c0f1d-mq9j4\" (UID: \"8403cb99-bff9-4129-b35b-ff6fd0af6f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-547c0f1d-mq9j4"
Oct 30 13:23:48.879708 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:48.879569    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrssj\" (UniqueName: \"kubernetes.io/projected/8403cb99-bff9-4129-b35b-ff6fd0af6f19-kube-api-access-xrssj\") pod \"maas-api-revoke-sub-547c0f1d-mq9j4\" (UID: \"8403cb99-bff9-4129-b35b-ff6fd0af6f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-547c0f1d-mq9j4"
Oct 30 13:23:48.879778 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:48.879746    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8403cb99-bff9-4129-b35b-ff6fd0af6f19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-547c0f1d-mq9j4\" (UID: \"8403cb99-bff9-4129-b35b-ff6fd0af6f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-547c0f1d-mq9j4"
Oct 30 13:23:48.880679 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:48.880650    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8403cb99-bff9-4129-b35b-ff6fd0af6f19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-547c0f1d-mq9j4\" (UID: \"8403cb99-bff9-4129-b35b-ff6fd0af6f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-547c0f1d-mq9j4"
Oct 30 13:23:48.883007 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:48.882978    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8403cb99-bff9-4129-b35b-ff6fd0af6f19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-547c0f1d-mq9j4\" (UID: \"8403cb99-bff9-4129-b35b-ff6fd0af6f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-547c0f1d-mq9j4"
Oct 30 13:23:48.904150 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:48.904124    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrssj\" (UniqueName: \"kubernetes.io/projected/8403cb99-bff9-4129-b35b-ff6fd0af6f19-kube-api-access-xrssj\") pod \"maas-api-revoke-sub-547c0f1d-mq9j4\" (UID: \"8403cb99-bff9-4129-b35b-ff6fd0af6f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-547c0f1d-mq9j4"
Oct 30 13:23:48.927210 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:48.927170    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:49.016814 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:49.016723    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-547c0f1d-mq9j4"
Oct 30 13:23:49.342142 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:49.342112    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-547c0f1d-mq9j4"]
Oct 30 13:23:49.343012 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:23:49.342974    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8403cb99_bff9_4129_b35b_ff6fd0af6f19.slice/crio-a01f0ec063ea770bdd0640eee91352420ccc226bd7e699114c550cdc9e922b92 WatchSource:0}: Error finding container a01f0ec063ea770bdd0640eee91352420ccc226bd7e699114c550cdc9e922b92: Status 404 returned error can't find the container with id a01f0ec063ea770bdd0640eee91352420ccc226bd7e699114c550cdc9e922b92
Oct 30 13:23:49.756732 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:49.756694    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-547c0f1d-mq9j4" event={"ID":"8403cb99-bff9-4129-b35b-ff6fd0af6f19","Type":"ContainerStarted","Data":"38dc114eef1c13fe6d0baa6f99d04271bf3000ef3169e19e318403c616a46364"}
Oct 30 13:23:49.757177 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:49.756754    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-547c0f1d-mq9j4" event={"ID":"8403cb99-bff9-4129-b35b-ff6fd0af6f19","Type":"ContainerStarted","Data":"a01f0ec063ea770bdd0640eee91352420ccc226bd7e699114c550cdc9e922b92"}
Oct 30 13:23:49.808463 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:49.808411    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-547c0f1d-mq9j4" podStartSLOduration=1.8083963189999999 podStartE2EDuration="1.808396319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:49.803712595 +0000 UTC m=+2324.850846548" watchObservedRunningTime="2026-10-30 13:23:49.808396319 +0000 UTC m=+2324.855530271"
Oct 30 13:23:50.766723 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:50.766686    4544 generic.go:356] "Generic (PLEG): container finished" podID="8403cb99-bff9-4129-b35b-ff6fd0af6f19" containerID="38dc114eef1c13fe6d0baa6f99d04271bf3000ef3169e19e318403c616a46364" exitCode=0
Oct 30 13:23:50.767109 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:50.766740    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-547c0f1d-mq9j4" event={"ID":"8403cb99-bff9-4129-b35b-ff6fd0af6f19","Type":"ContainerDied","Data":"38dc114eef1c13fe6d0baa6f99d04271bf3000ef3169e19e318403c616a46364"}
Oct 30 13:23:50.825253 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:50.825162    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:51.065230 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:51.065137    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4"]
Oct 30 13:23:51.101331 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:51.101293    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4"]
Oct 30 13:23:51.260538 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:51.260511    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:51.464856 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:51.464812    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:52.137028 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:52.136993    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-547c0f1d-mq9j4"
Oct 30 13:23:52.283502 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:52.283461    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8403cb99-bff9-4129-b35b-ff6fd0af6f19-kube-api-access-xrssj\" (UniqueName: \"kubernetes.io/projected/8403cb99-bff9-4129-b35b-ff6fd0af6f19-kube-api-access-xrssj\") pod \"8403cb99-bff9-4129-b35b-ff6fd0af6f19\" (UID: \"8403cb99-bff9-4129-b35b-ff6fd0af6f19\") "
Oct 30 13:23:52.283674 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:52.283614    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8403cb99-bff9-4129-b35b-ff6fd0af6f19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8403cb99-bff9-4129-b35b-ff6fd0af6f19-maas-api-openshift-ca\") pod \"8403cb99-bff9-4129-b35b-ff6fd0af6f19\" (UID: \"8403cb99-bff9-4129-b35b-ff6fd0af6f19\") "
Oct 30 13:23:52.283765 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:52.283677    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8403cb99-bff9-4129-b35b-ff6fd0af6f19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8403cb99-bff9-4129-b35b-ff6fd0af6f19-maas-api-xks-ca\") pod \"8403cb99-bff9-4129-b35b-ff6fd0af6f19\" (UID: \"8403cb99-bff9-4129-b35b-ff6fd0af6f19\") "
Oct 30 13:23:52.284004 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:52.283979    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8403cb99-bff9-4129-b35b-ff6fd0af6f19-maas-api-openshift-ca" pod "8403cb99-bff9-4129-b35b-ff6fd0af6f19" (UID: "8403cb99-bff9-4129-b35b-ff6fd0af6f19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:23:52.284381 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:52.284359    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8403cb99-bff9-4129-b35b-ff6fd0af6f19-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:23:52.285686 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:52.285659    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8403cb99-bff9-4129-b35b-ff6fd0af6f19-maas-api-xks-ca" pod "8403cb99-bff9-4129-b35b-ff6fd0af6f19" (UID: "8403cb99-bff9-4129-b35b-ff6fd0af6f19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:23:52.285802 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:52.285693    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8403cb99-bff9-4129-b35b-ff6fd0af6f19-kube-api-access-xrssj" pod "8403cb99-bff9-4129-b35b-ff6fd0af6f19" (UID: "8403cb99-bff9-4129-b35b-ff6fd0af6f19"). InnerVolumeSpecName "kube-api-access-xrssj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:23:52.385560 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:52.385483    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8403cb99-bff9-4129-b35b-ff6fd0af6f19-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:23:52.385560 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:52.385511    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrssj\" (UniqueName: \"kubernetes.io/projected/8403cb99-bff9-4129-b35b-ff6fd0af6f19-kube-api-access-xrssj\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:23:52.783712 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:52.783615    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:52.870045 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:52.869945    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-547c0f1d-mq9j4" event={"ID":"8403cb99-bff9-4129-b35b-ff6fd0af6f19","Type":"ContainerDied","Data":"a01f0ec063ea770bdd0640eee91352420ccc226bd7e699114c550cdc9e922b92"}
Oct 30 13:23:52.870045 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:52.870011    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a01f0ec063ea770bdd0640eee91352420ccc226bd7e699114c550cdc9e922b92"
Oct 30 13:23:52.870403 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:52.870382    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-547c0f1d-mq9j4"
Oct 30 13:23:54.674698 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:54.674665    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:55.365985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:55.365937    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz" podUID="41182c29-4530-4a65-87b6-781a2fd06ff8" containerName="main" containerID="cri-o://1c266916894d619ed4527ccfc5ac608560bfc85c40984188e1657620c40f6fe3" gracePeriod=45
Oct 30 13:23:55.385251 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:55.385078    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz" podUID="41182c29-4530-4a65-87b6-781a2fd06ff8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.216:8000/ready\": dial tcp 10.133.0.216:8000: connect: connection refused"
Oct 30 13:23:55.648321 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:55.648294    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz"
Oct 30 13:23:55.740725 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:55.740696    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41182c29-4530-4a65-87b6-781a2fd06ff8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41182c29-4530-4a65-87b6-781a2fd06ff8-tmp-dir\") pod \"41182c29-4530-4a65-87b6-781a2fd06ff8\" (UID: \"41182c29-4530-4a65-87b6-781a2fd06ff8\") "
Oct 30 13:23:55.741112 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:55.740783    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41182c29-4530-4a65-87b6-781a2fd06ff8-tls-certs\" (UniqueName: \"kubernetes.io/secret/41182c29-4530-4a65-87b6-781a2fd06ff8-tls-certs\") pod \"41182c29-4530-4a65-87b6-781a2fd06ff8\" (UID: \"41182c29-4530-4a65-87b6-781a2fd06ff8\") "
Oct 30 13:23:55.741112 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:55.740883    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41182c29-4530-4a65-87b6-781a2fd06ff8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41182c29-4530-4a65-87b6-781a2fd06ff8-model-cache\") pod \"41182c29-4530-4a65-87b6-781a2fd06ff8\" (UID: \"41182c29-4530-4a65-87b6-781a2fd06ff8\") "
Oct 30 13:23:55.741112 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:55.740956    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41182c29-4530-4a65-87b6-781a2fd06ff8-kube-api-access-swjj6\" (UniqueName: \"kubernetes.io/projected/41182c29-4530-4a65-87b6-781a2fd06ff8-kube-api-access-swjj6\") pod \"41182c29-4530-4a65-87b6-781a2fd06ff8\" (UID: \"41182c29-4530-4a65-87b6-781a2fd06ff8\") "
Oct 30 13:23:55.741112 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:55.740973    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41182c29-4530-4a65-87b6-781a2fd06ff8-tmp-dir" pod "41182c29-4530-4a65-87b6-781a2fd06ff8" (UID: "41182c29-4530-4a65-87b6-781a2fd06ff8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:23:55.741112 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:55.741023    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41182c29-4530-4a65-87b6-781a2fd06ff8-home\" (UniqueName: \"kubernetes.io/empty-dir/41182c29-4530-4a65-87b6-781a2fd06ff8-home\") pod \"41182c29-4530-4a65-87b6-781a2fd06ff8\" (UID: \"41182c29-4530-4a65-87b6-781a2fd06ff8\") "
Oct 30 13:23:55.741112 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:55.741097    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41182c29-4530-4a65-87b6-781a2fd06ff8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/41182c29-4530-4a65-87b6-781a2fd06ff8-dshm\") pod \"41182c29-4530-4a65-87b6-781a2fd06ff8\" (UID: \"41182c29-4530-4a65-87b6-781a2fd06ff8\") "
Oct 30 13:23:55.741391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:55.741164    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41182c29-4530-4a65-87b6-781a2fd06ff8-model-cache" pod "41182c29-4530-4a65-87b6-781a2fd06ff8" (UID: "41182c29-4530-4a65-87b6-781a2fd06ff8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:23:55.741391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:55.741324    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41182c29-4530-4a65-87b6-781a2fd06ff8-home" pod "41182c29-4530-4a65-87b6-781a2fd06ff8" (UID: "41182c29-4530-4a65-87b6-781a2fd06ff8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:23:55.741945 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:55.741921    4544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41182c29-4530-4a65-87b6-781a2fd06ff8-home\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:23:55.742044 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:55.741950    4544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41182c29-4530-4a65-87b6-781a2fd06ff8-tmp-dir\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:23:55.742044 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:55.741967    4544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41182c29-4530-4a65-87b6-781a2fd06ff8-model-cache\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:23:55.743330 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:55.743304    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41182c29-4530-4a65-87b6-781a2fd06ff8-tls-certs" pod "41182c29-4530-4a65-87b6-781a2fd06ff8" (UID: "41182c29-4530-4a65-87b6-781a2fd06ff8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:23:55.743468 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:55.743420    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41182c29-4530-4a65-87b6-781a2fd06ff8-dshm" pod "41182c29-4530-4a65-87b6-781a2fd06ff8" (UID: "41182c29-4530-4a65-87b6-781a2fd06ff8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:23:55.746504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:55.746486    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41182c29-4530-4a65-87b6-781a2fd06ff8-kube-api-access-swjj6" pod "41182c29-4530-4a65-87b6-781a2fd06ff8" (UID: "41182c29-4530-4a65-87b6-781a2fd06ff8"). InnerVolumeSpecName "kube-api-access-swjj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:23:55.843308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:55.843271    4544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41182c29-4530-4a65-87b6-781a2fd06ff8-tls-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:23:55.843308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:55.843303    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swjj6\" (UniqueName: \"kubernetes.io/projected/41182c29-4530-4a65-87b6-781a2fd06ff8-kube-api-access-swjj6\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:23:55.843308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:55.843314    4544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41182c29-4530-4a65-87b6-781a2fd06ff8-dshm\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:23:55.899672 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:55.899588    4544 generic.go:356] "Generic (PLEG): container finished" podID="41182c29-4530-4a65-87b6-781a2fd06ff8" containerID="1c266916894d619ed4527ccfc5ac608560bfc85c40984188e1657620c40f6fe3" exitCode=0
Oct 30 13:23:55.899672 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:55.899633    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz"
Oct 30 13:23:55.899672 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:55.899652    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz" event={"ID":"41182c29-4530-4a65-87b6-781a2fd06ff8","Type":"ContainerDied","Data":"1c266916894d619ed4527ccfc5ac608560bfc85c40984188e1657620c40f6fe3"}
Oct 30 13:23:55.899963 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:55.899706    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz" event={"ID":"41182c29-4530-4a65-87b6-781a2fd06ff8","Type":"ContainerDied","Data":"cb6eb8f8f02e7903f6f951f4f95c2d1f28d21f6e0e7dcdfd48d775b35c771543"}
Oct 30 13:23:55.899963 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:55.899729    4544 scope.go:122] "RemoveContainer" containerID="1c266916894d619ed4527ccfc5ac608560bfc85c40984188e1657620c40f6fe3"
Oct 30 13:23:55.912961 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:55.912937    4544 scope.go:122] "RemoveContainer" containerID="1c266916894d619ed4527ccfc5ac608560bfc85c40984188e1657620c40f6fe3"
Oct 30 13:23:55.913306 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:23:55.913277    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c266916894d619ed4527ccfc5ac608560bfc85c40984188e1657620c40f6fe3\": container with ID starting with 1c266916894d619ed4527ccfc5ac608560bfc85c40984188e1657620c40f6fe3 not found: ID does not exist" containerID="1c266916894d619ed4527ccfc5ac608560bfc85c40984188e1657620c40f6fe3"
Oct 30 13:23:55.913427 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:55.913313    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c266916894d619ed4527ccfc5ac608560bfc85c40984188e1657620c40f6fe3"} err="failed to get container status \"1c266916894d619ed4527ccfc5ac608560bfc85c40984188e1657620c40f6fe3\": rpc error: code = NotFound desc = could not find container \"1c266916894d619ed4527ccfc5ac608560bfc85c40984188e1657620c40f6fe3\": container with ID starting with 1c266916894d619ed4527ccfc5ac608560bfc85c40984188e1657620c40f6fe3 not found: ID does not exist"
Oct 30 13:23:55.953889 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:55.953857    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz"]
Oct 30 13:23:55.958153 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:55.958122    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-bec983/test-model-bec983-kserve-564766445f-5xbrz"]
Oct 30 13:23:56.066209 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:56.066160    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4" podUID="382ed63d-b520-4d49-ba0b-5fc982f309df" containerName="payload-processing" containerID="cri-o://f5cc933dd0777184a3067958f2af3c778e158ded5c5e7cfaa03a2b9c3363fd26" gracePeriod=25
Oct 30 13:23:56.102186 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:56.102144    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4" podUID="6abef82a-570c-494b-afe4-83bf81a4330c" containerName="payload-pre-processing" containerID="cri-o://480a0800895fd605124602bced6e4eaca04b4fb29d86ee75a55468e9d0b2004f" gracePeriod=25
Oct 30 13:23:56.321511 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:56.321397    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:56.443126 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:56.443094    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:56.848783 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:56.848743    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:57.103691 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:57.103609    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41182c29-4530-4a65-87b6-781a2fd06ff8" path="/var/lib/kubelet/pods/41182c29-4530-4a65-87b6-781a2fd06ff8/volumes"
Oct 30 13:23:58.278015 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:23:58.277975    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:23:59.009127 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:23:59.009086    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:24:00.081845 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:00.081802    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:01.651464 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:01.651414    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:02.330065 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:02.330021    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:02.573422 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:02.573384    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:02.651273 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:02.650299    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cd2d2e-jk6vl"]
Oct 30 13:24:02.656798 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:02.656763    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cd2d2e-jk6vl"]
Oct 30 13:24:02.706419 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:02.706385    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:03.091308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:03.091275    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38b8eead-3d07-4df3-a422-49c83e202231" path="/var/lib/kubelet/pods/38b8eead-3d07-4df3-a422-49c83e202231/volumes"
Oct 30 13:24:03.149365 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:03.149321    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:04.107773 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:04.107734    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4" podUID="6abef82a-570c-494b-afe4-83bf81a4330c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:24:04.108177 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:04.107737    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4" podUID="382ed63d-b520-4d49-ba0b-5fc982f309df" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:24:04.815620 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:04.815589    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:05.838532 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:05.838495    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:06.344910 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:06.344852    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e318c72d-htcc6"]
Oct 30 13:24:06.345822 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:06.345798    4544 cpu_manager.go:414] "containerMap: removing container" podUID="41182c29-4530-4a65-87b6-781a2fd06ff8" containerName="main"
Oct 30 13:24:06.345822 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:06.345819    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41182c29-4530-4a65-87b6-781a2fd06ff8" containerName="main"
Oct 30 13:24:06.346020 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:06.345855    4544 cpu_manager.go:414] "containerMap: removing container" podUID="8403cb99-bff9-4129-b35b-ff6fd0af6f19" containerName="revoke-keys"
Oct 30 13:24:06.346020 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:06.345863    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8403cb99-bff9-4129-b35b-ff6fd0af6f19" containerName="revoke-keys"
Oct 30 13:24:06.346020 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:06.345996    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="41182c29-4530-4a65-87b6-781a2fd06ff8" containerName="main"
Oct 30 13:24:06.346020 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:06.346010    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="8403cb99-bff9-4129-b35b-ff6fd0af6f19" containerName="revoke-keys"
Oct 30 13:24:06.350331 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:06.350302    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e318c72d-htcc6"
Oct 30 13:24:06.372564 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:06.372519    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:24:06.391247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:06.390390    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:24:06.401774 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:06.401267    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e318c72d-htcc6"]
Oct 30 13:24:06.466780 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:06.466744    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b60838-fbc5-41fe-be77-97c4101e6109-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e318c72d-htcc6\" (UID: \"95b60838-fbc5-41fe-be77-97c4101e6109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e318c72d-htcc6"
Oct 30 13:24:06.466972 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:06.466815    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b60838-fbc5-41fe-be77-97c4101e6109-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e318c72d-htcc6\" (UID: \"95b60838-fbc5-41fe-be77-97c4101e6109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e318c72d-htcc6"
Oct 30 13:24:06.467068 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:06.467045    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn5z8\" (UniqueName: \"kubernetes.io/projected/95b60838-fbc5-41fe-be77-97c4101e6109-kube-api-access-kn5z8\") pod \"maas-api-revoke-sub-e318c72d-htcc6\" (UID: \"95b60838-fbc5-41fe-be77-97c4101e6109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e318c72d-htcc6"
Oct 30 13:24:06.568508 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:06.568454    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn5z8\" (UniqueName: \"kubernetes.io/projected/95b60838-fbc5-41fe-be77-97c4101e6109-kube-api-access-kn5z8\") pod \"maas-api-revoke-sub-e318c72d-htcc6\" (UID: \"95b60838-fbc5-41fe-be77-97c4101e6109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e318c72d-htcc6"
Oct 30 13:24:06.568706 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:06.568587    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b60838-fbc5-41fe-be77-97c4101e6109-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e318c72d-htcc6\" (UID: \"95b60838-fbc5-41fe-be77-97c4101e6109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e318c72d-htcc6"
Oct 30 13:24:06.568706 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:06.568640    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b60838-fbc5-41fe-be77-97c4101e6109-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e318c72d-htcc6\" (UID: \"95b60838-fbc5-41fe-be77-97c4101e6109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e318c72d-htcc6"
Oct 30 13:24:06.569448 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:06.569421    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b60838-fbc5-41fe-be77-97c4101e6109-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e318c72d-htcc6\" (UID: \"95b60838-fbc5-41fe-be77-97c4101e6109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e318c72d-htcc6"
Oct 30 13:24:06.571328 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:06.571301    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b60838-fbc5-41fe-be77-97c4101e6109-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e318c72d-htcc6\" (UID: \"95b60838-fbc5-41fe-be77-97c4101e6109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e318c72d-htcc6"
Oct 30 13:24:06.593618 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:06.593584    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn5z8\" (UniqueName: \"kubernetes.io/projected/95b60838-fbc5-41fe-be77-97c4101e6109-kube-api-access-kn5z8\") pod \"maas-api-revoke-sub-e318c72d-htcc6\" (UID: \"95b60838-fbc5-41fe-be77-97c4101e6109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e318c72d-htcc6"
Oct 30 13:24:06.676211 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:06.676122    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e318c72d-htcc6"
Oct 30 13:24:06.816272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:06.815870    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:06.999148 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:06.999111    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e318c72d-htcc6"]
Oct 30 13:24:06.999900 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:24:06.999865    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b60838_fbc5_41fe_be77_97c4101e6109.slice/crio-a42203b9130d0b8badc25388350ddf1ca11513db905653a5bf658ae6c4184f71 WatchSource:0}: Error finding container a42203b9130d0b8badc25388350ddf1ca11513db905653a5bf658ae6c4184f71: Status 404 returned error can't find the container with id a42203b9130d0b8badc25388350ddf1ca11513db905653a5bf658ae6c4184f71
Oct 30 13:24:07.018621 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:07.018581    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e318c72d-htcc6" event={"ID":"95b60838-fbc5-41fe-be77-97c4101e6109","Type":"ContainerStarted","Data":"a42203b9130d0b8badc25388350ddf1ca11513db905653a5bf658ae6c4184f71"}
Oct 30 13:24:08.034424 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:08.034388    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e318c72d-htcc6" event={"ID":"95b60838-fbc5-41fe-be77-97c4101e6109","Type":"ContainerStarted","Data":"d4f279d2fe1fb18ab4103671744e9b4b7cb1133d4a0c65811e6b4f486fff38d1"}
Oct 30 13:24:08.586533 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:08.586491    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:09.044884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:09.044794    4544 generic.go:356] "Generic (PLEG): container finished" podID="95b60838-fbc5-41fe-be77-97c4101e6109" containerID="d4f279d2fe1fb18ab4103671744e9b4b7cb1133d4a0c65811e6b4f486fff38d1" exitCode=0
Oct 30 13:24:09.045321 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:09.044870    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e318c72d-htcc6" event={"ID":"95b60838-fbc5-41fe-be77-97c4101e6109","Type":"ContainerDied","Data":"d4f279d2fe1fb18ab4103671744e9b4b7cb1133d4a0c65811e6b4f486fff38d1"}
Oct 30 13:24:09.218868 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:09.218835    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e318c72d-htcc6"
Oct 30 13:24:09.323239 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:09.323199    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95b60838-fbc5-41fe-be77-97c4101e6109-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b60838-fbc5-41fe-be77-97c4101e6109-maas-api-openshift-ca\") pod \"95b60838-fbc5-41fe-be77-97c4101e6109\" (UID: \"95b60838-fbc5-41fe-be77-97c4101e6109\") "
Oct 30 13:24:09.323473 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:09.323450    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95b60838-fbc5-41fe-be77-97c4101e6109-kube-api-access-kn5z8\" (UniqueName: \"kubernetes.io/projected/95b60838-fbc5-41fe-be77-97c4101e6109-kube-api-access-kn5z8\") pod \"95b60838-fbc5-41fe-be77-97c4101e6109\" (UID: \"95b60838-fbc5-41fe-be77-97c4101e6109\") "
Oct 30 13:24:09.323618 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:09.323594    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95b60838-fbc5-41fe-be77-97c4101e6109-maas-api-openshift-ca" pod "95b60838-fbc5-41fe-be77-97c4101e6109" (UID: "95b60838-fbc5-41fe-be77-97c4101e6109"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:09.323685 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:09.323622    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95b60838-fbc5-41fe-be77-97c4101e6109-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b60838-fbc5-41fe-be77-97c4101e6109-maas-api-xks-ca\") pod \"95b60838-fbc5-41fe-be77-97c4101e6109\" (UID: \"95b60838-fbc5-41fe-be77-97c4101e6109\") "
Oct 30 13:24:09.324168 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:09.324146    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b60838-fbc5-41fe-be77-97c4101e6109-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:09.325782 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:09.325763    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95b60838-fbc5-41fe-be77-97c4101e6109-maas-api-xks-ca" pod "95b60838-fbc5-41fe-be77-97c4101e6109" (UID: "95b60838-fbc5-41fe-be77-97c4101e6109"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:24:09.325852 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:09.325803    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95b60838-fbc5-41fe-be77-97c4101e6109-kube-api-access-kn5z8" pod "95b60838-fbc5-41fe-be77-97c4101e6109" (UID: "95b60838-fbc5-41fe-be77-97c4101e6109"). InnerVolumeSpecName "kube-api-access-kn5z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:09.426085 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:09.426048    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn5z8\" (UniqueName: \"kubernetes.io/projected/95b60838-fbc5-41fe-be77-97c4101e6109-kube-api-access-kn5z8\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:09.426085 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:09.426082    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b60838-fbc5-41fe-be77-97c4101e6109-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:10.055741 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:10.055708    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e318c72d-htcc6" event={"ID":"95b60838-fbc5-41fe-be77-97c4101e6109","Type":"ContainerDied","Data":"a42203b9130d0b8badc25388350ddf1ca11513db905653a5bf658ae6c4184f71"}
Oct 30 13:24:10.055741 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:10.055730    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e318c72d-htcc6"
Oct 30 13:24:10.056199 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:10.055792    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a42203b9130d0b8badc25388350ddf1ca11513db905653a5bf658ae6c4184f71"
Oct 30 13:24:10.193789 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:10.193748    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:10.366785 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:10.366746    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:10.570237 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:10.570187    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:10.730920 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:10.730771    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:11.523169 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:11.523130    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:12.065036 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:12.064977    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:12.572326 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:12.572270    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:12.966260 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:12.964962    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:14.128198 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:14.128155    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4" podUID="382ed63d-b520-4d49-ba0b-5fc982f309df" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:24:14.129683 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:14.129648    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4" podUID="6abef82a-570c-494b-afe4-83bf81a4330c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:24:14.274316 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:14.274277    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"]
Oct 30 13:24:14.274600 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:14.274571    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz" podUID="3381ce8c-5918-4339-aa75-dd37e7723ee8" containerName="istio-proxy" containerID="cri-o://1f2ab98f4f237119db700d8fde4d491f25a005ece9fc417356b7d007cb2473ba" gracePeriod=30
Oct 30 13:24:15.278778 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:15.278737    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:16.257071 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:16.257033    4544 generic.go:356] "Generic (PLEG): container finished" podID="382ed63d-b520-4d49-ba0b-5fc982f309df" containerID="f5cc933dd0777184a3067958f2af3c778e158ded5c5e7cfaa03a2b9c3363fd26" exitCode=0
Oct 30 13:24:16.257325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:16.257292    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4" event={"ID":"382ed63d-b520-4d49-ba0b-5fc982f309df","Type":"ContainerDied","Data":"f5cc933dd0777184a3067958f2af3c778e158ded5c5e7cfaa03a2b9c3363fd26"}
Oct 30 13:24:16.259625 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:16.259594    4544 generic.go:356] "Generic (PLEG): container finished" podID="6abef82a-570c-494b-afe4-83bf81a4330c" containerID="480a0800895fd605124602bced6e4eaca04b4fb29d86ee75a55468e9d0b2004f" exitCode=0
Oct 30 13:24:16.259768 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:16.259667    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4" event={"ID":"6abef82a-570c-494b-afe4-83bf81a4330c","Type":"ContainerDied","Data":"480a0800895fd605124602bced6e4eaca04b4fb29d86ee75a55468e9d0b2004f"}
Oct 30 13:24:16.332602 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:16.332569    4544 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:24:16.333067 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:16.332649    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz" podUID="3381ce8c-5918-4339-aa75-dd37e7723ee8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:24:16.411738 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:16.411708    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4"
Oct 30 13:24:16.447045 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:16.447017    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4"
Oct 30 13:24:16.485468 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:16.485438    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/382ed63d-b520-4d49-ba0b-5fc982f309df-kube-api-access-2gg9b\" (UniqueName: \"kubernetes.io/projected/382ed63d-b520-4d49-ba0b-5fc982f309df-kube-api-access-2gg9b\") pod \"382ed63d-b520-4d49-ba0b-5fc982f309df\" (UID: \"382ed63d-b520-4d49-ba0b-5fc982f309df\") "
Oct 30 13:24:16.485662 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:16.485642    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/382ed63d-b520-4d49-ba0b-5fc982f309df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/382ed63d-b520-4d49-ba0b-5fc982f309df-plugins-config-volume\") pod \"382ed63d-b520-4d49-ba0b-5fc982f309df\" (UID: \"382ed63d-b520-4d49-ba0b-5fc982f309df\") "
Oct 30 13:24:16.486153 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:16.486115    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/382ed63d-b520-4d49-ba0b-5fc982f309df-plugins-config-volume" pod "382ed63d-b520-4d49-ba0b-5fc982f309df" (UID: "382ed63d-b520-4d49-ba0b-5fc982f309df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:16.486703 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:16.486679    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/382ed63d-b520-4d49-ba0b-5fc982f309df-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:16.487888 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:16.487861    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/382ed63d-b520-4d49-ba0b-5fc982f309df-kube-api-access-2gg9b" pod "382ed63d-b520-4d49-ba0b-5fc982f309df" (UID: "382ed63d-b520-4d49-ba0b-5fc982f309df"). InnerVolumeSpecName "kube-api-access-2gg9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:16.588031 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:16.587934    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6abef82a-570c-494b-afe4-83bf81a4330c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6abef82a-570c-494b-afe4-83bf81a4330c-plugins-config-volume\") pod \"6abef82a-570c-494b-afe4-83bf81a4330c\" (UID: \"6abef82a-570c-494b-afe4-83bf81a4330c\") "
Oct 30 13:24:16.588242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:16.588075    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6abef82a-570c-494b-afe4-83bf81a4330c-kube-api-access-swq8w\" (UniqueName: \"kubernetes.io/projected/6abef82a-570c-494b-afe4-83bf81a4330c-kube-api-access-swq8w\") pod \"6abef82a-570c-494b-afe4-83bf81a4330c\" (UID: \"6abef82a-570c-494b-afe4-83bf81a4330c\") "
Oct 30 13:24:16.588434 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:16.588406    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6abef82a-570c-494b-afe4-83bf81a4330c-plugins-config-volume" pod "6abef82a-570c-494b-afe4-83bf81a4330c" (UID: "6abef82a-570c-494b-afe4-83bf81a4330c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:16.588703 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:16.588685    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gg9b\" (UniqueName: \"kubernetes.io/projected/382ed63d-b520-4d49-ba0b-5fc982f309df-kube-api-access-2gg9b\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:16.588703 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:16.588708    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6abef82a-570c-494b-afe4-83bf81a4330c-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:16.590594 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:16.590565    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6abef82a-570c-494b-afe4-83bf81a4330c-kube-api-access-swq8w" pod "6abef82a-570c-494b-afe4-83bf81a4330c" (UID: "6abef82a-570c-494b-afe4-83bf81a4330c"). InnerVolumeSpecName "kube-api-access-swq8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:16.690884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:16.690846    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swq8w\" (UniqueName: \"kubernetes.io/projected/6abef82a-570c-494b-afe4-83bf81a4330c-kube-api-access-swq8w\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:16.692942 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:16.692910    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:16.893585 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:16.893486    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:17.269729 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:17.269651    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4"
Oct 30 13:24:17.269729 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:17.269688    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4" event={"ID":"6abef82a-570c-494b-afe4-83bf81a4330c","Type":"ContainerDied","Data":"e7208a35170642ec818a681096b28ea065210cb8241c69cb575274aa1ed836b6"}
Oct 30 13:24:17.269957 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:17.269756    4544 scope.go:122] "RemoveContainer" containerID="480a0800895fd605124602bced6e4eaca04b4fb29d86ee75a55468e9d0b2004f"
Oct 30 13:24:17.271538 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:17.271513    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4" event={"ID":"382ed63d-b520-4d49-ba0b-5fc982f309df","Type":"ContainerDied","Data":"072273e70a0a2732ed43b1975f98f02fe0ce84590faf1db3ae7afb6384816cdf"}
Oct 30 13:24:17.271538 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:17.271534    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4"
Oct 30 13:24:17.281207 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:17.281186    4544 scope.go:122] "RemoveContainer" containerID="f5cc933dd0777184a3067958f2af3c778e158ded5c5e7cfaa03a2b9c3363fd26"
Oct 30 13:24:17.331335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:17.331306    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4"]
Oct 30 13:24:17.343237 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:17.343189    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4"]
Oct 30 13:24:17.392499 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:17.392466    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4"]
Oct 30 13:24:17.426239 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:17.407972    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4"]
Oct 30 13:24:17.653610 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:17.653576    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:18.395158 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:18.395112    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:18.505024 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:18.504976    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:18.561788 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:18.561754    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6"]
Oct 30 13:24:18.562567 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:18.562551    4544 cpu_manager.go:414] "containerMap: removing container" podUID="95b60838-fbc5-41fe-be77-97c4101e6109" containerName="revoke-keys"
Oct 30 13:24:18.562628 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:18.562568    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95b60838-fbc5-41fe-be77-97c4101e6109" containerName="revoke-keys"
Oct 30 13:24:18.562628 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:18.562597    4544 cpu_manager.go:414] "containerMap: removing container" podUID="382ed63d-b520-4d49-ba0b-5fc982f309df" containerName="payload-processing"
Oct 30 13:24:18.562628 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:18.562602    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="382ed63d-b520-4d49-ba0b-5fc982f309df" containerName="payload-processing"
Oct 30 13:24:18.562734 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:18.562630    4544 cpu_manager.go:414] "containerMap: removing container" podUID="6abef82a-570c-494b-afe4-83bf81a4330c" containerName="payload-pre-processing"
Oct 30 13:24:18.562734 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:18.562636    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6abef82a-570c-494b-afe4-83bf81a4330c" containerName="payload-pre-processing"
Oct 30 13:24:18.562806 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:18.562752    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="95b60838-fbc5-41fe-be77-97c4101e6109" containerName="revoke-keys"
Oct 30 13:24:18.562806 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:18.562763    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="382ed63d-b520-4d49-ba0b-5fc982f309df" containerName="payload-processing"
Oct 30 13:24:18.562806 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:18.562775    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="6abef82a-570c-494b-afe4-83bf81a4330c" containerName="payload-pre-processing"
Oct 30 13:24:18.568274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:18.568247    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6"
Oct 30 13:24:18.588720 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:18.588691    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6"]
Oct 30 13:24:18.594848 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:18.594803    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-51288b-dockercfg-zvbph\""
Oct 30 13:24:18.595025 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:18.595001    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-51288b\""
Oct 30 13:24:18.728293 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:18.723383    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/480fecc2-7659-45df-b613-3e0f2b0f7fff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6\" (UID: \"480fecc2-7659-45df-b613-3e0f2b0f7fff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6"
Oct 30 13:24:18.728293 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:18.723491    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcr29\" (UniqueName: \"kubernetes.io/projected/480fecc2-7659-45df-b613-3e0f2b0f7fff-kube-api-access-pcr29\") pod \"payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6\" (UID: \"480fecc2-7659-45df-b613-3e0f2b0f7fff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6"
Oct 30 13:24:18.827242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:18.826729    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77"]
Oct 30 13:24:18.839259 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:18.836097    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77"]
Oct 30 13:24:18.839259 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:18.836191    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77"
Oct 30 13:24:18.852497 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:18.848528    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/480fecc2-7659-45df-b613-3e0f2b0f7fff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6\" (UID: \"480fecc2-7659-45df-b613-3e0f2b0f7fff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6"
Oct 30 13:24:18.852497 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:18.848618    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcr29\" (UniqueName: \"kubernetes.io/projected/480fecc2-7659-45df-b613-3e0f2b0f7fff-kube-api-access-pcr29\") pod \"payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6\" (UID: \"480fecc2-7659-45df-b613-3e0f2b0f7fff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6"
Oct 30 13:24:18.852497 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:18.849853    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/480fecc2-7659-45df-b613-3e0f2b0f7fff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6\" (UID: \"480fecc2-7659-45df-b613-3e0f2b0f7fff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6"
Oct 30 13:24:18.882200 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:18.873821    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcr29\" (UniqueName: \"kubernetes.io/projected/480fecc2-7659-45df-b613-3e0f2b0f7fff-kube-api-access-pcr29\") pod \"payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6\" (UID: \"480fecc2-7659-45df-b613-3e0f2b0f7fff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6"
Oct 30 13:24:18.884995 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:18.884941    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6"
Oct 30 13:24:18.950101 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:18.950071    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9mbb\" (UniqueName: \"kubernetes.io/projected/95241140-c937-47c1-9413-8bc8a68c840c-kube-api-access-l9mbb\") pod \"payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77\" (UID: \"95241140-c937-47c1-9413-8bc8a68c840c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77"
Oct 30 13:24:18.950307 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:18.950154    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95241140-c937-47c1-9413-8bc8a68c840c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77\" (UID: \"95241140-c937-47c1-9413-8bc8a68c840c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77"
Oct 30 13:24:19.051406 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.051317    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9mbb\" (UniqueName: \"kubernetes.io/projected/95241140-c937-47c1-9413-8bc8a68c840c-kube-api-access-l9mbb\") pod \"payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77\" (UID: \"95241140-c937-47c1-9413-8bc8a68c840c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77"
Oct 30 13:24:19.051580 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.051420    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95241140-c937-47c1-9413-8bc8a68c840c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77\" (UID: \"95241140-c937-47c1-9413-8bc8a68c840c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77"
Oct 30 13:24:19.052105 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.052083    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95241140-c937-47c1-9413-8bc8a68c840c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77\" (UID: \"95241140-c937-47c1-9413-8bc8a68c840c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77"
Oct 30 13:24:19.075176 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.075141    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9mbb\" (UniqueName: \"kubernetes.io/projected/95241140-c937-47c1-9413-8bc8a68c840c-kube-api-access-l9mbb\") pod \"payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77\" (UID: \"95241140-c937-47c1-9413-8bc8a68c840c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77"
Oct 30 13:24:19.091670 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.091640    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="382ed63d-b520-4d49-ba0b-5fc982f309df" path="/var/lib/kubelet/pods/382ed63d-b520-4d49-ba0b-5fc982f309df/volumes"
Oct 30 13:24:19.092092 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.092079    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6abef82a-570c-494b-afe4-83bf81a4330c" path="/var/lib/kubelet/pods/6abef82a-570c-494b-afe4-83bf81a4330c/volumes"
Oct 30 13:24:19.176712 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.176668    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77"
Oct 30 13:24:19.210566 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.210535    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6"]
Oct 30 13:24:19.211411 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:24:19.211387    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod480fecc2_7659_45df_b613_3e0f2b0f7fff.slice/crio-d9c74b63cd3a9f0e66e13de689a3f03bc780eb531fc7befbae61731dcc4b0754 WatchSource:0}: Error finding container d9c74b63cd3a9f0e66e13de689a3f03bc780eb531fc7befbae61731dcc4b0754: Status 404 returned error can't find the container with id d9c74b63cd3a9f0e66e13de689a3f03bc780eb531fc7befbae61731dcc4b0754
Oct 30 13:24:19.299534 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.299495    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6" event={"ID":"480fecc2-7659-45df-b613-3e0f2b0f7fff","Type":"ContainerStarted","Data":"d9c74b63cd3a9f0e66e13de689a3f03bc780eb531fc7befbae61731dcc4b0754"}
Oct 30 13:24:19.496465 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.496423    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77"]
Oct 30 13:24:19.535523 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.535489    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 13:24:19.560909 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.560870    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-workload-certs\") pod \"3381ce8c-5918-4339-aa75-dd37e7723ee8\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") "
Oct 30 13:24:19.561114 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.561017    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3381ce8c-5918-4339-aa75-dd37e7723ee8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3381ce8c-5918-4339-aa75-dd37e7723ee8-istio-podinfo\") pod \"3381ce8c-5918-4339-aa75-dd37e7723ee8\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") "
Oct 30 13:24:19.561114 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.561050    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3381ce8c-5918-4339-aa75-dd37e7723ee8-kube-api-access-6g77n\" (UniqueName: \"kubernetes.io/projected/3381ce8c-5918-4339-aa75-dd37e7723ee8-kube-api-access-6g77n\") pod \"3381ce8c-5918-4339-aa75-dd37e7723ee8\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") "
Oct 30 13:24:19.561114 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.561097    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-workload-certs" pod "3381ce8c-5918-4339-aa75-dd37e7723ee8" (UID: "3381ce8c-5918-4339-aa75-dd37e7723ee8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:19.561307 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.561147    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3381ce8c-5918-4339-aa75-dd37e7723ee8-istio-token\" (UniqueName: \"kubernetes.io/projected/3381ce8c-5918-4339-aa75-dd37e7723ee8-istio-token\") pod \"3381ce8c-5918-4339-aa75-dd37e7723ee8\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") "
Oct 30 13:24:19.561307 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.561245    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-workload-socket\") pod \"3381ce8c-5918-4339-aa75-dd37e7723ee8\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") "
Oct 30 13:24:19.561307 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.561273    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-istio-data\") pod \"3381ce8c-5918-4339-aa75-dd37e7723ee8\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") "
Oct 30 13:24:19.561492 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.561317    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-credential-socket\") pod \"3381ce8c-5918-4339-aa75-dd37e7723ee8\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") "
Oct 30 13:24:19.561492 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.561431    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3381ce8c-5918-4339-aa75-dd37e7723ee8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3381ce8c-5918-4339-aa75-dd37e7723ee8-istiod-ca-cert\") pod \"3381ce8c-5918-4339-aa75-dd37e7723ee8\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") "
Oct 30 13:24:19.561492 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.561464    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-istio-envoy\") pod \"3381ce8c-5918-4339-aa75-dd37e7723ee8\" (UID: \"3381ce8c-5918-4339-aa75-dd37e7723ee8\") "
Oct 30 13:24:19.561660 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.561581    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-istio-data" pod "3381ce8c-5918-4339-aa75-dd37e7723ee8" (UID: "3381ce8c-5918-4339-aa75-dd37e7723ee8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:19.561660 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.561593    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-workload-socket" pod "3381ce8c-5918-4339-aa75-dd37e7723ee8" (UID: "3381ce8c-5918-4339-aa75-dd37e7723ee8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:19.561787 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.561724    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-credential-socket" pod "3381ce8c-5918-4339-aa75-dd37e7723ee8" (UID: "3381ce8c-5918-4339-aa75-dd37e7723ee8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:19.561923 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.561897    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3381ce8c-5918-4339-aa75-dd37e7723ee8-istiod-ca-cert" pod "3381ce8c-5918-4339-aa75-dd37e7723ee8" (UID: "3381ce8c-5918-4339-aa75-dd37e7723ee8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:19.562275 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.562214    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3381ce8c-5918-4339-aa75-dd37e7723ee8-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:19.562275 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.562273    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:19.562466 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.562282    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:19.562466 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.562291    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:19.562466 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.562301    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:19.563567 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.563533    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3381ce8c-5918-4339-aa75-dd37e7723ee8-istio-podinfo" pod "3381ce8c-5918-4339-aa75-dd37e7723ee8" (UID: "3381ce8c-5918-4339-aa75-dd37e7723ee8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:24:19.563757 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.563735    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3381ce8c-5918-4339-aa75-dd37e7723ee8-istio-token" pod "3381ce8c-5918-4339-aa75-dd37e7723ee8" (UID: "3381ce8c-5918-4339-aa75-dd37e7723ee8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:19.564016 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.563994    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-istio-envoy" pod "3381ce8c-5918-4339-aa75-dd37e7723ee8" (UID: "3381ce8c-5918-4339-aa75-dd37e7723ee8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:19.564142 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.564112    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3381ce8c-5918-4339-aa75-dd37e7723ee8-kube-api-access-6g77n" pod "3381ce8c-5918-4339-aa75-dd37e7723ee8" (UID: "3381ce8c-5918-4339-aa75-dd37e7723ee8"). InnerVolumeSpecName "kube-api-access-6g77n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:19.651931 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.651832    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:19.663607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.663573    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3381ce8c-5918-4339-aa75-dd37e7723ee8-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:19.663607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.663602    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3381ce8c-5918-4339-aa75-dd37e7723ee8-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:19.663607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.663611    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3381ce8c-5918-4339-aa75-dd37e7723ee8-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:19.663849 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:19.663619    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g77n\" (UniqueName: \"kubernetes.io/projected/3381ce8c-5918-4339-aa75-dd37e7723ee8-kube-api-access-6g77n\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:20.309560 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:20.309527    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77" event={"ID":"95241140-c937-47c1-9413-8bc8a68c840c","Type":"ContainerStarted","Data":"abb695578156c6415810b0f7c200de205f3cb192fe6aa53448436344a42167f7"}
Oct 30 13:24:20.309770 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:20.309581    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77"
Oct 30 13:24:20.309770 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:20.309605    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77" event={"ID":"95241140-c937-47c1-9413-8bc8a68c840c","Type":"ContainerStarted","Data":"bcb326a362c34cf268e2a661860d8568e5f88ed9f02d2c553d270834f88eab09"}
Oct 30 13:24:20.311456 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:20.311425    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6" event={"ID":"480fecc2-7659-45df-b613-3e0f2b0f7fff","Type":"ContainerStarted","Data":"85d2ff318602232335b2e698eb4b684b3ac2c6772e540faebfa9c0d30145a327"}
Oct 30 13:24:20.311596 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:20.311514    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6"
Oct 30 13:24:20.313162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:20.313140    4544 generic.go:356] "Generic (PLEG): container finished" podID="3381ce8c-5918-4339-aa75-dd37e7723ee8" containerID="1f2ab98f4f237119db700d8fde4d491f25a005ece9fc417356b7d007cb2473ba" exitCode=0
Oct 30 13:24:20.313295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:20.313186    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"
Oct 30 13:24:20.313295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:20.313250    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz" event={"ID":"3381ce8c-5918-4339-aa75-dd37e7723ee8","Type":"ContainerDied","Data":"1f2ab98f4f237119db700d8fde4d491f25a005ece9fc417356b7d007cb2473ba"}
Oct 30 13:24:20.313400 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:20.313296    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz" event={"ID":"3381ce8c-5918-4339-aa75-dd37e7723ee8","Type":"ContainerDied","Data":"7b6292574bce6f1aeba42c62bcf6be45710b1617da3434e42f162719e2e56636"}
Oct 30 13:24:20.313400 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:20.313319    4544 scope.go:122] "RemoveContainer" containerID="1f2ab98f4f237119db700d8fde4d491f25a005ece9fc417356b7d007cb2473ba"
Oct 30 13:24:20.324842 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:20.324824    4544 scope.go:122] "RemoveContainer" containerID="76d71583d1b23504e34a005ae7664b45d7effd76ef14cfabba1f253e5c7bee59"
Oct 30 13:24:20.335147 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:20.335127    4544 scope.go:122] "RemoveContainer" containerID="1f2ab98f4f237119db700d8fde4d491f25a005ece9fc417356b7d007cb2473ba"
Oct 30 13:24:20.335503 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:24:20.335480    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f2ab98f4f237119db700d8fde4d491f25a005ece9fc417356b7d007cb2473ba\": container with ID starting with 1f2ab98f4f237119db700d8fde4d491f25a005ece9fc417356b7d007cb2473ba not found: ID does not exist" containerID="1f2ab98f4f237119db700d8fde4d491f25a005ece9fc417356b7d007cb2473ba"
Oct 30 13:24:20.335567 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:20.335519    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f2ab98f4f237119db700d8fde4d491f25a005ece9fc417356b7d007cb2473ba"} err="failed to get container status \"1f2ab98f4f237119db700d8fde4d491f25a005ece9fc417356b7d007cb2473ba\": rpc error: code = NotFound desc = could not find container \"1f2ab98f4f237119db700d8fde4d491f25a005ece9fc417356b7d007cb2473ba\": container with ID starting with 1f2ab98f4f237119db700d8fde4d491f25a005ece9fc417356b7d007cb2473ba not found: ID does not exist"
Oct 30 13:24:20.335567 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:20.335538    4544 scope.go:122] "RemoveContainer" containerID="76d71583d1b23504e34a005ae7664b45d7effd76ef14cfabba1f253e5c7bee59"
Oct 30 13:24:20.335812 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:24:20.335794    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76d71583d1b23504e34a005ae7664b45d7effd76ef14cfabba1f253e5c7bee59\": container with ID starting with 76d71583d1b23504e34a005ae7664b45d7effd76ef14cfabba1f253e5c7bee59 not found: ID does not exist" containerID="76d71583d1b23504e34a005ae7664b45d7effd76ef14cfabba1f253e5c7bee59"
Oct 30 13:24:20.335855 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:20.335821    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76d71583d1b23504e34a005ae7664b45d7effd76ef14cfabba1f253e5c7bee59"} err="failed to get container status \"76d71583d1b23504e34a005ae7664b45d7effd76ef14cfabba1f253e5c7bee59\": rpc error: code = NotFound desc = could not find container \"76d71583d1b23504e34a005ae7664b45d7effd76ef14cfabba1f253e5c7bee59\": container with ID starting with 76d71583d1b23504e34a005ae7664b45d7effd76ef14cfabba1f253e5c7bee59 not found: ID does not exist"
Oct 30 13:24:20.354496 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:20.354448    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77" podStartSLOduration=2.354436677 podStartE2EDuration="2.354436677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:24:20.349886558 +0000 UTC m=+2355.397020512" watchObservedRunningTime="2026-10-30 13:24:20.354436677 +0000 UTC m=+2355.401570630"
Oct 30 13:24:20.394355 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:20.394320    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"]
Oct 30 13:24:20.404259 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:20.404212    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz"]
Oct 30 13:24:20.431867 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:20.431812    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6" podStartSLOduration=2.431792915 podStartE2EDuration="2.431792915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:24:20.430831886 +0000 UTC m=+2355.477965838" watchObservedRunningTime="2026-10-30 13:24:20.431792915 +0000 UTC m=+2355.478926867"
Oct 30 13:24:21.091184 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:21.091149    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3381ce8c-5918-4339-aa75-dd37e7723ee8" path="/var/lib/kubelet/pods/3381ce8c-5918-4339-aa75-dd37e7723ee8/volumes"
Oct 30 13:24:21.423557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:21.423456    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:23.433922 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:23.433888    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:25.379074 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:25.379039    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:27.180570 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:27.180533    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:28.052993 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.052959    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"]
Oct 30 13:24:28.053999 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.053979    4544 cpu_manager.go:414] "containerMap: removing container" podUID="3381ce8c-5918-4339-aa75-dd37e7723ee8" containerName="istio-proxy"
Oct 30 13:24:28.053999 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.053996    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3381ce8c-5918-4339-aa75-dd37e7723ee8" containerName="istio-proxy"
Oct 30 13:24:28.054189 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.054180    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="3381ce8c-5918-4339-aa75-dd37e7723ee8" containerName="istio-proxy"
Oct 30 13:24:28.054278 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.054198    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="3381ce8c-5918-4339-aa75-dd37e7723ee8" containerName="istio-proxy"
Oct 30 13:24:28.059548 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.059524    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:24:28.084554 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.084528    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2b8461-openshift-default-dockercfg-rpkjj\""
Oct 30 13:24:28.104581 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.104542    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"]
Oct 30 13:24:28.107179 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.107150    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-credential-socket\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:24:28.107346 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.107213    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istio-envoy\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:24:28.107444 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.107414    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-workload-socket\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:24:28.107532 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.107497    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istio-podinfo\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:24:28.107842 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.107816    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istiod-ca-cert\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:24:28.107977 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.107865    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istio-data\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:24:28.108053 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.108015    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istio-token\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:24:28.108117 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.108051    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-workload-certs\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:24:28.108178 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.108125    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmbxz\" (UniqueName: \"kubernetes.io/projected/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-kube-api-access-lmbxz\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:24:28.209800 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.209761    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istio-podinfo\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:24:28.210300 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.209927    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istiod-ca-cert\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:24:28.210300 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.209958    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istio-data\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:24:28.210300 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.210026    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istio-token\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:24:28.210300 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.210054    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-workload-certs\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:24:28.210300 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.210122    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmbxz\" (UniqueName: \"kubernetes.io/projected/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-kube-api-access-lmbxz\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:24:28.210300 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.210162    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-credential-socket\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:24:28.210300 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.210211    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istio-envoy\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:24:28.210757 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.210367    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-workload-socket\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:24:28.210757 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.210435    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istio-data\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:24:28.210757 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.210580    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-credential-socket\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:24:28.210757 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.210659    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-workload-certs\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:24:28.210757 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.210727    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-workload-socket\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:24:28.211207 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.211187    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istiod-ca-cert\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:24:28.212446 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.212428    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istio-podinfo\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:24:28.212534 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.212516    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istio-envoy\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:24:28.234431 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.234398    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istio-token\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:24:28.235141 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.235122    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmbxz\" (UniqueName: \"kubernetes.io/projected/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-kube-api-access-lmbxz\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:24:28.377508 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.377472    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:24:28.724445 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.724412    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"]
Oct 30 13:24:28.724885 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:24:28.724861    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf08cecf0_f7ec_4d03_8168_8b35a8446ba4.slice/crio-140b3e1e52bb27d2ad593563954c3c186893579da0f9fe0d1493fff2e226377f WatchSource:0}: Error finding container 140b3e1e52bb27d2ad593563954c3c186893579da0f9fe0d1493fff2e226377f: Status 404 returned error can't find the container with id 140b3e1e52bb27d2ad593563954c3c186893579da0f9fe0d1493fff2e226377f
Oct 30 13:24:28.727644 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.727609    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:24:28.727755 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.727688    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:24:28.727755 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:28.727737    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:24:29.015235 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:24:29.015132    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:24:29.113449 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:29.113409    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:29.399690 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:29.399653    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7" event={"ID":"f08cecf0-f7ec-4d03-8168-8b35a8446ba4","Type":"ContainerStarted","Data":"0759e23d488af4acd49ab4173f96c35fe2f1e3d2bad81a2d4a6afb92535ab652"}
Oct 30 13:24:29.400093 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:29.399711    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7" event={"ID":"f08cecf0-f7ec-4d03-8168-8b35a8446ba4","Type":"ContainerStarted","Data":"140b3e1e52bb27d2ad593563954c3c186893579da0f9fe0d1493fff2e226377f"}
Oct 30 13:24:29.466884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:29.466820    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7" podStartSLOduration=1.466798979 podStartE2EDuration="1.466798979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:24:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:24:29.454231541 +0000 UTC m=+2364.501365486" watchObservedRunningTime="2026-10-30 13:24:29.466798979 +0000 UTC m=+2364.513932975"
Oct 30 13:24:29.505054 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:29.505017    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn"]
Oct 30 13:24:29.697787 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:29.696076    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:29.708022 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:29.707981    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-618c72-zczws"]
Oct 30 13:24:29.714508 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:29.709638    4544 cpu_manager.go:414] "containerMap: removing container" podUID="3381ce8c-5918-4339-aa75-dd37e7723ee8" containerName="istio-proxy"
Oct 30 13:24:29.714508 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:29.714498    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3381ce8c-5918-4339-aa75-dd37e7723ee8" containerName="istio-proxy"
Oct 30 13:24:29.723284 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:29.721179    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-618c72-zczws"
Oct 30 13:24:29.734352 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:29.734071    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-618c72-zczws"]
Oct 30 13:24:29.758242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:29.756135    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:24:29.758242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:29.756214    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:24:29.784682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:29.784212    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4adbe0d-49ae-4f5a-9cfa-449a4da119d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-618c72-zczws\" (UID: \"e4adbe0d-49ae-4f5a-9cfa-449a4da119d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-618c72-zczws"
Oct 30 13:24:29.784682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:29.784384    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76rfv\" (UniqueName: \"kubernetes.io/projected/e4adbe0d-49ae-4f5a-9cfa-449a4da119d5-kube-api-access-76rfv\") pod \"maas-api-revoke-keys-e2e-inf-a-618c72-zczws\" (UID: \"e4adbe0d-49ae-4f5a-9cfa-449a4da119d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-618c72-zczws"
Oct 30 13:24:29.784682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:29.784513    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4adbe0d-49ae-4f5a-9cfa-449a4da119d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-618c72-zczws\" (UID: \"e4adbe0d-49ae-4f5a-9cfa-449a4da119d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-618c72-zczws"
Oct 30 13:24:29.886466 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:29.886426    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76rfv\" (UniqueName: \"kubernetes.io/projected/e4adbe0d-49ae-4f5a-9cfa-449a4da119d5-kube-api-access-76rfv\") pod \"maas-api-revoke-keys-e2e-inf-a-618c72-zczws\" (UID: \"e4adbe0d-49ae-4f5a-9cfa-449a4da119d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-618c72-zczws"
Oct 30 13:24:29.886655 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:29.886552    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4adbe0d-49ae-4f5a-9cfa-449a4da119d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-618c72-zczws\" (UID: \"e4adbe0d-49ae-4f5a-9cfa-449a4da119d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-618c72-zczws"
Oct 30 13:24:29.886712 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:29.886669    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4adbe0d-49ae-4f5a-9cfa-449a4da119d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-618c72-zczws\" (UID: \"e4adbe0d-49ae-4f5a-9cfa-449a4da119d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-618c72-zczws"
Oct 30 13:24:29.887286 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:29.887261    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4adbe0d-49ae-4f5a-9cfa-449a4da119d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-618c72-zczws\" (UID: \"e4adbe0d-49ae-4f5a-9cfa-449a4da119d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-618c72-zczws"
Oct 30 13:24:29.889099 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:29.889071    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4adbe0d-49ae-4f5a-9cfa-449a4da119d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-618c72-zczws\" (UID: \"e4adbe0d-49ae-4f5a-9cfa-449a4da119d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-618c72-zczws"
Oct 30 13:24:29.949479 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:29.949389    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76rfv\" (UniqueName: \"kubernetes.io/projected/e4adbe0d-49ae-4f5a-9cfa-449a4da119d5-kube-api-access-76rfv\") pod \"maas-api-revoke-keys-e2e-inf-a-618c72-zczws\" (UID: \"e4adbe0d-49ae-4f5a-9cfa-449a4da119d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-618c72-zczws"
Oct 30 13:24:30.043050 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:30.043007    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11ad6987-rn5cj"]
Oct 30 13:24:30.048001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:30.047965    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11ad6987-rn5cj"]
Oct 30 13:24:30.091472 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:30.091441    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-618c72-zczws"
Oct 30 13:24:30.377792 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:30.377758    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:24:30.379274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:30.379211    4544 patch_prober.go:29] interesting pod/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Oct 30 13:24:30.379406 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:30.379290    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7" podUID="f08cecf0-f7ec-4d03-8168-8b35a8446ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Oct 30 13:24:30.444115 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:30.444061    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-618c72-zczws"]
Oct 30 13:24:30.444828 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:24:30.444803    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4adbe0d_49ae_4f5a_9cfa_449a4da119d5.slice/crio-c85ffd8e1e29da95f2f649f3a00cb46fe4808cf1072d2e2f81d3250fcd732afd WatchSource:0}: Error finding container c85ffd8e1e29da95f2f649f3a00cb46fe4808cf1072d2e2f81d3250fcd732afd: Status 404 returned error can't find the container with id c85ffd8e1e29da95f2f649f3a00cb46fe4808cf1072d2e2f81d3250fcd732afd
Oct 30 13:24:31.051109 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:31.051012    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh"]
Oct 30 13:24:31.051378 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:31.051266    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh" podUID="55d83701-6415-49ca-877d-d66fc282b2eb" containerName="maas-api" containerID="cri-o://4d23d901d6a8d9c6051cd44d48671c07821bf27c398f906ac0738cc47cf1920b" gracePeriod=30
Oct 30 13:24:31.095857 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:31.095713    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8" path="/var/lib/kubelet/pods/0bdcd77f-9e49-468f-a3f1-96ef6cc33eb8/volumes"
Oct 30 13:24:31.150182 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:31.150120    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:31.198440 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:24:31.198363    4544 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55d83701_6415_49ca_877d_d66fc282b2eb.slice/crio-4d23d901d6a8d9c6051cd44d48671c07821bf27c398f906ac0738cc47cf1920b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55d83701_6415_49ca_877d_d66fc282b2eb.slice/crio-conmon-4d23d901d6a8d9c6051cd44d48671c07821bf27c398f906ac0738cc47cf1920b.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 13:24:31.404076 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:31.402658    4544 patch_prober.go:29] interesting pod/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Oct 30 13:24:31.404076 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:31.402715    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7" podUID="f08cecf0-f7ec-4d03-8168-8b35a8446ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Oct 30 13:24:31.446242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:31.445631    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77"
Oct 30 13:24:31.545955 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:31.536650    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-618c72-zczws" event={"ID":"e4adbe0d-49ae-4f5a-9cfa-449a4da119d5","Type":"ContainerStarted","Data":"2f0850854403dfd8f2340d0c15cb5f1e9a3b056e803c8ecd4592f7d94c7c4d50"}
Oct 30 13:24:31.545955 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:31.536716    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-618c72-zczws" event={"ID":"e4adbe0d-49ae-4f5a-9cfa-449a4da119d5","Type":"ContainerStarted","Data":"c85ffd8e1e29da95f2f649f3a00cb46fe4808cf1072d2e2f81d3250fcd732afd"}
Oct 30 13:24:31.813563 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:31.812201    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6"
Oct 30 13:24:31.849247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:31.848127    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-618c72-zczws" podStartSLOduration=2.848105324 podStartE2EDuration="2.848105324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:24:31.731458547 +0000 UTC m=+2366.778592523" watchObservedRunningTime="2026-10-30 13:24:31.848105324 +0000 UTC m=+2366.895239278"
Oct 30 13:24:31.858664 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:31.858630    4544 generic.go:356] "Generic (PLEG): container finished" podID="55d83701-6415-49ca-877d-d66fc282b2eb" containerID="4d23d901d6a8d9c6051cd44d48671c07821bf27c398f906ac0738cc47cf1920b" exitCode=0
Oct 30 13:24:31.859654 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:31.859610    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh" event={"ID":"55d83701-6415-49ca-877d-d66fc282b2eb","Type":"ContainerDied","Data":"4d23d901d6a8d9c6051cd44d48671c07821bf27c398f906ac0738cc47cf1920b"}
Oct 30 13:24:31.862081 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:31.862046    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh"
Oct 30 13:24:31.952593 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:31.952554    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55d83701-6415-49ca-877d-d66fc282b2eb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55d83701-6415-49ca-877d-d66fc282b2eb-openshift-service-ca\") pod \"55d83701-6415-49ca-877d-d66fc282b2eb\" (UID: \"55d83701-6415-49ca-877d-d66fc282b2eb\") "
Oct 30 13:24:31.953970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:31.953918    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55d83701-6415-49ca-877d-d66fc282b2eb-openshift-service-ca" pod "55d83701-6415-49ca-877d-d66fc282b2eb" (UID: "55d83701-6415-49ca-877d-d66fc282b2eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:31.961187 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:31.952619    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55d83701-6415-49ca-877d-d66fc282b2eb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55d83701-6415-49ca-877d-d66fc282b2eb-maas-api-tls\") pod \"55d83701-6415-49ca-877d-d66fc282b2eb\" (UID: \"55d83701-6415-49ca-877d-d66fc282b2eb\") "
Oct 30 13:24:31.961488 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:31.961214    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55d83701-6415-49ca-877d-d66fc282b2eb-maas-api-tls" pod "55d83701-6415-49ca-877d-d66fc282b2eb" (UID: "55d83701-6415-49ca-877d-d66fc282b2eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:24:31.961761 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:31.961722    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55d83701-6415-49ca-877d-d66fc282b2eb-kube-api-access-5vl8p\" (UniqueName: \"kubernetes.io/projected/55d83701-6415-49ca-877d-d66fc282b2eb-kube-api-access-5vl8p\") pod \"55d83701-6415-49ca-877d-d66fc282b2eb\" (UID: \"55d83701-6415-49ca-877d-d66fc282b2eb\") "
Oct 30 13:24:31.961907 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:31.961836    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55d83701-6415-49ca-877d-d66fc282b2eb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/55d83701-6415-49ca-877d-d66fc282b2eb-metrics-certs\") pod \"55d83701-6415-49ca-877d-d66fc282b2eb\" (UID: \"55d83701-6415-49ca-877d-d66fc282b2eb\") "
Oct 30 13:24:31.964381 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:31.963092    4544 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55d83701-6415-49ca-877d-d66fc282b2eb-openshift-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:31.964381 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:31.963119    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55d83701-6415-49ca-877d-d66fc282b2eb-maas-api-tls\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:31.964381 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:31.964327    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55d83701-6415-49ca-877d-d66fc282b2eb-kube-api-access-5vl8p" pod "55d83701-6415-49ca-877d-d66fc282b2eb" (UID: "55d83701-6415-49ca-877d-d66fc282b2eb"). InnerVolumeSpecName "kube-api-access-5vl8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:31.965993 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:31.965966    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55d83701-6415-49ca-877d-d66fc282b2eb-metrics-certs" pod "55d83701-6415-49ca-877d-d66fc282b2eb" (UID: "55d83701-6415-49ca-877d-d66fc282b2eb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:24:32.033347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:32.033156    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:32.065831 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:32.065713    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vl8p\" (UniqueName: \"kubernetes.io/projected/55d83701-6415-49ca-877d-d66fc282b2eb-kube-api-access-5vl8p\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:32.065831 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:32.065758    4544 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55d83701-6415-49ca-877d-d66fc282b2eb-metrics-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:32.285675 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:32.285640    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:32.389327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:32.384045    4544 patch_prober.go:29] interesting pod/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Oct 30 13:24:32.389327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:32.384110    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7" podUID="f08cecf0-f7ec-4d03-8168-8b35a8446ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Oct 30 13:24:32.389327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:32.384529    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80fbd9-6ncg8"]
Oct 30 13:24:32.390728 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:32.390437    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80fbd9-6ncg8"]
Oct 30 13:24:32.428407 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:32.427057    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:32.883500 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:32.883454    4544 generic.go:356] "Generic (PLEG): container finished" podID="e4adbe0d-49ae-4f5a-9cfa-449a4da119d5" containerID="2f0850854403dfd8f2340d0c15cb5f1e9a3b056e803c8ecd4592f7d94c7c4d50" exitCode=0
Oct 30 13:24:32.884001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:32.883553    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-618c72-zczws" event={"ID":"e4adbe0d-49ae-4f5a-9cfa-449a4da119d5","Type":"ContainerDied","Data":"2f0850854403dfd8f2340d0c15cb5f1e9a3b056e803c8ecd4592f7d94c7c4d50"}
Oct 30 13:24:32.885428 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:32.885405    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh"
Oct 30 13:24:32.885565 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:32.885410    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh" event={"ID":"55d83701-6415-49ca-877d-d66fc282b2eb","Type":"ContainerDied","Data":"457a605c1f03ee686dc76b263d68cbb0c1c2ad75c06857287909cb4edb4133f4"}
Oct 30 13:24:32.885565 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:32.885531    4544 scope.go:122] "RemoveContainer" containerID="4d23d901d6a8d9c6051cd44d48671c07821bf27c398f906ac0738cc47cf1920b"
Oct 30 13:24:32.993532 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:32.993490    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh"]
Oct 30 13:24:32.998336 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:32.998302    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-80fbd9-59974f67d-pnbgh"]
Oct 30 13:24:33.007736 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:33.007705    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:33.088915 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:33.088884    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55d83701-6415-49ca-877d-d66fc282b2eb" path="/var/lib/kubelet/pods/55d83701-6415-49ca-877d-d66fc282b2eb/volumes"
Oct 30 13:24:33.089354 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:33.089338    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="710a5656-e79a-4c00-8df1-99bcbd92e400" path="/var/lib/kubelet/pods/710a5656-e79a-4c00-8df1-99bcbd92e400/volumes"
Oct 30 13:24:33.378555 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:33.378514    4544 patch_prober.go:29] interesting pod/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Oct 30 13:24:33.378751 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:33.378590    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7" podUID="f08cecf0-f7ec-4d03-8168-8b35a8446ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Oct 30 13:24:34.150957 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:34.150881    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-618c72-zczws"
Oct 30 13:24:34.203108 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:34.203073    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4adbe0d-49ae-4f5a-9cfa-449a4da119d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4adbe0d-49ae-4f5a-9cfa-449a4da119d5-maas-api-openshift-ca\") pod \"e4adbe0d-49ae-4f5a-9cfa-449a4da119d5\" (UID: \"e4adbe0d-49ae-4f5a-9cfa-449a4da119d5\") "
Oct 30 13:24:34.203327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:34.203177    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4adbe0d-49ae-4f5a-9cfa-449a4da119d5-kube-api-access-76rfv\" (UniqueName: \"kubernetes.io/projected/e4adbe0d-49ae-4f5a-9cfa-449a4da119d5-kube-api-access-76rfv\") pod \"e4adbe0d-49ae-4f5a-9cfa-449a4da119d5\" (UID: \"e4adbe0d-49ae-4f5a-9cfa-449a4da119d5\") "
Oct 30 13:24:34.203327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:34.203229    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4adbe0d-49ae-4f5a-9cfa-449a4da119d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4adbe0d-49ae-4f5a-9cfa-449a4da119d5-maas-api-xks-ca\") pod \"e4adbe0d-49ae-4f5a-9cfa-449a4da119d5\" (UID: \"e4adbe0d-49ae-4f5a-9cfa-449a4da119d5\") "
Oct 30 13:24:34.203460 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:34.203432    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4adbe0d-49ae-4f5a-9cfa-449a4da119d5-maas-api-openshift-ca" pod "e4adbe0d-49ae-4f5a-9cfa-449a4da119d5" (UID: "e4adbe0d-49ae-4f5a-9cfa-449a4da119d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:34.203834 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:34.203791    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4adbe0d-49ae-4f5a-9cfa-449a4da119d5-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:34.205666 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:34.205644    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4adbe0d-49ae-4f5a-9cfa-449a4da119d5-maas-api-xks-ca" pod "e4adbe0d-49ae-4f5a-9cfa-449a4da119d5" (UID: "e4adbe0d-49ae-4f5a-9cfa-449a4da119d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:24:34.205879 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:34.205852    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4adbe0d-49ae-4f5a-9cfa-449a4da119d5-kube-api-access-76rfv" pod "e4adbe0d-49ae-4f5a-9cfa-449a4da119d5" (UID: "e4adbe0d-49ae-4f5a-9cfa-449a4da119d5"). InnerVolumeSpecName "kube-api-access-76rfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:34.305654 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:34.305615    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76rfv\" (UniqueName: \"kubernetes.io/projected/e4adbe0d-49ae-4f5a-9cfa-449a4da119d5-kube-api-access-76rfv\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:34.305654 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:34.305647    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4adbe0d-49ae-4f5a-9cfa-449a4da119d5-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:34.378150 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:34.378111    4544 patch_prober.go:29] interesting pod/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Oct 30 13:24:34.378390 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:34.378167    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7" podUID="f08cecf0-f7ec-4d03-8168-8b35a8446ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Oct 30 13:24:34.846868 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:34.846836    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:34.960808 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:34.960775    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-618c72-zczws"
Oct 30 13:24:34.961546 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:34.960784    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-618c72-zczws" event={"ID":"e4adbe0d-49ae-4f5a-9cfa-449a4da119d5","Type":"ContainerDied","Data":"c85ffd8e1e29da95f2f649f3a00cb46fe4808cf1072d2e2f81d3250fcd732afd"}
Oct 30 13:24:34.961634 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:34.961582    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c85ffd8e1e29da95f2f649f3a00cb46fe4808cf1072d2e2f81d3250fcd732afd"
Oct 30 13:24:35.378139 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:35.378097    4544 patch_prober.go:29] interesting pod/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Oct 30 13:24:35.378660 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:35.378182    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7" podUID="f08cecf0-f7ec-4d03-8168-8b35a8446ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Oct 30 13:24:36.383193 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:36.383158    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:24:36.383709 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:36.383687    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:24:36.384426 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:36.384407    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:24:36.611141 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:36.611106    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:38.376786 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:38.376755    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:40.230115 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:40.230079    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:40.399775 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:40.399728    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:40.551643 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:40.551546    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"]
Oct 30 13:24:40.551851 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:40.551809    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f" podUID="d685dcc9-430c-4f84-aa00-7506306061d1" containerName="istio-proxy" containerID="cri-o://a684ee71634864b805b6ef5845f903152c480e87f7342fa9edd32012cffe3ac3" gracePeriod=30
Oct 30 13:24:40.552819 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:40.552792    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:40.740620 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:40.737653    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:41.155671 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:41.155632    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bec983-pljc7"]
Oct 30 13:24:41.161780 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:41.161743    4544 cpu_manager.go:414] "containerMap: removing container" podUID="e4adbe0d-49ae-4f5a-9cfa-449a4da119d5" containerName="revoke-keys"
Oct 30 13:24:41.162015 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:41.161997    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4adbe0d-49ae-4f5a-9cfa-449a4da119d5" containerName="revoke-keys"
Oct 30 13:24:41.162257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:41.162239    4544 cpu_manager.go:414] "containerMap: removing container" podUID="55d83701-6415-49ca-877d-d66fc282b2eb" containerName="maas-api"
Oct 30 13:24:41.162378 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:41.162366    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55d83701-6415-49ca-877d-d66fc282b2eb" containerName="maas-api"
Oct 30 13:24:41.162916 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:41.162893    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4adbe0d-49ae-4f5a-9cfa-449a4da119d5" containerName="revoke-keys"
Oct 30 13:24:41.163088 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:41.163074    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="55d83701-6415-49ca-877d-d66fc282b2eb" containerName="maas-api"
Oct 30 13:24:41.178777 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:41.178741    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bec983-pljc7"
Oct 30 13:24:41.188440 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:41.178871    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bec983-pljc7"]
Oct 30 13:24:41.268570 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:41.268541    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:24:41.268963 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:41.268609    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:24:41.293116 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:41.293068    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9bb4805-2bd1-4cb0-9b36-200c9146dbc9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bec983-pljc7\" (UID: \"d9bb4805-2bd1-4cb0-9b36-200c9146dbc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bec983-pljc7"
Oct 30 13:24:41.293369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:41.293151    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkprg\" (UniqueName: \"kubernetes.io/projected/d9bb4805-2bd1-4cb0-9b36-200c9146dbc9-kube-api-access-rkprg\") pod \"maas-api-revoke-keys-e2e-inf-b-bec983-pljc7\" (UID: \"d9bb4805-2bd1-4cb0-9b36-200c9146dbc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bec983-pljc7"
Oct 30 13:24:41.293369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:41.293299    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9bb4805-2bd1-4cb0-9b36-200c9146dbc9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bec983-pljc7\" (UID: \"d9bb4805-2bd1-4cb0-9b36-200c9146dbc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bec983-pljc7"
Oct 30 13:24:41.394833 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:41.394782    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9bb4805-2bd1-4cb0-9b36-200c9146dbc9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bec983-pljc7\" (UID: \"d9bb4805-2bd1-4cb0-9b36-200c9146dbc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bec983-pljc7"
Oct 30 13:24:41.395074 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:41.394883    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkprg\" (UniqueName: \"kubernetes.io/projected/d9bb4805-2bd1-4cb0-9b36-200c9146dbc9-kube-api-access-rkprg\") pod \"maas-api-revoke-keys-e2e-inf-b-bec983-pljc7\" (UID: \"d9bb4805-2bd1-4cb0-9b36-200c9146dbc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bec983-pljc7"
Oct 30 13:24:41.395074 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:41.394967    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9bb4805-2bd1-4cb0-9b36-200c9146dbc9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bec983-pljc7\" (UID: \"d9bb4805-2bd1-4cb0-9b36-200c9146dbc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bec983-pljc7"
Oct 30 13:24:41.395855 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:41.395812    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9bb4805-2bd1-4cb0-9b36-200c9146dbc9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bec983-pljc7\" (UID: \"d9bb4805-2bd1-4cb0-9b36-200c9146dbc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bec983-pljc7"
Oct 30 13:24:41.397527 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:41.397493    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9bb4805-2bd1-4cb0-9b36-200c9146dbc9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bec983-pljc7\" (UID: \"d9bb4805-2bd1-4cb0-9b36-200c9146dbc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bec983-pljc7"
Oct 30 13:24:41.420117 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:41.420021    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkprg\" (UniqueName: \"kubernetes.io/projected/d9bb4805-2bd1-4cb0-9b36-200c9146dbc9-kube-api-access-rkprg\") pod \"maas-api-revoke-keys-e2e-inf-b-bec983-pljc7\" (UID: \"d9bb4805-2bd1-4cb0-9b36-200c9146dbc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bec983-pljc7"
Oct 30 13:24:41.507752 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:41.507711    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bec983-pljc7"
Oct 30 13:24:41.845448 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:41.845417    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bec983-pljc7"]
Oct 30 13:24:41.846966 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:24:41.846939    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9bb4805_2bd1_4cb0_9b36_200c9146dbc9.slice/crio-5bca7e21d06ee96d322775c421fd88102a521485d5396a310ca878aacab2f125 WatchSource:0}: Error finding container 5bca7e21d06ee96d322775c421fd88102a521485d5396a310ca878aacab2f125: Status 404 returned error can't find the container with id 5bca7e21d06ee96d322775c421fd88102a521485d5396a310ca878aacab2f125
Oct 30 13:24:42.109507 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:42.109408    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bec983-pljc7" event={"ID":"d9bb4805-2bd1-4cb0-9b36-200c9146dbc9","Type":"ContainerStarted","Data":"5bca7e21d06ee96d322775c421fd88102a521485d5396a310ca878aacab2f125"}
Oct 30 13:24:42.592939 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:42.591618    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:43.119347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:43.119315    4544 generic.go:356] "Generic (PLEG): container finished" podID="d9bb4805-2bd1-4cb0-9b36-200c9146dbc9" containerID="0553ce613f24bae9f1de2bd92b19b0799e347cc0594cd26d4d244251e06575ef" exitCode=0
Oct 30 13:24:43.119546 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:43.119400    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bec983-pljc7" event={"ID":"d9bb4805-2bd1-4cb0-9b36-200c9146dbc9","Type":"ContainerDied","Data":"0553ce613f24bae9f1de2bd92b19b0799e347cc0594cd26d4d244251e06575ef"}
Oct 30 13:24:44.299497 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:44.299470    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bec983-pljc7"
Oct 30 13:24:44.339641 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:44.339585    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9bb4805-2bd1-4cb0-9b36-200c9146dbc9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9bb4805-2bd1-4cb0-9b36-200c9146dbc9-maas-api-xks-ca\") pod \"d9bb4805-2bd1-4cb0-9b36-200c9146dbc9\" (UID: \"d9bb4805-2bd1-4cb0-9b36-200c9146dbc9\") "
Oct 30 13:24:44.339831 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:44.339786    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9bb4805-2bd1-4cb0-9b36-200c9146dbc9-kube-api-access-rkprg\" (UniqueName: \"kubernetes.io/projected/d9bb4805-2bd1-4cb0-9b36-200c9146dbc9-kube-api-access-rkprg\") pod \"d9bb4805-2bd1-4cb0-9b36-200c9146dbc9\" (UID: \"d9bb4805-2bd1-4cb0-9b36-200c9146dbc9\") "
Oct 30 13:24:44.339957 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:44.339926    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9bb4805-2bd1-4cb0-9b36-200c9146dbc9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9bb4805-2bd1-4cb0-9b36-200c9146dbc9-maas-api-openshift-ca\") pod \"d9bb4805-2bd1-4cb0-9b36-200c9146dbc9\" (UID: \"d9bb4805-2bd1-4cb0-9b36-200c9146dbc9\") "
Oct 30 13:24:44.340414 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:44.340374    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9bb4805-2bd1-4cb0-9b36-200c9146dbc9-maas-api-openshift-ca" pod "d9bb4805-2bd1-4cb0-9b36-200c9146dbc9" (UID: "d9bb4805-2bd1-4cb0-9b36-200c9146dbc9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:44.340623 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:44.340606    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9bb4805-2bd1-4cb0-9b36-200c9146dbc9-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:44.342642 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:44.342475    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9bb4805-2bd1-4cb0-9b36-200c9146dbc9-kube-api-access-rkprg" pod "d9bb4805-2bd1-4cb0-9b36-200c9146dbc9" (UID: "d9bb4805-2bd1-4cb0-9b36-200c9146dbc9"). InnerVolumeSpecName "kube-api-access-rkprg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:44.342642 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:44.342540    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9bb4805-2bd1-4cb0-9b36-200c9146dbc9-maas-api-xks-ca" pod "d9bb4805-2bd1-4cb0-9b36-200c9146dbc9" (UID: "d9bb4805-2bd1-4cb0-9b36-200c9146dbc9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:24:44.442652 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:44.442573    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkprg\" (UniqueName: \"kubernetes.io/projected/d9bb4805-2bd1-4cb0-9b36-200c9146dbc9-kube-api-access-rkprg\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:44.442652 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:44.442604    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9bb4805-2bd1-4cb0-9b36-200c9146dbc9-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:44.529421 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:44.529371    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn" podUID="ecb9846d-c5b3-4d4d-8243-4169400da4ca" containerName="main" containerID="cri-o://3697c8615d103a05fcf3bab13349506c6610f6483aab3161c85e11a1eb7d38c5" gracePeriod=45
Oct 30 13:24:44.712465 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:44.712432    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:44.836773 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:44.836744    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn"
Oct 30 13:24:44.949368 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:44.949329    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb9846d-c5b3-4d4d-8243-4169400da4ca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb9846d-c5b3-4d4d-8243-4169400da4ca-model-cache\") pod \"ecb9846d-c5b3-4d4d-8243-4169400da4ca\" (UID: \"ecb9846d-c5b3-4d4d-8243-4169400da4ca\") "
Oct 30 13:24:44.949548 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:44.949444    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb9846d-c5b3-4d4d-8243-4169400da4ca-home\" (UniqueName: \"kubernetes.io/empty-dir/ecb9846d-c5b3-4d4d-8243-4169400da4ca-home\") pod \"ecb9846d-c5b3-4d4d-8243-4169400da4ca\" (UID: \"ecb9846d-c5b3-4d4d-8243-4169400da4ca\") "
Oct 30 13:24:44.949548 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:44.949511    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb9846d-c5b3-4d4d-8243-4169400da4ca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb9846d-c5b3-4d4d-8243-4169400da4ca-tmp-dir\") pod \"ecb9846d-c5b3-4d4d-8243-4169400da4ca\" (UID: \"ecb9846d-c5b3-4d4d-8243-4169400da4ca\") "
Oct 30 13:24:44.949695 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:44.949574    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb9846d-c5b3-4d4d-8243-4169400da4ca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb9846d-c5b3-4d4d-8243-4169400da4ca-dshm\") pod \"ecb9846d-c5b3-4d4d-8243-4169400da4ca\" (UID: \"ecb9846d-c5b3-4d4d-8243-4169400da4ca\") "
Oct 30 13:24:44.949695 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:44.949595    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb9846d-c5b3-4d4d-8243-4169400da4ca-model-cache" pod "ecb9846d-c5b3-4d4d-8243-4169400da4ca" (UID: "ecb9846d-c5b3-4d4d-8243-4169400da4ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:44.949836 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:44.949691    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecb9846d-c5b3-4d4d-8243-4169400da4ca-kube-api-access-w5zc2\" (UniqueName: \"kubernetes.io/projected/ecb9846d-c5b3-4d4d-8243-4169400da4ca-kube-api-access-w5zc2\") pod \"ecb9846d-c5b3-4d4d-8243-4169400da4ca\" (UID: \"ecb9846d-c5b3-4d4d-8243-4169400da4ca\") "
Oct 30 13:24:44.949836 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:44.949746    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecb9846d-c5b3-4d4d-8243-4169400da4ca-tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb9846d-c5b3-4d4d-8243-4169400da4ca-tls-certs\") pod \"ecb9846d-c5b3-4d4d-8243-4169400da4ca\" (UID: \"ecb9846d-c5b3-4d4d-8243-4169400da4ca\") "
Oct 30 13:24:44.949836 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:44.949755    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb9846d-c5b3-4d4d-8243-4169400da4ca-home" pod "ecb9846d-c5b3-4d4d-8243-4169400da4ca" (UID: "ecb9846d-c5b3-4d4d-8243-4169400da4ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:44.949836 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:44.949781    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb9846d-c5b3-4d4d-8243-4169400da4ca-tmp-dir" pod "ecb9846d-c5b3-4d4d-8243-4169400da4ca" (UID: "ecb9846d-c5b3-4d4d-8243-4169400da4ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:44.950362 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:44.950341    4544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb9846d-c5b3-4d4d-8243-4169400da4ca-home\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:44.950362 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:44.950363    4544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb9846d-c5b3-4d4d-8243-4169400da4ca-tmp-dir\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:44.950513 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:44.950377    4544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb9846d-c5b3-4d4d-8243-4169400da4ca-model-cache\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:44.951955 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:44.951928    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb9846d-c5b3-4d4d-8243-4169400da4ca-dshm" pod "ecb9846d-c5b3-4d4d-8243-4169400da4ca" (UID: "ecb9846d-c5b3-4d4d-8243-4169400da4ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:44.951955 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:44.951941    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb9846d-c5b3-4d4d-8243-4169400da4ca-kube-api-access-w5zc2" pod "ecb9846d-c5b3-4d4d-8243-4169400da4ca" (UID: "ecb9846d-c5b3-4d4d-8243-4169400da4ca"). InnerVolumeSpecName "kube-api-access-w5zc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:44.951955 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:44.951944    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecb9846d-c5b3-4d4d-8243-4169400da4ca-tls-certs" pod "ecb9846d-c5b3-4d4d-8243-4169400da4ca" (UID: "ecb9846d-c5b3-4d4d-8243-4169400da4ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:24:45.051724 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.051631    4544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb9846d-c5b3-4d4d-8243-4169400da4ca-dshm\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:45.051724 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.051664    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5zc2\" (UniqueName: \"kubernetes.io/projected/ecb9846d-c5b3-4d4d-8243-4169400da4ca-kube-api-access-w5zc2\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:45.051724 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.051683    4544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb9846d-c5b3-4d4d-8243-4169400da4ca-tls-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:45.152795 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.152761    4544 generic.go:356] "Generic (PLEG): container finished" podID="ecb9846d-c5b3-4d4d-8243-4169400da4ca" containerID="3697c8615d103a05fcf3bab13349506c6610f6483aab3161c85e11a1eb7d38c5" exitCode=0
Oct 30 13:24:45.152967 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.152834    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn"
Oct 30 13:24:45.152967 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.152838    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn" event={"ID":"ecb9846d-c5b3-4d4d-8243-4169400da4ca","Type":"ContainerDied","Data":"3697c8615d103a05fcf3bab13349506c6610f6483aab3161c85e11a1eb7d38c5"}
Oct 30 13:24:45.152967 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.152881    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn" event={"ID":"ecb9846d-c5b3-4d4d-8243-4169400da4ca","Type":"ContainerDied","Data":"69730359c0402e5b28d77a76f16a9285680aec0bfb44ae567f88df14d543b7e9"}
Oct 30 13:24:45.152967 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.152898    4544 scope.go:122] "RemoveContainer" containerID="3697c8615d103a05fcf3bab13349506c6610f6483aab3161c85e11a1eb7d38c5"
Oct 30 13:24:45.154719 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.154695    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bec983-pljc7" event={"ID":"d9bb4805-2bd1-4cb0-9b36-200c9146dbc9","Type":"ContainerDied","Data":"5bca7e21d06ee96d322775c421fd88102a521485d5396a310ca878aacab2f125"}
Oct 30 13:24:45.154847 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.154716    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bec983-pljc7"
Oct 30 13:24:45.154847 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.154794    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bca7e21d06ee96d322775c421fd88102a521485d5396a310ca878aacab2f125"
Oct 30 13:24:45.164750 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.164728    4544 scope.go:122] "RemoveContainer" containerID="3697c8615d103a05fcf3bab13349506c6610f6483aab3161c85e11a1eb7d38c5"
Oct 30 13:24:45.165084 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:24:45.165062    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3697c8615d103a05fcf3bab13349506c6610f6483aab3161c85e11a1eb7d38c5\": container with ID starting with 3697c8615d103a05fcf3bab13349506c6610f6483aab3161c85e11a1eb7d38c5 not found: ID does not exist" containerID="3697c8615d103a05fcf3bab13349506c6610f6483aab3161c85e11a1eb7d38c5"
Oct 30 13:24:45.165154 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.165096    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3697c8615d103a05fcf3bab13349506c6610f6483aab3161c85e11a1eb7d38c5"} err="failed to get container status \"3697c8615d103a05fcf3bab13349506c6610f6483aab3161c85e11a1eb7d38c5\": rpc error: code = NotFound desc = could not find container \"3697c8615d103a05fcf3bab13349506c6610f6483aab3161c85e11a1eb7d38c5\": container with ID starting with 3697c8615d103a05fcf3bab13349506c6610f6483aab3161c85e11a1eb7d38c5 not found: ID does not exist"
Oct 30 13:24:45.235870 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.235832    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn"]
Oct 30 13:24:45.248114 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.248080    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-619b9b/test-model-619b9b-kserve-86b9bbfc-kl9pn"]
Oct 30 13:24:45.836075 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.836038    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:24:45.962595 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.962555    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-credential-socket\") pod \"d685dcc9-430c-4f84-aa00-7506306061d1\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") "
Oct 30 13:24:45.962783 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.962661    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d685dcc9-430c-4f84-aa00-7506306061d1-istio-token\" (UniqueName: \"kubernetes.io/projected/d685dcc9-430c-4f84-aa00-7506306061d1-istio-token\") pod \"d685dcc9-430c-4f84-aa00-7506306061d1\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") "
Oct 30 13:24:45.962783 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.962704    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d685dcc9-430c-4f84-aa00-7506306061d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d685dcc9-430c-4f84-aa00-7506306061d1-istio-podinfo\") pod \"d685dcc9-430c-4f84-aa00-7506306061d1\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") "
Oct 30 13:24:45.962783 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.962732    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-workload-certs\") pod \"d685dcc9-430c-4f84-aa00-7506306061d1\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") "
Oct 30 13:24:45.962783 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.962765    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d685dcc9-430c-4f84-aa00-7506306061d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d685dcc9-430c-4f84-aa00-7506306061d1-istiod-ca-cert\") pod \"d685dcc9-430c-4f84-aa00-7506306061d1\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") "
Oct 30 13:24:45.962959 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.962804    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d685dcc9-430c-4f84-aa00-7506306061d1-kube-api-access-7xmkm\" (UniqueName: \"kubernetes.io/projected/d685dcc9-430c-4f84-aa00-7506306061d1-kube-api-access-7xmkm\") pod \"d685dcc9-430c-4f84-aa00-7506306061d1\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") "
Oct 30 13:24:45.962959 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.962820    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-credential-socket" pod "d685dcc9-430c-4f84-aa00-7506306061d1" (UID: "d685dcc9-430c-4f84-aa00-7506306061d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:45.962959 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.962884    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-istio-envoy\") pod \"d685dcc9-430c-4f84-aa00-7506306061d1\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") "
Oct 30 13:24:45.963115 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.963014    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-istio-data\") pod \"d685dcc9-430c-4f84-aa00-7506306061d1\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") "
Oct 30 13:24:45.963115 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.963026    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-workload-certs" pod "d685dcc9-430c-4f84-aa00-7506306061d1" (UID: "d685dcc9-430c-4f84-aa00-7506306061d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:45.963115 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.963047    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-workload-socket\") pod \"d685dcc9-430c-4f84-aa00-7506306061d1\" (UID: \"d685dcc9-430c-4f84-aa00-7506306061d1\") "
Oct 30 13:24:45.963335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.963163    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d685dcc9-430c-4f84-aa00-7506306061d1-istiod-ca-cert" pod "d685dcc9-430c-4f84-aa00-7506306061d1" (UID: "d685dcc9-430c-4f84-aa00-7506306061d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:45.963335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.963303    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-istio-data" pod "d685dcc9-430c-4f84-aa00-7506306061d1" (UID: "d685dcc9-430c-4f84-aa00-7506306061d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:45.963457 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.963376    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-workload-socket" pod "d685dcc9-430c-4f84-aa00-7506306061d1" (UID: "d685dcc9-430c-4f84-aa00-7506306061d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:45.963758 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.963736    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:45.963835 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.963762    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:45.963835 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.963776    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d685dcc9-430c-4f84-aa00-7506306061d1-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:45.963835 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.963788    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:45.963835 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.963801    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:45.965139 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.965112    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d685dcc9-430c-4f84-aa00-7506306061d1-istio-podinfo" pod "d685dcc9-430c-4f84-aa00-7506306061d1" (UID: "d685dcc9-430c-4f84-aa00-7506306061d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:24:45.965296 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.965160    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d685dcc9-430c-4f84-aa00-7506306061d1-istio-token" pod "d685dcc9-430c-4f84-aa00-7506306061d1" (UID: "d685dcc9-430c-4f84-aa00-7506306061d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:45.965296 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.965274    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d685dcc9-430c-4f84-aa00-7506306061d1-kube-api-access-7xmkm" pod "d685dcc9-430c-4f84-aa00-7506306061d1" (UID: "d685dcc9-430c-4f84-aa00-7506306061d1"). InnerVolumeSpecName "kube-api-access-7xmkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:45.965571 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:45.965551    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-istio-envoy" pod "d685dcc9-430c-4f84-aa00-7506306061d1" (UID: "d685dcc9-430c-4f84-aa00-7506306061d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:46.067245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:46.066809    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d685dcc9-430c-4f84-aa00-7506306061d1-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:46.067245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:46.066849    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d685dcc9-430c-4f84-aa00-7506306061d1-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:46.067245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:46.066863    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d685dcc9-430c-4f84-aa00-7506306061d1-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:46.067245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:46.066881    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xmkm\" (UniqueName: \"kubernetes.io/projected/d685dcc9-430c-4f84-aa00-7506306061d1-kube-api-access-7xmkm\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:46.224376 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:46.224280    4544 generic.go:356] "Generic (PLEG): container finished" podID="d685dcc9-430c-4f84-aa00-7506306061d1" containerID="a684ee71634864b805b6ef5845f903152c480e87f7342fa9edd32012cffe3ac3" exitCode=0
Oct 30 13:24:46.224376 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:46.224351    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"
Oct 30 13:24:46.224571 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:46.224515    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f" event={"ID":"d685dcc9-430c-4f84-aa00-7506306061d1","Type":"ContainerDied","Data":"a684ee71634864b805b6ef5845f903152c480e87f7342fa9edd32012cffe3ac3"}
Oct 30 13:24:46.224571 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:46.224565    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f" event={"ID":"d685dcc9-430c-4f84-aa00-7506306061d1","Type":"ContainerDied","Data":"f2a62dd5274fb81f34792a24f26519f9b17c9e4cdd4ca06473b4ed116e424f5c"}
Oct 30 13:24:46.224642 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:46.224597    4544 scope.go:122] "RemoveContainer" containerID="a684ee71634864b805b6ef5845f903152c480e87f7342fa9edd32012cffe3ac3"
Oct 30 13:24:46.236751 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:46.236727    4544 scope.go:122] "RemoveContainer" containerID="b61ba5361033a17783ae4e7598056be0f5835e19e3ede4aae9c71d67d1bac2a0"
Oct 30 13:24:46.248395 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:46.248373    4544 scope.go:122] "RemoveContainer" containerID="a684ee71634864b805b6ef5845f903152c480e87f7342fa9edd32012cffe3ac3"
Oct 30 13:24:46.248753 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:24:46.248731    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a684ee71634864b805b6ef5845f903152c480e87f7342fa9edd32012cffe3ac3\": container with ID starting with a684ee71634864b805b6ef5845f903152c480e87f7342fa9edd32012cffe3ac3 not found: ID does not exist" containerID="a684ee71634864b805b6ef5845f903152c480e87f7342fa9edd32012cffe3ac3"
Oct 30 13:24:46.248839 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:46.248767    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a684ee71634864b805b6ef5845f903152c480e87f7342fa9edd32012cffe3ac3"} err="failed to get container status \"a684ee71634864b805b6ef5845f903152c480e87f7342fa9edd32012cffe3ac3\": rpc error: code = NotFound desc = could not find container \"a684ee71634864b805b6ef5845f903152c480e87f7342fa9edd32012cffe3ac3\": container with ID starting with a684ee71634864b805b6ef5845f903152c480e87f7342fa9edd32012cffe3ac3 not found: ID does not exist"
Oct 30 13:24:46.248839 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:46.248793    4544 scope.go:122] "RemoveContainer" containerID="b61ba5361033a17783ae4e7598056be0f5835e19e3ede4aae9c71d67d1bac2a0"
Oct 30 13:24:46.249074 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:24:46.249055    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b61ba5361033a17783ae4e7598056be0f5835e19e3ede4aae9c71d67d1bac2a0\": container with ID starting with b61ba5361033a17783ae4e7598056be0f5835e19e3ede4aae9c71d67d1bac2a0 not found: ID does not exist" containerID="b61ba5361033a17783ae4e7598056be0f5835e19e3ede4aae9c71d67d1bac2a0"
Oct 30 13:24:46.249126 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:46.249080    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b61ba5361033a17783ae4e7598056be0f5835e19e3ede4aae9c71d67d1bac2a0"} err="failed to get container status \"b61ba5361033a17783ae4e7598056be0f5835e19e3ede4aae9c71d67d1bac2a0\": rpc error: code = NotFound desc = could not find container \"b61ba5361033a17783ae4e7598056be0f5835e19e3ede4aae9c71d67d1bac2a0\": container with ID starting with b61ba5361033a17783ae4e7598056be0f5835e19e3ede4aae9c71d67d1bac2a0 not found: ID does not exist"
Oct 30 13:24:46.282855 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:46.282819    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"]
Oct 30 13:24:46.293266 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:46.293167    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f"]
Oct 30 13:24:46.694108 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:46.693938    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:47.193986 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:47.193942    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d685dcc9-430c-4f84-aa00-7506306061d1" path="/var/lib/kubelet/pods/d685dcc9-430c-4f84-aa00-7506306061d1/volumes"
Oct 30 13:24:47.194847 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:47.194820    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecb9846d-c5b3-4d4d-8243-4169400da4ca" path="/var/lib/kubelet/pods/ecb9846d-c5b3-4d4d-8243-4169400da4ca/volumes"
Oct 30 13:24:48.499142 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:48.499105    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2366f66d-znpvm"]
Oct 30 13:24:48.499924 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:48.499906    4544 cpu_manager.go:414] "containerMap: removing container" podUID="d685dcc9-430c-4f84-aa00-7506306061d1" containerName="istio-proxy"
Oct 30 13:24:48.500000 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:48.499928    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d685dcc9-430c-4f84-aa00-7506306061d1" containerName="istio-proxy"
Oct 30 13:24:48.500000 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:48.499979    4544 cpu_manager.go:414] "containerMap: removing container" podUID="d9bb4805-2bd1-4cb0-9b36-200c9146dbc9" containerName="revoke-keys"
Oct 30 13:24:48.500000 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:48.499989    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9bb4805-2bd1-4cb0-9b36-200c9146dbc9" containerName="revoke-keys"
Oct 30 13:24:48.500164 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:48.500064    4544 cpu_manager.go:414] "containerMap: removing container" podUID="ecb9846d-c5b3-4d4d-8243-4169400da4ca" containerName="main"
Oct 30 13:24:48.500164 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:48.500071    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecb9846d-c5b3-4d4d-8243-4169400da4ca" containerName="main"
Oct 30 13:24:48.500257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:48.500237    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="d685dcc9-430c-4f84-aa00-7506306061d1" containerName="istio-proxy"
Oct 30 13:24:48.500257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:48.500250    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="d685dcc9-430c-4f84-aa00-7506306061d1" containerName="istio-proxy"
Oct 30 13:24:48.500318 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:48.500260    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9bb4805-2bd1-4cb0-9b36-200c9146dbc9" containerName="revoke-keys"
Oct 30 13:24:48.500318 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:48.500273    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecb9846d-c5b3-4d4d-8243-4169400da4ca" containerName="main"
Oct 30 13:24:48.505704 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:48.505684    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2366f66d-znpvm"
Oct 30 13:24:48.538069 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:48.538026    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:24:48.538348 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:48.538101    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:24:48.544417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:48.544376    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2366f66d-znpvm"]
Oct 30 13:24:48.642654 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:48.642617    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2366f66d-znpvm\" (UID: \"2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2366f66d-znpvm"
Oct 30 13:24:48.642903 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:48.642879    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2366f66d-znpvm\" (UID: \"2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2366f66d-znpvm"
Oct 30 13:24:48.642980 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:48.642927    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv94v\" (UniqueName: \"kubernetes.io/projected/2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f-kube-api-access-rv94v\") pod \"maas-api-revoke-sub-2366f66d-znpvm\" (UID: \"2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2366f66d-znpvm"
Oct 30 13:24:48.732933 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:48.732902    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:48.743928 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:48.743894    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2366f66d-znpvm\" (UID: \"2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2366f66d-znpvm"
Oct 30 13:24:48.744108 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:48.743938    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv94v\" (UniqueName: \"kubernetes.io/projected/2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f-kube-api-access-rv94v\") pod \"maas-api-revoke-sub-2366f66d-znpvm\" (UID: \"2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2366f66d-znpvm"
Oct 30 13:24:48.744108 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:48.744063    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2366f66d-znpvm\" (UID: \"2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2366f66d-znpvm"
Oct 30 13:24:48.744674 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:48.744651    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2366f66d-znpvm\" (UID: \"2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2366f66d-znpvm"
Oct 30 13:24:48.746789 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:48.746759    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2366f66d-znpvm\" (UID: \"2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2366f66d-znpvm"
Oct 30 13:24:48.797515 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:48.797419    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv94v\" (UniqueName: \"kubernetes.io/projected/2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f-kube-api-access-rv94v\") pod \"maas-api-revoke-sub-2366f66d-znpvm\" (UID: \"2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2366f66d-znpvm"
Oct 30 13:24:48.830248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:48.826487    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2366f66d-znpvm"
Oct 30 13:24:49.144505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:49.144479    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2366f66d-znpvm"]
Oct 30 13:24:49.146365 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:24:49.146333    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b26e4dc_4156_49e8_aa8b_dbfab9b4ec6f.slice/crio-39a6f74e96320eed462ab9fc99cfa0c7d9a2308c6667eec17ee6ca9eac71a3fc WatchSource:0}: Error finding container 39a6f74e96320eed462ab9fc99cfa0c7d9a2308c6667eec17ee6ca9eac71a3fc: Status 404 returned error can't find the container with id 39a6f74e96320eed462ab9fc99cfa0c7d9a2308c6667eec17ee6ca9eac71a3fc
Oct 30 13:24:49.265369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:49.265333    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2366f66d-znpvm" event={"ID":"2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f","Type":"ContainerStarted","Data":"39a6f74e96320eed462ab9fc99cfa0c7d9a2308c6667eec17ee6ca9eac71a3fc"}
Oct 30 13:24:50.276531 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:50.276440    4544 generic.go:356] "Generic (PLEG): container finished" podID="2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f" containerID="f62d0b6957524f6007ae278eaa650a014668a974592c82a49ffae083a801bbe4" exitCode=0
Oct 30 13:24:50.277017 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:50.276525    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2366f66d-znpvm" event={"ID":"2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f","Type":"ContainerDied","Data":"f62d0b6957524f6007ae278eaa650a014668a974592c82a49ffae083a801bbe4"}
Oct 30 13:24:50.594645 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:50.594601    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:51.472619 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:51.472586    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2366f66d-znpvm"
Oct 30 13:24:51.489471 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:51.489436    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f-maas-api-xks-ca\") pod \"2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f\" (UID: \"2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f\") "
Oct 30 13:24:51.491633 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:51.491600    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f-maas-api-xks-ca" pod "2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f" (UID: "2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:24:51.590941 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:51.590897    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f-kube-api-access-rv94v\" (UniqueName: \"kubernetes.io/projected/2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f-kube-api-access-rv94v\") pod \"2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f\" (UID: \"2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f\") "
Oct 30 13:24:51.591136 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:51.591029    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f-maas-api-openshift-ca\") pod \"2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f\" (UID: \"2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f\") "
Oct 30 13:24:51.591501 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:51.591472    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f-maas-api-openshift-ca" pod "2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f" (UID: "2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:51.591825 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:51.591809    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:51.591880 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:51.591828    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:51.593266 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:51.593216    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f-kube-api-access-rv94v" pod "2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f" (UID: "2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f"). InnerVolumeSpecName "kube-api-access-rv94v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:51.699298 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:51.699253    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv94v\" (UniqueName: \"kubernetes.io/projected/2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f-kube-api-access-rv94v\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:24:51.764386 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:51.764342    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:51.920656 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:51.920552    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:52.343545 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:52.343506    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2366f66d-znpvm" event={"ID":"2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f","Type":"ContainerDied","Data":"39a6f74e96320eed462ab9fc99cfa0c7d9a2308c6667eec17ee6ca9eac71a3fc"}
Oct 30 13:24:52.344112 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:52.344086    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39a6f74e96320eed462ab9fc99cfa0c7d9a2308c6667eec17ee6ca9eac71a3fc"
Oct 30 13:24:52.344280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:52.343864    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2366f66d-znpvm"
Oct 30 13:24:52.402898 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:52.402864    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:54.407463 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:54.407429    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:56.151962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:56.151920    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:56.460286 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:56.460185    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:57.039597 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:57.039559    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4eabbb8-bbncg"]
Oct 30 13:24:57.049213 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:57.049176    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4eabbb8-bbncg"]
Oct 30 13:24:57.090349 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:57.090311    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb" path="/var/lib/kubelet/pods/64a5d1f0-bcfe-4bf9-bb2f-791a3180b3fb/volumes"
Oct 30 13:24:57.863032 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:57.862985    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:24:58.842376 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:58.842328    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0557492-99fxd"]
Oct 30 13:24:58.843309 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:58.843212    4544 cpu_manager.go:414] "containerMap: removing container" podUID="d685dcc9-430c-4f84-aa00-7506306061d1" containerName="istio-proxy"
Oct 30 13:24:58.843309 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:58.843256    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d685dcc9-430c-4f84-aa00-7506306061d1" containerName="istio-proxy"
Oct 30 13:24:58.843309 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:58.843279    4544 cpu_manager.go:414] "containerMap: removing container" podUID="2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f" containerName="revoke-keys"
Oct 30 13:24:58.843309 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:58.843287    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f" containerName="revoke-keys"
Oct 30 13:24:58.843558 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:58.843472    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f" containerName="revoke-keys"
Oct 30 13:24:58.848841 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:58.848814    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0557492-99fxd"
Oct 30 13:24:58.867805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:58.867610    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0557492-99fxd"]
Oct 30 13:24:58.874398 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:58.874367    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:24:58.874592 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:58.874430    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:24:58.947092 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:58.947049    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2336220-cd50-48ec-8dfa-18fec98bcd39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0557492-99fxd\" (UID: \"e2336220-cd50-48ec-8dfa-18fec98bcd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0557492-99fxd"
Oct 30 13:24:58.947420 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:58.947397    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9twc\" (UniqueName: \"kubernetes.io/projected/e2336220-cd50-48ec-8dfa-18fec98bcd39-kube-api-access-w9twc\") pod \"maas-api-revoke-sub-e0557492-99fxd\" (UID: \"e2336220-cd50-48ec-8dfa-18fec98bcd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0557492-99fxd"
Oct 30 13:24:58.947522 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:58.947442    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2336220-cd50-48ec-8dfa-18fec98bcd39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0557492-99fxd\" (UID: \"e2336220-cd50-48ec-8dfa-18fec98bcd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0557492-99fxd"
Oct 30 13:24:59.009597 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:24:59.009555    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:24:59.048874 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:59.048832    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9twc\" (UniqueName: \"kubernetes.io/projected/e2336220-cd50-48ec-8dfa-18fec98bcd39-kube-api-access-w9twc\") pod \"maas-api-revoke-sub-e0557492-99fxd\" (UID: \"e2336220-cd50-48ec-8dfa-18fec98bcd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0557492-99fxd"
Oct 30 13:24:59.048874 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:59.048878    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2336220-cd50-48ec-8dfa-18fec98bcd39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0557492-99fxd\" (UID: \"e2336220-cd50-48ec-8dfa-18fec98bcd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0557492-99fxd"
Oct 30 13:24:59.049153 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:59.048927    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2336220-cd50-48ec-8dfa-18fec98bcd39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0557492-99fxd\" (UID: \"e2336220-cd50-48ec-8dfa-18fec98bcd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0557492-99fxd"
Oct 30 13:24:59.049689 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:59.049657    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2336220-cd50-48ec-8dfa-18fec98bcd39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0557492-99fxd\" (UID: \"e2336220-cd50-48ec-8dfa-18fec98bcd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0557492-99fxd"
Oct 30 13:24:59.051512 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:59.051487    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2336220-cd50-48ec-8dfa-18fec98bcd39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0557492-99fxd\" (UID: \"e2336220-cd50-48ec-8dfa-18fec98bcd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0557492-99fxd"
Oct 30 13:24:59.072561 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:59.072529    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9twc\" (UniqueName: \"kubernetes.io/projected/e2336220-cd50-48ec-8dfa-18fec98bcd39-kube-api-access-w9twc\") pod \"maas-api-revoke-sub-e0557492-99fxd\" (UID: \"e2336220-cd50-48ec-8dfa-18fec98bcd39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0557492-99fxd"
Oct 30 13:24:59.165033 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:59.164942    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0557492-99fxd"
Oct 30 13:24:59.474860 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:59.474829    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0557492-99fxd"]
Oct 30 13:24:59.475966 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:24:59.475934    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2336220_cd50_48ec_8dfa_18fec98bcd39.slice/crio-58186823f8aa7bb90cb0751c05e1c2ff3735cb1bb99dd6ca5980f47a248376ef WatchSource:0}: Error finding container 58186823f8aa7bb90cb0751c05e1c2ff3735cb1bb99dd6ca5980f47a248376ef: Status 404 returned error can't find the container with id 58186823f8aa7bb90cb0751c05e1c2ff3735cb1bb99dd6ca5980f47a248376ef
Oct 30 13:24:59.634340 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:24:59.634308    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:00.415738 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:00.415702    4544 generic.go:356] "Generic (PLEG): container finished" podID="e2336220-cd50-48ec-8dfa-18fec98bcd39" containerID="b9ea0c17d77c80c3b8d423bbb510ec4c4b8101feea9db52cfd2997fb3d61091f" exitCode=0
Oct 30 13:25:00.416169 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:00.415792    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0557492-99fxd" event={"ID":"e2336220-cd50-48ec-8dfa-18fec98bcd39","Type":"ContainerDied","Data":"b9ea0c17d77c80c3b8d423bbb510ec4c4b8101feea9db52cfd2997fb3d61091f"}
Oct 30 13:25:00.416169 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:00.415853    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0557492-99fxd" event={"ID":"e2336220-cd50-48ec-8dfa-18fec98bcd39","Type":"ContainerStarted","Data":"58186823f8aa7bb90cb0751c05e1c2ff3735cb1bb99dd6ca5980f47a248376ef"}
Oct 30 13:25:01.172004 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:01.171967    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:01.872560 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:01.872532    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0557492-99fxd"
Oct 30 13:25:01.922727 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:01.922690    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2336220-cd50-48ec-8dfa-18fec98bcd39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2336220-cd50-48ec-8dfa-18fec98bcd39-maas-api-openshift-ca\") pod \"e2336220-cd50-48ec-8dfa-18fec98bcd39\" (UID: \"e2336220-cd50-48ec-8dfa-18fec98bcd39\") "
Oct 30 13:25:01.922948 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:01.922781    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2336220-cd50-48ec-8dfa-18fec98bcd39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2336220-cd50-48ec-8dfa-18fec98bcd39-maas-api-xks-ca\") pod \"e2336220-cd50-48ec-8dfa-18fec98bcd39\" (UID: \"e2336220-cd50-48ec-8dfa-18fec98bcd39\") "
Oct 30 13:25:01.922948 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:01.922814    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2336220-cd50-48ec-8dfa-18fec98bcd39-kube-api-access-w9twc\" (UniqueName: \"kubernetes.io/projected/e2336220-cd50-48ec-8dfa-18fec98bcd39-kube-api-access-w9twc\") pod \"e2336220-cd50-48ec-8dfa-18fec98bcd39\" (UID: \"e2336220-cd50-48ec-8dfa-18fec98bcd39\") "
Oct 30 13:25:01.923638 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:01.923600    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2336220-cd50-48ec-8dfa-18fec98bcd39-maas-api-openshift-ca" pod "e2336220-cd50-48ec-8dfa-18fec98bcd39" (UID: "e2336220-cd50-48ec-8dfa-18fec98bcd39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:01.925353 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:01.925326    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2336220-cd50-48ec-8dfa-18fec98bcd39-kube-api-access-w9twc" pod "e2336220-cd50-48ec-8dfa-18fec98bcd39" (UID: "e2336220-cd50-48ec-8dfa-18fec98bcd39"). InnerVolumeSpecName "kube-api-access-w9twc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:01.926026 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:01.926005    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2336220-cd50-48ec-8dfa-18fec98bcd39-maas-api-xks-ca" pod "e2336220-cd50-48ec-8dfa-18fec98bcd39" (UID: "e2336220-cd50-48ec-8dfa-18fec98bcd39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:25:01.970364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:01.970326    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:02.026279 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:02.026157    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2336220-cd50-48ec-8dfa-18fec98bcd39-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:02.026518 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:02.026497    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2336220-cd50-48ec-8dfa-18fec98bcd39-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:02.026641 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:02.026615    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9twc\" (UniqueName: \"kubernetes.io/projected/e2336220-cd50-48ec-8dfa-18fec98bcd39-kube-api-access-w9twc\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:02.033577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:02.033541    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd"]
Oct 30 13:25:02.034409 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:02.034389    4544 cpu_manager.go:414] "containerMap: removing container" podUID="e2336220-cd50-48ec-8dfa-18fec98bcd39" containerName="revoke-keys"
Oct 30 13:25:02.034409 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:02.034409    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2336220-cd50-48ec-8dfa-18fec98bcd39" containerName="revoke-keys"
Oct 30 13:25:02.034569 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:02.034525    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2336220-cd50-48ec-8dfa-18fec98bcd39" containerName="revoke-keys"
Oct 30 13:25:02.038559 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:02.038532    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd"
Oct 30 13:25:02.082410 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:02.082379    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-51288b-serving-cert\""
Oct 30 13:25:02.100242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:02.100189    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd"]
Oct 30 13:25:02.128873 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:02.128832    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhdzq\" (UniqueName: \"kubernetes.io/projected/3253ffac-6c08-4f4c-a3e1-c0bc6debf82a-kube-api-access-xhdzq\") pod \"maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd\" (UID: \"3253ffac-6c08-4f4c-a3e1-c0bc6debf82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd"
Oct 30 13:25:02.129100 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:02.128918    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3253ffac-6c08-4f4c-a3e1-c0bc6debf82a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd\" (UID: \"3253ffac-6c08-4f4c-a3e1-c0bc6debf82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd"
Oct 30 13:25:02.129183 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:02.129089    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3253ffac-6c08-4f4c-a3e1-c0bc6debf82a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd\" (UID: \"3253ffac-6c08-4f4c-a3e1-c0bc6debf82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd"
Oct 30 13:25:02.129333 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:02.129309    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3253ffac-6c08-4f4c-a3e1-c0bc6debf82a-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd\" (UID: \"3253ffac-6c08-4f4c-a3e1-c0bc6debf82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd"
Oct 30 13:25:02.198757 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:02.198718    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-618c72-zczws"]
Oct 30 13:25:02.207435 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:02.207401    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-618c72-zczws"]
Oct 30 13:25:02.231303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:02.231266    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3253ffac-6c08-4f4c-a3e1-c0bc6debf82a-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd\" (UID: \"3253ffac-6c08-4f4c-a3e1-c0bc6debf82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd"
Oct 30 13:25:02.231479 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:02.231403    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhdzq\" (UniqueName: \"kubernetes.io/projected/3253ffac-6c08-4f4c-a3e1-c0bc6debf82a-kube-api-access-xhdzq\") pod \"maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd\" (UID: \"3253ffac-6c08-4f4c-a3e1-c0bc6debf82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd"
Oct 30 13:25:02.231584 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:02.231478    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3253ffac-6c08-4f4c-a3e1-c0bc6debf82a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd\" (UID: \"3253ffac-6c08-4f4c-a3e1-c0bc6debf82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd"
Oct 30 13:25:02.231645 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:02.231590    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3253ffac-6c08-4f4c-a3e1-c0bc6debf82a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd\" (UID: \"3253ffac-6c08-4f4c-a3e1-c0bc6debf82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd"
Oct 30 13:25:02.232484 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:02.232450    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3253ffac-6c08-4f4c-a3e1-c0bc6debf82a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd\" (UID: \"3253ffac-6c08-4f4c-a3e1-c0bc6debf82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd"
Oct 30 13:25:02.234395 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:02.234370    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3253ffac-6c08-4f4c-a3e1-c0bc6debf82a-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd\" (UID: \"3253ffac-6c08-4f4c-a3e1-c0bc6debf82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd"
Oct 30 13:25:02.238028 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:02.238000    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3253ffac-6c08-4f4c-a3e1-c0bc6debf82a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd\" (UID: \"3253ffac-6c08-4f4c-a3e1-c0bc6debf82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd"
Oct 30 13:25:02.251647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:02.251602    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:02.255236 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:02.255193    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhdzq\" (UniqueName: \"kubernetes.io/projected/3253ffac-6c08-4f4c-a3e1-c0bc6debf82a-kube-api-access-xhdzq\") pod \"maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd\" (UID: \"3253ffac-6c08-4f4c-a3e1-c0bc6debf82a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd"
Oct 30 13:25:02.362247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:02.359560    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd"
Oct 30 13:25:02.412216 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:02.412190    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:02.467484 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:02.466561    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0557492-99fxd" event={"ID":"e2336220-cd50-48ec-8dfa-18fec98bcd39","Type":"ContainerDied","Data":"58186823f8aa7bb90cb0751c05e1c2ff3735cb1bb99dd6ca5980f47a248376ef"}
Oct 30 13:25:02.467484 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:02.466630    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58186823f8aa7bb90cb0751c05e1c2ff3735cb1bb99dd6ca5980f47a248376ef"
Oct 30 13:25:02.467484 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:02.466723    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0557492-99fxd"
Oct 30 13:25:02.695509 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:02.695392    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:02.718434 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:02.718397    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd"]
Oct 30 13:25:02.718756 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:25:02.718728    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3253ffac_6c08_4f4c_a3e1_c0bc6debf82a.slice/crio-fb7f08bcc2b0785761bf28ab242a9e2c18ea3f9d68ec99f29d23452136bda8b9 WatchSource:0}: Error finding container fb7f08bcc2b0785761bf28ab242a9e2c18ea3f9d68ec99f29d23452136bda8b9: Status 404 returned error can't find the container with id fb7f08bcc2b0785761bf28ab242a9e2c18ea3f9d68ec99f29d23452136bda8b9
Oct 30 13:25:02.720507 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:02.720491    4544 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:25:03.117235 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:03.117194    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4adbe0d-49ae-4f5a-9cfa-449a4da119d5" path="/var/lib/kubelet/pods/e4adbe0d-49ae-4f5a-9cfa-449a4da119d5/volumes"
Oct 30 13:25:03.477153 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:03.477047    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd" event={"ID":"3253ffac-6c08-4f4c-a3e1-c0bc6debf82a","Type":"ContainerStarted","Data":"850a7c7b7f1eb6eab2c1e52810d698dd07cec94ab6c4908996850977d3215656"}
Oct 30 13:25:03.477153 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:03.477133    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd" event={"ID":"3253ffac-6c08-4f4c-a3e1-c0bc6debf82a","Type":"ContainerStarted","Data":"fb7f08bcc2b0785761bf28ab242a9e2c18ea3f9d68ec99f29d23452136bda8b9"}
Oct 30 13:25:03.477422 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:03.477302    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd"
Oct 30 13:25:04.207540 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:04.207504    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:06.172920 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:06.172883    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:08.007022 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:08.006982    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:09.492469 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:09.492439    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd"
Oct 30 13:25:09.539842 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:09.539789    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd" podStartSLOduration=8.092323787 podStartE2EDuration="8.539773788s" totalImagesPullingTime="447.450001ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:03.527236701 +0000 UTC m=+2398.574370655" watchObservedRunningTime="2026-10-30 13:25:09.539773788 +0000 UTC m=+2404.586907750"
Oct 30 13:25:09.610698 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:09.610658    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:09.904427 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:09.904391    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"]
Oct 30 13:25:09.904929 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:09.904896    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv" podUID="c84b2e67-2800-4616-a136-a0912e553b55" containerName="istio-proxy" containerID="cri-o://61a94be2ddf37f7258e9fe8fdcdc0ae7f03feb95f7757778113d05cf9ae5b6d8" gracePeriod=30
Oct 30 13:25:10.482574 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:10.482541    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-619b9b-qjfvk"]
Oct 30 13:25:10.487629 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:10.487606    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-619b9b-qjfvk"
Oct 30 13:25:10.516716 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:10.516683    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-619b9b-qjfvk"]
Oct 30 13:25:10.532355 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:10.532309    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:25:10.532553 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:10.532369    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:25:10.610787 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:10.610748    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ffdd227-1389-4c92-a0e2-2e6db870d4bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-619b9b-qjfvk\" (UID: \"1ffdd227-1389-4c92-a0e2-2e6db870d4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-619b9b-qjfvk"
Oct 30 13:25:10.610959 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:10.610819    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ffdd227-1389-4c92-a0e2-2e6db870d4bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-619b9b-qjfvk\" (UID: \"1ffdd227-1389-4c92-a0e2-2e6db870d4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-619b9b-qjfvk"
Oct 30 13:25:10.611061 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:10.611047    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4w4l\" (UniqueName: \"kubernetes.io/projected/1ffdd227-1389-4c92-a0e2-2e6db870d4bb-kube-api-access-q4w4l\") pod \"maas-api-revoke-keys-e2e-inf-b-619b9b-qjfvk\" (UID: \"1ffdd227-1389-4c92-a0e2-2e6db870d4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-619b9b-qjfvk"
Oct 30 13:25:10.677363 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:10.677317    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:10.712328 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:10.712290    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ffdd227-1389-4c92-a0e2-2e6db870d4bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-619b9b-qjfvk\" (UID: \"1ffdd227-1389-4c92-a0e2-2e6db870d4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-619b9b-qjfvk"
Oct 30 13:25:10.712529 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:10.712362    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ffdd227-1389-4c92-a0e2-2e6db870d4bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-619b9b-qjfvk\" (UID: \"1ffdd227-1389-4c92-a0e2-2e6db870d4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-619b9b-qjfvk"
Oct 30 13:25:10.712612 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:10.712595    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4w4l\" (UniqueName: \"kubernetes.io/projected/1ffdd227-1389-4c92-a0e2-2e6db870d4bb-kube-api-access-q4w4l\") pod \"maas-api-revoke-keys-e2e-inf-b-619b9b-qjfvk\" (UID: \"1ffdd227-1389-4c92-a0e2-2e6db870d4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-619b9b-qjfvk"
Oct 30 13:25:10.713166 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:10.713145    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ffdd227-1389-4c92-a0e2-2e6db870d4bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-619b9b-qjfvk\" (UID: \"1ffdd227-1389-4c92-a0e2-2e6db870d4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-619b9b-qjfvk"
Oct 30 13:25:10.714968 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:10.714940    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ffdd227-1389-4c92-a0e2-2e6db870d4bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-619b9b-qjfvk\" (UID: \"1ffdd227-1389-4c92-a0e2-2e6db870d4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-619b9b-qjfvk"
Oct 30 13:25:10.736078 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:10.736014    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4w4l\" (UniqueName: \"kubernetes.io/projected/1ffdd227-1389-4c92-a0e2-2e6db870d4bb-kube-api-access-q4w4l\") pod \"maas-api-revoke-keys-e2e-inf-b-619b9b-qjfvk\" (UID: \"1ffdd227-1389-4c92-a0e2-2e6db870d4bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-619b9b-qjfvk"
Oct 30 13:25:10.802968 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:10.802940    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-619b9b-qjfvk"
Oct 30 13:25:11.120744 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:11.120712    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-619b9b-qjfvk"]
Oct 30 13:25:11.120870 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:25:11.120712    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ffdd227_1389_4c92_a0e2_2e6db870d4bb.slice/crio-e6148dfe8895414014b4db23467fdc146fdcb5efbf50d3784770dc34c4988a50 WatchSource:0}: Error finding container e6148dfe8895414014b4db23467fdc146fdcb5efbf50d3784770dc34c4988a50: Status 404 returned error can't find the container with id e6148dfe8895414014b4db23467fdc146fdcb5efbf50d3784770dc34c4988a50
Oct 30 13:25:11.286834 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:11.286799    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:11.409298 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:25:11.405636    4544 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ffdd227_1389_4c92_a0e2_2e6db870d4bb.slice/crio-bc0143cd624a916ae8df24651d937acd4a25f27d512f7b29425d16ef1c75ab90.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 13:25:11.555059 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:11.555023    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-619b9b-qjfvk" event={"ID":"1ffdd227-1389-4c92-a0e2-2e6db870d4bb","Type":"ContainerStarted","Data":"e6148dfe8895414014b4db23467fdc146fdcb5efbf50d3784770dc34c4988a50"}
Oct 30 13:25:11.807215 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:11.807107    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:12.567700 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:12.567606    4544 generic.go:356] "Generic (PLEG): container finished" podID="1ffdd227-1389-4c92-a0e2-2e6db870d4bb" containerID="bc0143cd624a916ae8df24651d937acd4a25f27d512f7b29425d16ef1c75ab90" exitCode=0
Oct 30 13:25:12.568149 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:12.567694    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-619b9b-qjfvk" event={"ID":"1ffdd227-1389-4c92-a0e2-2e6db870d4bb","Type":"ContainerDied","Data":"bc0143cd624a916ae8df24651d937acd4a25f27d512f7b29425d16ef1c75ab90"}
Oct 30 13:25:13.745698 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:13.745666    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-619b9b-qjfvk"
Oct 30 13:25:13.770311 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:13.770206    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:13.863809 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:13.863768    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ffdd227-1389-4c92-a0e2-2e6db870d4bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ffdd227-1389-4c92-a0e2-2e6db870d4bb-maas-api-openshift-ca\") pod \"1ffdd227-1389-4c92-a0e2-2e6db870d4bb\" (UID: \"1ffdd227-1389-4c92-a0e2-2e6db870d4bb\") "
Oct 30 13:25:13.864020 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:13.863864    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ffdd227-1389-4c92-a0e2-2e6db870d4bb-kube-api-access-q4w4l\" (UniqueName: \"kubernetes.io/projected/1ffdd227-1389-4c92-a0e2-2e6db870d4bb-kube-api-access-q4w4l\") pod \"1ffdd227-1389-4c92-a0e2-2e6db870d4bb\" (UID: \"1ffdd227-1389-4c92-a0e2-2e6db870d4bb\") "
Oct 30 13:25:13.864020 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:13.863973    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ffdd227-1389-4c92-a0e2-2e6db870d4bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ffdd227-1389-4c92-a0e2-2e6db870d4bb-maas-api-xks-ca\") pod \"1ffdd227-1389-4c92-a0e2-2e6db870d4bb\" (UID: \"1ffdd227-1389-4c92-a0e2-2e6db870d4bb\") "
Oct 30 13:25:13.864273 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:13.864242    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ffdd227-1389-4c92-a0e2-2e6db870d4bb-maas-api-openshift-ca" pod "1ffdd227-1389-4c92-a0e2-2e6db870d4bb" (UID: "1ffdd227-1389-4c92-a0e2-2e6db870d4bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:13.864725 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:13.864699    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ffdd227-1389-4c92-a0e2-2e6db870d4bb-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:13.866434 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:13.866407    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ffdd227-1389-4c92-a0e2-2e6db870d4bb-maas-api-xks-ca" pod "1ffdd227-1389-4c92-a0e2-2e6db870d4bb" (UID: "1ffdd227-1389-4c92-a0e2-2e6db870d4bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:25:13.867713 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:13.867686    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ffdd227-1389-4c92-a0e2-2e6db870d4bb-kube-api-access-q4w4l" pod "1ffdd227-1389-4c92-a0e2-2e6db870d4bb" (UID: "1ffdd227-1389-4c92-a0e2-2e6db870d4bb"). InnerVolumeSpecName "kube-api-access-q4w4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:13.966364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:13.966325    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4w4l\" (UniqueName: \"kubernetes.io/projected/1ffdd227-1389-4c92-a0e2-2e6db870d4bb-kube-api-access-q4w4l\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:13.966364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:13.966359    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ffdd227-1389-4c92-a0e2-2e6db870d4bb-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:14.585691 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:14.585657    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-619b9b-qjfvk" event={"ID":"1ffdd227-1389-4c92-a0e2-2e6db870d4bb","Type":"ContainerDied","Data":"e6148dfe8895414014b4db23467fdc146fdcb5efbf50d3784770dc34c4988a50"}
Oct 30 13:25:14.585691 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:14.585679    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-619b9b-qjfvk"
Oct 30 13:25:14.585971 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:14.585740    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6148dfe8895414014b4db23467fdc146fdcb5efbf50d3784770dc34c4988a50"
Oct 30 13:25:15.234340 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.234307    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:25:15.284515 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.284484    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-istio-envoy\") pod \"c84b2e67-2800-4616-a136-a0912e553b55\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") "
Oct 30 13:25:15.284718 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.284542    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-workload-certs\") pod \"c84b2e67-2800-4616-a136-a0912e553b55\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") "
Oct 30 13:25:15.284718 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.284673    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-istio-data\") pod \"c84b2e67-2800-4616-a136-a0912e553b55\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") "
Oct 30 13:25:15.284852 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.284720    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c84b2e67-2800-4616-a136-a0912e553b55-kube-api-access-n4bbc\" (UniqueName: \"kubernetes.io/projected/c84b2e67-2800-4616-a136-a0912e553b55-kube-api-access-n4bbc\") pod \"c84b2e67-2800-4616-a136-a0912e553b55\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") "
Oct 30 13:25:15.284852 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.284764    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c84b2e67-2800-4616-a136-a0912e553b55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c84b2e67-2800-4616-a136-a0912e553b55-istiod-ca-cert\") pod \"c84b2e67-2800-4616-a136-a0912e553b55\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") "
Oct 30 13:25:15.284969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.284872    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-workload-socket\") pod \"c84b2e67-2800-4616-a136-a0912e553b55\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") "
Oct 30 13:25:15.285023 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.284990    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c84b2e67-2800-4616-a136-a0912e553b55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c84b2e67-2800-4616-a136-a0912e553b55-istio-podinfo\") pod \"c84b2e67-2800-4616-a136-a0912e553b55\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") "
Oct 30 13:25:15.285136 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.285092    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-credential-socket\") pod \"c84b2e67-2800-4616-a136-a0912e553b55\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") "
Oct 30 13:25:15.285199 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.285174    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c84b2e67-2800-4616-a136-a0912e553b55-istio-token\" (UniqueName: \"kubernetes.io/projected/c84b2e67-2800-4616-a136-a0912e553b55-istio-token\") pod \"c84b2e67-2800-4616-a136-a0912e553b55\" (UID: \"c84b2e67-2800-4616-a136-a0912e553b55\") "
Oct 30 13:25:15.286130 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.286078    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-istio-data" pod "c84b2e67-2800-4616-a136-a0912e553b55" (UID: "c84b2e67-2800-4616-a136-a0912e553b55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:15.286294 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.286130    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-workload-certs" pod "c84b2e67-2800-4616-a136-a0912e553b55" (UID: "c84b2e67-2800-4616-a136-a0912e553b55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:15.286294 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.286211    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-credential-socket" pod "c84b2e67-2800-4616-a136-a0912e553b55" (UID: "c84b2e67-2800-4616-a136-a0912e553b55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:15.286658 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.286611    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-workload-socket" pod "c84b2e67-2800-4616-a136-a0912e553b55" (UID: "c84b2e67-2800-4616-a136-a0912e553b55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:15.287110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.287071    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c84b2e67-2800-4616-a136-a0912e553b55-istiod-ca-cert" pod "c84b2e67-2800-4616-a136-a0912e553b55" (UID: "c84b2e67-2800-4616-a136-a0912e553b55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:15.287415 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.287395    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-istio-envoy" pod "c84b2e67-2800-4616-a136-a0912e553b55" (UID: "c84b2e67-2800-4616-a136-a0912e553b55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:15.288741 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.288715    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c84b2e67-2800-4616-a136-a0912e553b55-kube-api-access-n4bbc" pod "c84b2e67-2800-4616-a136-a0912e553b55" (UID: "c84b2e67-2800-4616-a136-a0912e553b55"). InnerVolumeSpecName "kube-api-access-n4bbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:15.289256 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.289198    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c84b2e67-2800-4616-a136-a0912e553b55-istio-token" pod "c84b2e67-2800-4616-a136-a0912e553b55" (UID: "c84b2e67-2800-4616-a136-a0912e553b55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:15.289364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.289345    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c84b2e67-2800-4616-a136-a0912e553b55-istio-podinfo" pod "c84b2e67-2800-4616-a136-a0912e553b55" (UID: "c84b2e67-2800-4616-a136-a0912e553b55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:25:15.387537 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.387495    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:15.387537 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.387528    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c84b2e67-2800-4616-a136-a0912e553b55-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:15.387537 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.387543    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:15.387805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.387556    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c84b2e67-2800-4616-a136-a0912e553b55-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:15.387805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.387567    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:15.387805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.387578    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:15.387805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.387588    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c84b2e67-2800-4616-a136-a0912e553b55-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:15.387805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.387599    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4bbc\" (UniqueName: \"kubernetes.io/projected/c84b2e67-2800-4616-a136-a0912e553b55-kube-api-access-n4bbc\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:15.387805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.387610    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c84b2e67-2800-4616-a136-a0912e553b55-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:15.598023 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.597984    4544 generic.go:356] "Generic (PLEG): container finished" podID="c84b2e67-2800-4616-a136-a0912e553b55" containerID="61a94be2ddf37f7258e9fe8fdcdc0ae7f03feb95f7757778113d05cf9ae5b6d8" exitCode=0
Oct 30 13:25:15.598265 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.598039    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"
Oct 30 13:25:15.598265 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.598072    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv" event={"ID":"c84b2e67-2800-4616-a136-a0912e553b55","Type":"ContainerDied","Data":"61a94be2ddf37f7258e9fe8fdcdc0ae7f03feb95f7757778113d05cf9ae5b6d8"}
Oct 30 13:25:15.598265 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.598137    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv" event={"ID":"c84b2e67-2800-4616-a136-a0912e553b55","Type":"ContainerDied","Data":"23aa5d295dc4da2b00fc07e2bd07363ea1da8bf0e1304994067c06534b34007c"}
Oct 30 13:25:15.598265 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.598166    4544 scope.go:122] "RemoveContainer" containerID="61a94be2ddf37f7258e9fe8fdcdc0ae7f03feb95f7757778113d05cf9ae5b6d8"
Oct 30 13:25:15.610325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.610305    4544 scope.go:122] "RemoveContainer" containerID="61a94be2ddf37f7258e9fe8fdcdc0ae7f03feb95f7757778113d05cf9ae5b6d8"
Oct 30 13:25:15.610602 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:25:15.610585    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61a94be2ddf37f7258e9fe8fdcdc0ae7f03feb95f7757778113d05cf9ae5b6d8\": container with ID starting with 61a94be2ddf37f7258e9fe8fdcdc0ae7f03feb95f7757778113d05cf9ae5b6d8 not found: ID does not exist" containerID="61a94be2ddf37f7258e9fe8fdcdc0ae7f03feb95f7757778113d05cf9ae5b6d8"
Oct 30 13:25:15.610646 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.610611    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61a94be2ddf37f7258e9fe8fdcdc0ae7f03feb95f7757778113d05cf9ae5b6d8"} err="failed to get container status \"61a94be2ddf37f7258e9fe8fdcdc0ae7f03feb95f7757778113d05cf9ae5b6d8\": rpc error: code = NotFound desc = could not find container \"61a94be2ddf37f7258e9fe8fdcdc0ae7f03feb95f7757778113d05cf9ae5b6d8\": container with ID starting with 61a94be2ddf37f7258e9fe8fdcdc0ae7f03feb95f7757778113d05cf9ae5b6d8 not found: ID does not exist"
Oct 30 13:25:15.669281 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.669246    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"]
Oct 30 13:25:15.682199 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.682170    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv"]
Oct 30 13:25:15.721287 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:15.720200    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:16.630815 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:16.630774    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:17.090586 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:17.090548    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c84b2e67-2800-4616-a136-a0912e553b55" path="/var/lib/kubelet/pods/c84b2e67-2800-4616-a136-a0912e553b55/volumes"
Oct 30 13:25:17.851388 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:17.848581    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:18.042527 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:18.042486    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9a91dcc-vprwx"]
Oct 30 13:25:18.054663 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:18.054633    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9a91dcc-vprwx"]
Oct 30 13:25:19.088027 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:19.087990    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="983f6389-65d1-4dea-b68b-4fc4d23098fe" path="/var/lib/kubelet/pods/983f6389-65d1-4dea-b68b-4fc4d23098fe/volumes"
Oct 30 13:25:19.599315 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:19.599274    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:21.125849 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:21.125813    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:21.257047 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:21.255199    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:21.480012 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:21.479851    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:23.192862 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:23.192817    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:24.993902 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:24.993855    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:26.551084 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:26.551035    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:27.218017 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:27.217973    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:27.363194 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:27.363153    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:27.365684 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:27.365654    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bec983-pljc7"]
Oct 30 13:25:27.371205 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:27.371177    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bec983-pljc7"]
Oct 30 13:25:27.489999 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:27.489888    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:27.857521 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:27.857483    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:29.009210 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:25:29.009173    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:25:29.087380 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:29.087344    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9bb4805-2bd1-4cb0-9b36-200c9146dbc9" path="/var/lib/kubelet/pods/d9bb4805-2bd1-4cb0-9b36-200c9146dbc9/volumes"
Oct 30 13:25:29.477405 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:29.477372    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:31.518615 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:31.518565    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:33.772920 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:33.772872    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:34.335136 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:34.335097    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:34.446000 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:34.445960    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:34.687695 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:34.687609    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"]
Oct 30 13:25:34.687920 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:34.687890    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m" podUID="2e32d130-6ba4-4391-bf2a-7bd6548c1c88" containerName="istio-proxy" containerID="cri-o://fe886e9d20ad0a1ea488f5d69254df6d276da05e6f3b0d107eeb10c70a8795a1" gracePeriod=30
Oct 30 13:25:34.725360 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:34.725326    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:35.782853 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:35.782810    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:37.479960 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:37.479919    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:38.301685 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:38.301653    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8f8991c-pnl9v"]
Oct 30 13:25:38.302496 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:38.302474    4544 cpu_manager.go:414] "containerMap: removing container" podUID="c84b2e67-2800-4616-a136-a0912e553b55" containerName="istio-proxy"
Oct 30 13:25:38.302587 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:38.302498    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c84b2e67-2800-4616-a136-a0912e553b55" containerName="istio-proxy"
Oct 30 13:25:38.302587 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:38.302514    4544 cpu_manager.go:414] "containerMap: removing container" podUID="1ffdd227-1389-4c92-a0e2-2e6db870d4bb" containerName="revoke-keys"
Oct 30 13:25:38.302587 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:38.302519    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ffdd227-1389-4c92-a0e2-2e6db870d4bb" containerName="revoke-keys"
Oct 30 13:25:38.302704 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:38.302652    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="c84b2e67-2800-4616-a136-a0912e553b55" containerName="istio-proxy"
Oct 30 13:25:38.302704 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:38.302660    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ffdd227-1389-4c92-a0e2-2e6db870d4bb" containerName="revoke-keys"
Oct 30 13:25:38.306719 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:38.306703    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f8991c-pnl9v"
Oct 30 13:25:38.327008 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:38.326967    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8f8991c-pnl9v"]
Oct 30 13:25:38.335431 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:38.334013    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:25:38.335431 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:38.334090    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:25:38.346045 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:38.346009    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02995b0b-f05b-40a7-b390-4c59835aad3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8f8991c-pnl9v\" (UID: \"02995b0b-f05b-40a7-b390-4c59835aad3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f8991c-pnl9v"
Oct 30 13:25:38.346201 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:38.346071    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9shd4\" (UniqueName: \"kubernetes.io/projected/02995b0b-f05b-40a7-b390-4c59835aad3b-kube-api-access-9shd4\") pod \"maas-api-revoke-sub-b8f8991c-pnl9v\" (UID: \"02995b0b-f05b-40a7-b390-4c59835aad3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f8991c-pnl9v"
Oct 30 13:25:38.346290 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:38.346233    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02995b0b-f05b-40a7-b390-4c59835aad3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8f8991c-pnl9v\" (UID: \"02995b0b-f05b-40a7-b390-4c59835aad3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f8991c-pnl9v"
Oct 30 13:25:38.447865 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:38.447829    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02995b0b-f05b-40a7-b390-4c59835aad3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8f8991c-pnl9v\" (UID: \"02995b0b-f05b-40a7-b390-4c59835aad3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f8991c-pnl9v"
Oct 30 13:25:38.448082 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:38.447871    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9shd4\" (UniqueName: \"kubernetes.io/projected/02995b0b-f05b-40a7-b390-4c59835aad3b-kube-api-access-9shd4\") pod \"maas-api-revoke-sub-b8f8991c-pnl9v\" (UID: \"02995b0b-f05b-40a7-b390-4c59835aad3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f8991c-pnl9v"
Oct 30 13:25:38.448082 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:38.447930    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02995b0b-f05b-40a7-b390-4c59835aad3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8f8991c-pnl9v\" (UID: \"02995b0b-f05b-40a7-b390-4c59835aad3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f8991c-pnl9v"
Oct 30 13:25:38.448616 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:38.448589    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02995b0b-f05b-40a7-b390-4c59835aad3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8f8991c-pnl9v\" (UID: \"02995b0b-f05b-40a7-b390-4c59835aad3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f8991c-pnl9v"
Oct 30 13:25:38.450361 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:38.450340    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02995b0b-f05b-40a7-b390-4c59835aad3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8f8991c-pnl9v\" (UID: \"02995b0b-f05b-40a7-b390-4c59835aad3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f8991c-pnl9v"
Oct 30 13:25:38.471978 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:38.471953    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9shd4\" (UniqueName: \"kubernetes.io/projected/02995b0b-f05b-40a7-b390-4c59835aad3b-kube-api-access-9shd4\") pod \"maas-api-revoke-sub-b8f8991c-pnl9v\" (UID: \"02995b0b-f05b-40a7-b390-4c59835aad3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f8991c-pnl9v"
Oct 30 13:25:38.622327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:38.622294    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f8991c-pnl9v"
Oct 30 13:25:38.675585 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:38.675549    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq"]
Oct 30 13:25:38.733625 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:38.733549    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:38.957989 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:38.957964    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8f8991c-pnl9v"]
Oct 30 13:25:38.959629 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:25:38.959606    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02995b0b_f05b_40a7_b390_4c59835aad3b.slice/crio-81bbc23e7ed3bb1f7b53f3f496d3fcbe029025f80420431e1f6c39568626f4d3 WatchSource:0}: Error finding container 81bbc23e7ed3bb1f7b53f3f496d3fcbe029025f80420431e1f6c39568626f4d3: Status 404 returned error can't find the container with id 81bbc23e7ed3bb1f7b53f3f496d3fcbe029025f80420431e1f6c39568626f4d3
Oct 30 13:25:38.993594 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:38.993566    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f8991c-pnl9v" event={"ID":"02995b0b-f05b-40a7-b390-4c59835aad3b","Type":"ContainerStarted","Data":"81bbc23e7ed3bb1f7b53f3f496d3fcbe029025f80420431e1f6c39568626f4d3"}
Oct 30 13:25:39.422525 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:39.422487    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:40.002475 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.002452    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:25:40.003195 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.003173    4544 generic.go:356] "Generic (PLEG): container finished" podID="2e32d130-6ba4-4391-bf2a-7bd6548c1c88" containerID="fe886e9d20ad0a1ea488f5d69254df6d276da05e6f3b0d107eeb10c70a8795a1" exitCode=0
Oct 30 13:25:40.003322 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.003248    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m" event={"ID":"2e32d130-6ba4-4391-bf2a-7bd6548c1c88","Type":"ContainerDied","Data":"fe886e9d20ad0a1ea488f5d69254df6d276da05e6f3b0d107eeb10c70a8795a1"}
Oct 30 13:25:40.003322 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.003295    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m" event={"ID":"2e32d130-6ba4-4391-bf2a-7bd6548c1c88","Type":"ContainerDied","Data":"0f91ca6113fa978ae028d11e067b8668840718150ac88f8da3e3aa59edc40bf7"}
Oct 30 13:25:40.003322 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.003317    4544 scope.go:122] "RemoveContainer" containerID="fe886e9d20ad0a1ea488f5d69254df6d276da05e6f3b0d107eeb10c70a8795a1"
Oct 30 13:25:40.005089 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.005066    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f8991c-pnl9v" event={"ID":"02995b0b-f05b-40a7-b390-4c59835aad3b","Type":"ContainerStarted","Data":"58dfca7d7b51d2ad7683d60b02fa6c1e37e6390795f66617421c30c830fde1cd"}
Oct 30 13:25:40.014636 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.014617    4544 scope.go:122] "RemoveContainer" containerID="a578a827da5dbae043fd6b228775c44f833308d1c2bb162eb2e7f39f47d1a53f"
Oct 30 13:25:40.080261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.075380    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istio-podinfo\") pod \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") "
Oct 30 13:25:40.080261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.075861    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-credential-socket\") pod \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") "
Oct 30 13:25:40.080261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.075916    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istio-token\" (UniqueName: \"kubernetes.io/projected/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istio-token\") pod \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") "
Oct 30 13:25:40.080261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.076671    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istio-data\") pod \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") "
Oct 30 13:25:40.080261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.076602    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-credential-socket" pod "2e32d130-6ba4-4391-bf2a-7bd6548c1c88" (UID: "2e32d130-6ba4-4391-bf2a-7bd6548c1c88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:40.080261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.076982    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-kube-api-access-w6cgx\" (UniqueName: \"kubernetes.io/projected/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-kube-api-access-w6cgx\") pod \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") "
Oct 30 13:25:40.080261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.077069    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istio-data" pod "2e32d130-6ba4-4391-bf2a-7bd6548c1c88" (UID: "2e32d130-6ba4-4391-bf2a-7bd6548c1c88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:40.080261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.077131    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-workload-certs\") pod \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") "
Oct 30 13:25:40.080261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.077189    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-workload-socket\") pod \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") "
Oct 30 13:25:40.080261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.077245    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istiod-ca-cert\") pod \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") "
Oct 30 13:25:40.080261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.077582    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-workload-socket" pod "2e32d130-6ba4-4391-bf2a-7bd6548c1c88" (UID: "2e32d130-6ba4-4391-bf2a-7bd6548c1c88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:40.080261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.077780    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-workload-certs" pod "2e32d130-6ba4-4391-bf2a-7bd6548c1c88" (UID: "2e32d130-6ba4-4391-bf2a-7bd6548c1c88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:40.080261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.078076    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istiod-ca-cert" pod "2e32d130-6ba4-4391-bf2a-7bd6548c1c88" (UID: "2e32d130-6ba4-4391-bf2a-7bd6548c1c88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:40.080261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.077332    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istio-envoy\") pod \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\" (UID: \"2e32d130-6ba4-4391-bf2a-7bd6548c1c88\") "
Oct 30 13:25:40.080261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.079041    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istio-token" pod "2e32d130-6ba4-4391-bf2a-7bd6548c1c88" (UID: "2e32d130-6ba4-4391-bf2a-7bd6548c1c88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:40.080261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.079335    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:40.080261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.079356    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:40.080261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.079369    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:40.080261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.079381    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:40.080261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.079395    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:40.080261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.079408    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:40.081150 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.080684    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istio-podinfo" pod "2e32d130-6ba4-4391-bf2a-7bd6548c1c88" (UID: "2e32d130-6ba4-4391-bf2a-7bd6548c1c88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:25:40.081234 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.081186    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istio-envoy" pod "2e32d130-6ba4-4391-bf2a-7bd6548c1c88" (UID: "2e32d130-6ba4-4391-bf2a-7bd6548c1c88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:40.081713 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.081693    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-kube-api-access-w6cgx" pod "2e32d130-6ba4-4391-bf2a-7bd6548c1c88" (UID: "2e32d130-6ba4-4391-bf2a-7bd6548c1c88"). InnerVolumeSpecName "kube-api-access-w6cgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:40.100845 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.100789    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f8991c-pnl9v" podStartSLOduration=2.100774785 podStartE2EDuration="2.100774785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:40.099380554 +0000 UTC m=+2435.146514510" watchObservedRunningTime="2026-10-30 13:25:40.100774785 +0000 UTC m=+2435.147908738"
Oct 30 13:25:40.102849 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.102824    4544 scope.go:122] "RemoveContainer" containerID="fe886e9d20ad0a1ea488f5d69254df6d276da05e6f3b0d107eeb10c70a8795a1"
Oct 30 13:25:40.103230 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:25:40.103192    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe886e9d20ad0a1ea488f5d69254df6d276da05e6f3b0d107eeb10c70a8795a1\": container with ID starting with fe886e9d20ad0a1ea488f5d69254df6d276da05e6f3b0d107eeb10c70a8795a1 not found: ID does not exist" containerID="fe886e9d20ad0a1ea488f5d69254df6d276da05e6f3b0d107eeb10c70a8795a1"
Oct 30 13:25:40.103337 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.103262    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe886e9d20ad0a1ea488f5d69254df6d276da05e6f3b0d107eeb10c70a8795a1"} err="failed to get container status \"fe886e9d20ad0a1ea488f5d69254df6d276da05e6f3b0d107eeb10c70a8795a1\": rpc error: code = NotFound desc = could not find container \"fe886e9d20ad0a1ea488f5d69254df6d276da05e6f3b0d107eeb10c70a8795a1\": container with ID starting with fe886e9d20ad0a1ea488f5d69254df6d276da05e6f3b0d107eeb10c70a8795a1 not found: ID does not exist"
Oct 30 13:25:40.103337 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.103289    4544 scope.go:122] "RemoveContainer" containerID="a578a827da5dbae043fd6b228775c44f833308d1c2bb162eb2e7f39f47d1a53f"
Oct 30 13:25:40.103645 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:25:40.103619    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a578a827da5dbae043fd6b228775c44f833308d1c2bb162eb2e7f39f47d1a53f\": container with ID starting with a578a827da5dbae043fd6b228775c44f833308d1c2bb162eb2e7f39f47d1a53f not found: ID does not exist" containerID="a578a827da5dbae043fd6b228775c44f833308d1c2bb162eb2e7f39f47d1a53f"
Oct 30 13:25:40.103743 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.103657    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a578a827da5dbae043fd6b228775c44f833308d1c2bb162eb2e7f39f47d1a53f"} err="failed to get container status \"a578a827da5dbae043fd6b228775c44f833308d1c2bb162eb2e7f39f47d1a53f\": rpc error: code = NotFound desc = could not find container \"a578a827da5dbae043fd6b228775c44f833308d1c2bb162eb2e7f39f47d1a53f\": container with ID starting with a578a827da5dbae043fd6b228775c44f833308d1c2bb162eb2e7f39f47d1a53f not found: ID does not exist"
Oct 30 13:25:40.181190 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.181157    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:40.181190 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.181188    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6cgx\" (UniqueName: \"kubernetes.io/projected/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-kube-api-access-w6cgx\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:40.181401 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:40.181203    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e32d130-6ba4-4391-bf2a-7bd6548c1c88-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:41.014729 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:41.014696    4544 generic.go:356] "Generic (PLEG): container finished" podID="02995b0b-f05b-40a7-b390-4c59835aad3b" containerID="58dfca7d7b51d2ad7683d60b02fa6c1e37e6390795f66617421c30c830fde1cd" exitCode=0
Oct 30 13:25:41.015186 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:41.014787    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f8991c-pnl9v" event={"ID":"02995b0b-f05b-40a7-b390-4c59835aad3b","Type":"ContainerDied","Data":"58dfca7d7b51d2ad7683d60b02fa6c1e37e6390795f66617421c30c830fde1cd"}
Oct 30 13:25:41.016194 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:41.016174    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"
Oct 30 13:25:41.105629 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:41.105591    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"]
Oct 30 13:25:41.118551 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:41.118521    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m"]
Oct 30 13:25:41.233266 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:41.228800    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:42.191044 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:42.191001    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f8991c-pnl9v"
Oct 30 13:25:42.323918 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:42.323893    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02995b0b-f05b-40a7-b390-4c59835aad3b-kube-api-access-9shd4\" (UniqueName: \"kubernetes.io/projected/02995b0b-f05b-40a7-b390-4c59835aad3b-kube-api-access-9shd4\") pod \"02995b0b-f05b-40a7-b390-4c59835aad3b\" (UID: \"02995b0b-f05b-40a7-b390-4c59835aad3b\") "
Oct 30 13:25:42.324080 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:42.324020    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02995b0b-f05b-40a7-b390-4c59835aad3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02995b0b-f05b-40a7-b390-4c59835aad3b-maas-api-xks-ca\") pod \"02995b0b-f05b-40a7-b390-4c59835aad3b\" (UID: \"02995b0b-f05b-40a7-b390-4c59835aad3b\") "
Oct 30 13:25:42.324346 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:42.324326    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02995b0b-f05b-40a7-b390-4c59835aad3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02995b0b-f05b-40a7-b390-4c59835aad3b-maas-api-openshift-ca\") pod \"02995b0b-f05b-40a7-b390-4c59835aad3b\" (UID: \"02995b0b-f05b-40a7-b390-4c59835aad3b\") "
Oct 30 13:25:42.324738 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:42.324712    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02995b0b-f05b-40a7-b390-4c59835aad3b-maas-api-openshift-ca" pod "02995b0b-f05b-40a7-b390-4c59835aad3b" (UID: "02995b0b-f05b-40a7-b390-4c59835aad3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:42.324906 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:42.324888    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02995b0b-f05b-40a7-b390-4c59835aad3b-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:42.326086 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:42.326057    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02995b0b-f05b-40a7-b390-4c59835aad3b-maas-api-xks-ca" pod "02995b0b-f05b-40a7-b390-4c59835aad3b" (UID: "02995b0b-f05b-40a7-b390-4c59835aad3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:25:42.326212 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:42.326140    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02995b0b-f05b-40a7-b390-4c59835aad3b-kube-api-access-9shd4" pod "02995b0b-f05b-40a7-b390-4c59835aad3b" (UID: "02995b0b-f05b-40a7-b390-4c59835aad3b"). InnerVolumeSpecName "kube-api-access-9shd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:42.426412 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:42.426381    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9shd4\" (UniqueName: \"kubernetes.io/projected/02995b0b-f05b-40a7-b390-4c59835aad3b-kube-api-access-9shd4\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:42.426412 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:42.426411    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02995b0b-f05b-40a7-b390-4c59835aad3b-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:42.868370 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:42.868330    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:43.036439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:43.036407    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f8991c-pnl9v"
Oct 30 13:25:43.036611 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:43.036398    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f8991c-pnl9v" event={"ID":"02995b0b-f05b-40a7-b390-4c59835aad3b","Type":"ContainerDied","Data":"81bbc23e7ed3bb1f7b53f3f496d3fcbe029025f80420431e1f6c39568626f4d3"}
Oct 30 13:25:43.036611 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:43.036536    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81bbc23e7ed3bb1f7b53f3f496d3fcbe029025f80420431e1f6c39568626f4d3"
Oct 30 13:25:43.088793 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:43.088761    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e32d130-6ba4-4391-bf2a-7bd6548c1c88" path="/var/lib/kubelet/pods/2e32d130-6ba4-4391-bf2a-7bd6548c1c88/volumes"
Oct 30 13:25:44.672584 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:44.672551    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:45.338376 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:45.338341    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:46.400864 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:46.400812    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:48.354834 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:48.354794    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:48.715780 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:48.715703    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8089916f-mz2lj"]
Oct 30 13:25:48.716478 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:48.716463    4544 cpu_manager.go:414] "containerMap: removing container" podUID="2e32d130-6ba4-4391-bf2a-7bd6548c1c88" containerName="istio-proxy"
Oct 30 13:25:48.716544 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:48.716480    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e32d130-6ba4-4391-bf2a-7bd6548c1c88" containerName="istio-proxy"
Oct 30 13:25:48.716544 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:48.716512    4544 cpu_manager.go:414] "containerMap: removing container" podUID="02995b0b-f05b-40a7-b390-4c59835aad3b" containerName="revoke-keys"
Oct 30 13:25:48.716544 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:48.716517    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02995b0b-f05b-40a7-b390-4c59835aad3b" containerName="revoke-keys"
Oct 30 13:25:48.716635 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:48.716627    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e32d130-6ba4-4391-bf2a-7bd6548c1c88" containerName="istio-proxy"
Oct 30 13:25:48.716669 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:48.716639    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="02995b0b-f05b-40a7-b390-4c59835aad3b" containerName="revoke-keys"
Oct 30 13:25:48.721628 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:48.721604    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8089916f-mz2lj"
Oct 30 13:25:48.738100 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:48.738071    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:25:48.742739 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:48.742699    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8089916f-mz2lj"]
Oct 30 13:25:48.749211 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:48.749021    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:25:48.830236 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:48.830194    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:48.836719 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:48.836693    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9682bb73-16ea-4b2b-b4d2-e88469657da3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8089916f-mz2lj\" (UID: \"9682bb73-16ea-4b2b-b4d2-e88469657da3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8089916f-mz2lj"
Oct 30 13:25:48.836900 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:48.836884    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9682bb73-16ea-4b2b-b4d2-e88469657da3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8089916f-mz2lj\" (UID: \"9682bb73-16ea-4b2b-b4d2-e88469657da3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8089916f-mz2lj"
Oct 30 13:25:48.836952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:48.836911    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmqnw\" (UniqueName: \"kubernetes.io/projected/9682bb73-16ea-4b2b-b4d2-e88469657da3-kube-api-access-bmqnw\") pod \"maas-api-revoke-sub-8089916f-mz2lj\" (UID: \"9682bb73-16ea-4b2b-b4d2-e88469657da3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8089916f-mz2lj"
Oct 30 13:25:48.938422 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:48.938384    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9682bb73-16ea-4b2b-b4d2-e88469657da3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8089916f-mz2lj\" (UID: \"9682bb73-16ea-4b2b-b4d2-e88469657da3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8089916f-mz2lj"
Oct 30 13:25:48.938638 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:48.938432    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmqnw\" (UniqueName: \"kubernetes.io/projected/9682bb73-16ea-4b2b-b4d2-e88469657da3-kube-api-access-bmqnw\") pod \"maas-api-revoke-sub-8089916f-mz2lj\" (UID: \"9682bb73-16ea-4b2b-b4d2-e88469657da3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8089916f-mz2lj"
Oct 30 13:25:48.938638 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:48.938599    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9682bb73-16ea-4b2b-b4d2-e88469657da3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8089916f-mz2lj\" (UID: \"9682bb73-16ea-4b2b-b4d2-e88469657da3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8089916f-mz2lj"
Oct 30 13:25:48.939384 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:48.939355    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9682bb73-16ea-4b2b-b4d2-e88469657da3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8089916f-mz2lj\" (UID: \"9682bb73-16ea-4b2b-b4d2-e88469657da3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8089916f-mz2lj"
Oct 30 13:25:48.941131 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:48.941092    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9682bb73-16ea-4b2b-b4d2-e88469657da3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8089916f-mz2lj\" (UID: \"9682bb73-16ea-4b2b-b4d2-e88469657da3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8089916f-mz2lj"
Oct 30 13:25:48.963667 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:48.963635    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmqnw\" (UniqueName: \"kubernetes.io/projected/9682bb73-16ea-4b2b-b4d2-e88469657da3-kube-api-access-bmqnw\") pod \"maas-api-revoke-sub-8089916f-mz2lj\" (UID: \"9682bb73-16ea-4b2b-b4d2-e88469657da3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8089916f-mz2lj"
Oct 30 13:25:49.036885 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:49.036798    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8089916f-mz2lj"
Oct 30 13:25:49.267924 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:49.267886    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:49.351299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:49.351263    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8089916f-mz2lj"]
Oct 30 13:25:49.351650 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:25:49.351632    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9682bb73_16ea_4b2b_b4d2_e88469657da3.slice/crio-94fcc77e7d1f339b67ab2bee075477f553acd2b4b7010c374d34053edb9418d5 WatchSource:0}: Error finding container 94fcc77e7d1f339b67ab2bee075477f553acd2b4b7010c374d34053edb9418d5: Status 404 returned error can't find the container with id 94fcc77e7d1f339b67ab2bee075477f553acd2b4b7010c374d34053edb9418d5
Oct 30 13:25:49.775830 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:49.775780    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:50.103109 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:50.103070    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8089916f-mz2lj" event={"ID":"9682bb73-16ea-4b2b-b4d2-e88469657da3","Type":"ContainerStarted","Data":"7b2fd25e9bbd5d7118b09b9c812a1dcf2229709314ad855a8cb0212f4d9a91f7"}
Oct 30 13:25:50.103355 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:50.103138    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8089916f-mz2lj" event={"ID":"9682bb73-16ea-4b2b-b4d2-e88469657da3","Type":"ContainerStarted","Data":"94fcc77e7d1f339b67ab2bee075477f553acd2b4b7010c374d34053edb9418d5"}
Oct 30 13:25:50.210412 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:50.210355    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8089916f-mz2lj" podStartSLOduration=2.210338894 podStartE2EDuration="2.210338894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:50.205420413 +0000 UTC m=+2445.252554366" watchObservedRunningTime="2026-10-30 13:25:50.210338894 +0000 UTC m=+2445.257472846"
Oct 30 13:25:50.335803 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:50.335766    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:50.523948 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:50.523905    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:50.671569 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:50.671488    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:51.113908 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:51.113872    4544 generic.go:356] "Generic (PLEG): container finished" podID="9682bb73-16ea-4b2b-b4d2-e88469657da3" containerID="7b2fd25e9bbd5d7118b09b9c812a1dcf2229709314ad855a8cb0212f4d9a91f7" exitCode=0
Oct 30 13:25:51.114426 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:51.113953    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8089916f-mz2lj" event={"ID":"9682bb73-16ea-4b2b-b4d2-e88469657da3","Type":"ContainerDied","Data":"7b2fd25e9bbd5d7118b09b9c812a1dcf2229709314ad855a8cb0212f4d9a91f7"}
Oct 30 13:25:52.050197 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:52.050121    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:52.372936 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:52.372901    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8089916f-mz2lj"
Oct 30 13:25:52.449456 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:52.449416    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9682bb73-16ea-4b2b-b4d2-e88469657da3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9682bb73-16ea-4b2b-b4d2-e88469657da3-maas-api-xks-ca\") pod \"9682bb73-16ea-4b2b-b4d2-e88469657da3\" (UID: \"9682bb73-16ea-4b2b-b4d2-e88469657da3\") "
Oct 30 13:25:52.449661 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:52.449497    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9682bb73-16ea-4b2b-b4d2-e88469657da3-kube-api-access-bmqnw\" (UniqueName: \"kubernetes.io/projected/9682bb73-16ea-4b2b-b4d2-e88469657da3-kube-api-access-bmqnw\") pod \"9682bb73-16ea-4b2b-b4d2-e88469657da3\" (UID: \"9682bb73-16ea-4b2b-b4d2-e88469657da3\") "
Oct 30 13:25:52.449778 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:52.449754    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9682bb73-16ea-4b2b-b4d2-e88469657da3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9682bb73-16ea-4b2b-b4d2-e88469657da3-maas-api-openshift-ca\") pod \"9682bb73-16ea-4b2b-b4d2-e88469657da3\" (UID: \"9682bb73-16ea-4b2b-b4d2-e88469657da3\") "
Oct 30 13:25:52.450147 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:52.450118    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9682bb73-16ea-4b2b-b4d2-e88469657da3-maas-api-openshift-ca" pod "9682bb73-16ea-4b2b-b4d2-e88469657da3" (UID: "9682bb73-16ea-4b2b-b4d2-e88469657da3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:52.450831 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:52.450808    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9682bb73-16ea-4b2b-b4d2-e88469657da3-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:52.452727 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:52.452665    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9682bb73-16ea-4b2b-b4d2-e88469657da3-kube-api-access-bmqnw" pod "9682bb73-16ea-4b2b-b4d2-e88469657da3" (UID: "9682bb73-16ea-4b2b-b4d2-e88469657da3"). InnerVolumeSpecName "kube-api-access-bmqnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:52.456241 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:52.456197    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9682bb73-16ea-4b2b-b4d2-e88469657da3-maas-api-xks-ca" pod "9682bb73-16ea-4b2b-b4d2-e88469657da3" (UID: "9682bb73-16ea-4b2b-b4d2-e88469657da3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:25:52.579247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:52.576734    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9682bb73-16ea-4b2b-b4d2-e88469657da3-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:52.579247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:52.576780    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmqnw\" (UniqueName: \"kubernetes.io/projected/9682bb73-16ea-4b2b-b4d2-e88469657da3-kube-api-access-bmqnw\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:52.932709 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:52.932667    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:53.065196 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:53.065153    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:53.101846 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:53.101817    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-619b9b-qjfvk"]
Oct 30 13:25:53.112765 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:53.112733    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-619b9b-qjfvk"]
Oct 30 13:25:53.202928 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:53.200018    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8089916f-mz2lj"
Oct 30 13:25:53.202928 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:53.200043    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8089916f-mz2lj" event={"ID":"9682bb73-16ea-4b2b-b4d2-e88469657da3","Type":"ContainerDied","Data":"94fcc77e7d1f339b67ab2bee075477f553acd2b4b7010c374d34053edb9418d5"}
Oct 30 13:25:53.202928 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:53.200105    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94fcc77e7d1f339b67ab2bee075477f553acd2b4b7010c374d34053edb9418d5"
Oct 30 13:25:53.608759 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:53.608723    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:53.703792 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:53.703745    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq" podUID="0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf" containerName="main" containerID="cri-o://09585a9abbaec9c381ba9c05a5fb6f5fcdf57084ee65a4af4bcad94cc6759f60" gracePeriod=45
Oct 30 13:25:53.995994 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:53.995963    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq"
Oct 30 13:25:54.099413 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:54.099367    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-model-cache\") pod \"0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf\" (UID: \"0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf\") "
Oct 30 13:25:54.099614 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:54.099512    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-home\" (UniqueName: \"kubernetes.io/empty-dir/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-home\") pod \"0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf\" (UID: \"0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf\") "
Oct 30 13:25:54.099614 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:54.099569    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-dshm\") pod \"0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf\" (UID: \"0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf\") "
Oct 30 13:25:54.099703 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:54.099617    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-tls-certs\" (UniqueName: \"kubernetes.io/secret/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-tls-certs\") pod \"0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf\" (UID: \"0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf\") "
Oct 30 13:25:54.099703 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:54.099644    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-tmp-dir\") pod \"0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf\" (UID: \"0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf\") "
Oct 30 13:25:54.099703 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:54.099680    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-kube-api-access-kz676\" (UniqueName: \"kubernetes.io/projected/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-kube-api-access-kz676\") pod \"0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf\" (UID: \"0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf\") "
Oct 30 13:25:54.099703 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:54.099681    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-model-cache" pod "0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf" (UID: "0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:54.099988 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:54.099948    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-tmp-dir" pod "0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf" (UID: "0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:54.100184 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:54.100160    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-home" pod "0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf" (UID: "0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:54.100571 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:54.100543    4544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-home\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:54.100571 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:54.100567    4544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-tmp-dir\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:54.100738 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:54.100581    4544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-model-cache\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:54.102289 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:54.102265    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-kube-api-access-kz676" pod "0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf" (UID: "0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf"). InnerVolumeSpecName "kube-api-access-kz676". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:54.102401 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:54.102378    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-tls-certs" pod "0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf" (UID: "0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:25:54.102471 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:54.102455    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-dshm" pod "0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf" (UID: "0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:54.201935 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:54.201834    4544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-dshm\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:54.201935 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:54.201877    4544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-tls-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:54.201935 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:54.201893    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz676\" (UniqueName: \"kubernetes.io/projected/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf-kube-api-access-kz676\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:25:54.210216 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:54.210178    4544 generic.go:356] "Generic (PLEG): container finished" podID="0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf" containerID="09585a9abbaec9c381ba9c05a5fb6f5fcdf57084ee65a4af4bcad94cc6759f60" exitCode=0
Oct 30 13:25:54.210439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:54.210244    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq"
Oct 30 13:25:54.210439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:54.210272    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq" event={"ID":"0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf","Type":"ContainerDied","Data":"09585a9abbaec9c381ba9c05a5fb6f5fcdf57084ee65a4af4bcad94cc6759f60"}
Oct 30 13:25:54.210439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:54.210330    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq" event={"ID":"0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf","Type":"ContainerDied","Data":"afde71b914de7f6bf0aa0d8f75737d2f778280d4cfb565fc38dcdba80ac8d414"}
Oct 30 13:25:54.210439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:54.210353    4544 scope.go:122] "RemoveContainer" containerID="09585a9abbaec9c381ba9c05a5fb6f5fcdf57084ee65a4af4bcad94cc6759f60"
Oct 30 13:25:54.222985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:54.222963    4544 scope.go:122] "RemoveContainer" containerID="09585a9abbaec9c381ba9c05a5fb6f5fcdf57084ee65a4af4bcad94cc6759f60"
Oct 30 13:25:54.223364 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:25:54.223342    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09585a9abbaec9c381ba9c05a5fb6f5fcdf57084ee65a4af4bcad94cc6759f60\": container with ID starting with 09585a9abbaec9c381ba9c05a5fb6f5fcdf57084ee65a4af4bcad94cc6759f60 not found: ID does not exist" containerID="09585a9abbaec9c381ba9c05a5fb6f5fcdf57084ee65a4af4bcad94cc6759f60"
Oct 30 13:25:54.223420 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:54.223375    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09585a9abbaec9c381ba9c05a5fb6f5fcdf57084ee65a4af4bcad94cc6759f60"} err="failed to get container status \"09585a9abbaec9c381ba9c05a5fb6f5fcdf57084ee65a4af4bcad94cc6759f60\": rpc error: code = NotFound desc = could not find container \"09585a9abbaec9c381ba9c05a5fb6f5fcdf57084ee65a4af4bcad94cc6759f60\": container with ID starting with 09585a9abbaec9c381ba9c05a5fb6f5fcdf57084ee65a4af4bcad94cc6759f60 not found: ID does not exist"
Oct 30 13:25:54.266543 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:54.266498    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq"]
Oct 30 13:25:54.271080 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:54.271037    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-60a9a6/test-model-60a9a6-kserve-7db4fcd49-6m4bq"]
Oct 30 13:25:54.425678 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:54.425640    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:54.545720 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:54.545643    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:55.036665 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:55.036631    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:55.088049 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:55.088019    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf" path="/var/lib/kubelet/pods/0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf/volumes"
Oct 30 13:25:55.088557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:55.088535    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ffdd227-1389-4c92-a0e2-2e6db870d4bb" path="/var/lib/kubelet/pods/1ffdd227-1389-4c92-a0e2-2e6db870d4bb/volumes"
Oct 30 13:25:56.146971 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.146918    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:56.287826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.287699    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:56.319838 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.319789    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558"]
Oct 30 13:25:56.320672 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.320648    4544 cpu_manager.go:414] "containerMap: removing container" podUID="9682bb73-16ea-4b2b-b4d2-e88469657da3" containerName="revoke-keys"
Oct 30 13:25:56.320672 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.320670    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9682bb73-16ea-4b2b-b4d2-e88469657da3" containerName="revoke-keys"
Oct 30 13:25:56.320828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.320684    4544 cpu_manager.go:414] "containerMap: removing container" podUID="2e32d130-6ba4-4391-bf2a-7bd6548c1c88" containerName="istio-proxy"
Oct 30 13:25:56.320828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.320691    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e32d130-6ba4-4391-bf2a-7bd6548c1c88" containerName="istio-proxy"
Oct 30 13:25:56.320828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.320707    4544 cpu_manager.go:414] "containerMap: removing container" podUID="0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf" containerName="main"
Oct 30 13:25:56.320828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.320712    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf" containerName="main"
Oct 30 13:25:56.320979 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.320858    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bbf1f7a-a241-4a1d-9cc7-f6fa136979bf" containerName="main"
Oct 30 13:25:56.320979 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.320879    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="9682bb73-16ea-4b2b-b4d2-e88469657da3" containerName="revoke-keys"
Oct 30 13:25:56.320979 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.320887    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e32d130-6ba4-4391-bf2a-7bd6548c1c88" containerName="istio-proxy"
Oct 30 13:25:56.325694 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.325678    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558"
Oct 30 13:25:56.346333 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.346308    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558"]
Oct 30 13:25:56.353209 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.353044    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-627dfa\""
Oct 30 13:25:56.353209 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.353157    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-627dfa-dockercfg-lnxng\""
Oct 30 13:25:56.417172 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.417101    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q"]
Oct 30 13:25:56.421860 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.421835    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q"
Oct 30 13:25:56.430258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.430202    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7da56bd-b1ed-419f-b90d-c9a68a8f756d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558\" (UID: \"d7da56bd-b1ed-419f-b90d-c9a68a8f756d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558"
Oct 30 13:25:56.430512 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.430493    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxb5r\" (UniqueName: \"kubernetes.io/projected/d7da56bd-b1ed-419f-b90d-c9a68a8f756d-kube-api-access-lxb5r\") pod \"payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558\" (UID: \"d7da56bd-b1ed-419f-b90d-c9a68a8f756d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558"
Oct 30 13:25:56.445244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.445199    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q"]
Oct 30 13:25:56.558455 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.558418    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b63682f2-c062-4ef4-801f-b84a3a7c10e0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q\" (UID: \"b63682f2-c062-4ef4-801f-b84a3a7c10e0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q"
Oct 30 13:25:56.558780 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.558754    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxb5r\" (UniqueName: \"kubernetes.io/projected/d7da56bd-b1ed-419f-b90d-c9a68a8f756d-kube-api-access-lxb5r\") pod \"payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558\" (UID: \"d7da56bd-b1ed-419f-b90d-c9a68a8f756d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558"
Oct 30 13:25:56.559029 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.559014    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7da56bd-b1ed-419f-b90d-c9a68a8f756d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558\" (UID: \"d7da56bd-b1ed-419f-b90d-c9a68a8f756d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558"
Oct 30 13:25:56.559288 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.559272    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kpfg\" (UniqueName: \"kubernetes.io/projected/b63682f2-c062-4ef4-801f-b84a3a7c10e0-kube-api-access-2kpfg\") pod \"payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q\" (UID: \"b63682f2-c062-4ef4-801f-b84a3a7c10e0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q"
Oct 30 13:25:56.560603 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.560577    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7da56bd-b1ed-419f-b90d-c9a68a8f756d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558\" (UID: \"d7da56bd-b1ed-419f-b90d-c9a68a8f756d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558"
Oct 30 13:25:56.587161 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.587124    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:56.644248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.644031    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxb5r\" (UniqueName: \"kubernetes.io/projected/d7da56bd-b1ed-419f-b90d-c9a68a8f756d-kube-api-access-lxb5r\") pod \"payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558\" (UID: \"d7da56bd-b1ed-419f-b90d-c9a68a8f756d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558"
Oct 30 13:25:56.644449 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.644262    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558"
Oct 30 13:25:56.662283 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.661438    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kpfg\" (UniqueName: \"kubernetes.io/projected/b63682f2-c062-4ef4-801f-b84a3a7c10e0-kube-api-access-2kpfg\") pod \"payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q\" (UID: \"b63682f2-c062-4ef4-801f-b84a3a7c10e0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q"
Oct 30 13:25:56.662283 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.661564    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b63682f2-c062-4ef4-801f-b84a3a7c10e0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q\" (UID: \"b63682f2-c062-4ef4-801f-b84a3a7c10e0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q"
Oct 30 13:25:56.662591 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.662567    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b63682f2-c062-4ef4-801f-b84a3a7c10e0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q\" (UID: \"b63682f2-c062-4ef4-801f-b84a3a7c10e0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q"
Oct 30 13:25:56.687459 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.687393    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kpfg\" (UniqueName: \"kubernetes.io/projected/b63682f2-c062-4ef4-801f-b84a3a7c10e0-kube-api-access-2kpfg\") pod \"payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q\" (UID: \"b63682f2-c062-4ef4-801f-b84a3a7c10e0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q"
Oct 30 13:25:56.736854 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.736601    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q"
Oct 30 13:25:56.965814 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:56.965790    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558"]
Oct 30 13:25:56.966754 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:25:56.966723    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7da56bd_b1ed_419f_b90d_c9a68a8f756d.slice/crio-c82d1096b6617c7f92fd0ddbcdaf1c67c2c1a930cb1c7ed305840037ed0a2e2a WatchSource:0}: Error finding container c82d1096b6617c7f92fd0ddbcdaf1c67c2c1a930cb1c7ed305840037ed0a2e2a: Status 404 returned error can't find the container with id c82d1096b6617c7f92fd0ddbcdaf1c67c2c1a930cb1c7ed305840037ed0a2e2a
Oct 30 13:25:57.063696 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:25:57.063665    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb63682f2_c062_4ef4_801f_b84a3a7c10e0.slice/crio-291f2b2a330606457866e60696010efed5e924766a60b6053f764666de074c3f WatchSource:0}: Error finding container 291f2b2a330606457866e60696010efed5e924766a60b6053f764666de074c3f: Status 404 returned error can't find the container with id 291f2b2a330606457866e60696010efed5e924766a60b6053f764666de074c3f
Oct 30 13:25:57.064270 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:57.064242    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q"]
Oct 30 13:25:57.077683 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:57.077656    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:57.240804 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:57.240718    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q" event={"ID":"b63682f2-c062-4ef4-801f-b84a3a7c10e0","Type":"ContainerStarted","Data":"c0dad761436aeb0a5652b16312563ee313497554ada9a11e09b8a647a7cd5206"}
Oct 30 13:25:57.240804 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:57.240770    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q" event={"ID":"b63682f2-c062-4ef4-801f-b84a3a7c10e0","Type":"ContainerStarted","Data":"291f2b2a330606457866e60696010efed5e924766a60b6053f764666de074c3f"}
Oct 30 13:25:57.241341 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:57.240812    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q"
Oct 30 13:25:57.242611 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:57.242586    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558" event={"ID":"d7da56bd-b1ed-419f-b90d-c9a68a8f756d","Type":"ContainerStarted","Data":"10566545545ef4c2c6cb6c768c48d6805dcb63683bb3e1bfa783b1851e71f7c2"}
Oct 30 13:25:57.242726 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:57.242631    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558" event={"ID":"d7da56bd-b1ed-419f-b90d-c9a68a8f756d","Type":"ContainerStarted","Data":"c82d1096b6617c7f92fd0ddbcdaf1c67c2c1a930cb1c7ed305840037ed0a2e2a"}
Oct 30 13:25:57.242726 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:57.242650    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558"
Oct 30 13:25:57.284900 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:57.284859    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q" podStartSLOduration=1.284846023 podStartE2EDuration="1.284846023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:57.280734805 +0000 UTC m=+2452.327868770" watchObservedRunningTime="2026-10-30 13:25:57.284846023 +0000 UTC m=+2452.331979969"
Oct 30 13:25:57.326985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:57.326938    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558" podStartSLOduration=1.326926487 podStartE2EDuration="1.326926487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:57.32329673 +0000 UTC m=+2452.370430683" watchObservedRunningTime="2026-10-30 13:25:57.326926487 +0000 UTC m=+2452.374060467"
Oct 30 13:25:58.672293 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:58.672262    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg"]
Oct 30 13:25:58.708885 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:58.708854    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:59.012344 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:25:59.012245    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:25:59.019363 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:59.019336    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:59.812212 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:59.812178    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:25:59.974622 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:25:59.974585    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:00.619629 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:00.619598    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:00.755242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:00.754405    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:00.986771 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:00.986686    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:01.158046 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:01.157912    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:01.828188 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:01.828152    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:02.397057 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:02.397011    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"]
Oct 30 13:26:02.402093 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:02.402068    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:26:02.425868 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:02.425835    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-96ab73-openshift-default-dockercfg-nshzl\""
Oct 30 13:26:02.438191 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:02.438149    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-workload-socket\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:26:02.438446 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:02.438290    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e66eb26-5727-4ad8-881f-64b6711fe440-istio-podinfo\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:26:02.438446 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:02.438402    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e66eb26-5727-4ad8-881f-64b6711fe440-istio-token\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:26:02.438446 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:02.438436    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-credential-socket\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:26:02.438627 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:02.438472    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-workload-certs\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:26:02.438627 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:02.438529    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e66eb26-5727-4ad8-881f-64b6711fe440-istiod-ca-cert\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:26:02.438627 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:02.438600    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7cg4\" (UniqueName: \"kubernetes.io/projected/4e66eb26-5727-4ad8-881f-64b6711fe440-kube-api-access-m7cg4\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:26:02.438870 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:02.438636    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-istio-data\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:26:02.440384 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:02.440357    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-istio-envoy\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:26:02.442651 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:02.442628    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"]
Oct 30 13:26:02.541293 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:02.540660    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:02.598245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:02.592034    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-workload-socket\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:26:02.598245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:02.592099    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e66eb26-5727-4ad8-881f-64b6711fe440-istio-podinfo\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:26:02.598245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:02.592196    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e66eb26-5727-4ad8-881f-64b6711fe440-istio-token\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:26:02.598245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:02.592248    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-credential-socket\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:26:02.598245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:02.592284    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-workload-certs\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:26:02.598245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:02.592353    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e66eb26-5727-4ad8-881f-64b6711fe440-istiod-ca-cert\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:26:02.598245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:02.592439    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7cg4\" (UniqueName: \"kubernetes.io/projected/4e66eb26-5727-4ad8-881f-64b6711fe440-kube-api-access-m7cg4\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:26:02.598245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:02.592478    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-istio-data\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:26:02.598245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:02.592596    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-istio-envoy\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:26:02.610777 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:02.610690    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-credential-socket\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:26:02.629552 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:02.611364    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-workload-certs\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:26:02.631653 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:02.630987    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-istio-data\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:26:02.632909 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:02.632881    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-workload-socket\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:26:02.634070 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:02.634045    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e66eb26-5727-4ad8-881f-64b6711fe440-istiod-ca-cert\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:26:02.647284 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:02.641748    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e66eb26-5727-4ad8-881f-64b6711fe440-istio-token\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:26:02.648323 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:02.648111    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7cg4\" (UniqueName: \"kubernetes.io/projected/4e66eb26-5727-4ad8-881f-64b6711fe440-kube-api-access-m7cg4\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:26:02.690243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:02.681803    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e66eb26-5727-4ad8-881f-64b6711fe440-istio-podinfo\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:26:02.690243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:02.682055    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-istio-envoy\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:26:02.690243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:02.683246    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:02.732832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:02.732579    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:26:03.063958 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:03.063932    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"]
Oct 30 13:26:03.065245 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:26:03.065201    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e66eb26_5727_4ad8_881f_64b6711fe440.slice/crio-a275c8e06afd8d6ff095545a5abb025b7fb6f079593934867ab7bc6fe57c73c1 WatchSource:0}: Error finding container a275c8e06afd8d6ff095545a5abb025b7fb6f079593934867ab7bc6fe57c73c1: Status 404 returned error can't find the container with id a275c8e06afd8d6ff095545a5abb025b7fb6f079593934867ab7bc6fe57c73c1
Oct 30 13:26:03.068108 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:03.068067    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:26:03.068238 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:03.068156    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:26:03.068315 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:03.068211    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:26:03.355067 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:03.355029    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt" event={"ID":"4e66eb26-5727-4ad8-881f-64b6711fe440","Type":"ContainerStarted","Data":"af461ba3c237d09537277a0321974f745885b1401311091e4ff9e15a5550f79a"}
Oct 30 13:26:03.355301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:03.355090    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt" event={"ID":"4e66eb26-5727-4ad8-881f-64b6711fe440","Type":"ContainerStarted","Data":"a275c8e06afd8d6ff095545a5abb025b7fb6f079593934867ab7bc6fe57c73c1"}
Oct 30 13:26:03.413057 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:03.412995    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt" podStartSLOduration=1.41298027 podStartE2EDuration="1.41298027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:03.409367227 +0000 UTC m=+2458.456501192" watchObservedRunningTime="2026-10-30 13:26:03.41298027 +0000 UTC m=+2458.460114257"
Oct 30 13:26:03.502874 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:03.502833    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:03.733325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:03.733245    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:26:04.740145 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:04.740115    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:26:05.372003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:05.371973    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:26:05.372961 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:05.372941    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:26:05.377478 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:05.377453    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:05.965851 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:05.965810    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:06.113340 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:06.113308    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:07.842532 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:07.842472    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:08.115366 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:08.115279    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:08.292246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:08.288786    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q"
Oct 30 13:26:08.299075 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:08.299046    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558"
Oct 30 13:26:09.438285 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:09.438249    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:09.736435 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:09.736321    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:10.098671 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:10.098635    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:11.880861 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:11.880817    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:13.546181 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:13.546134    4544 scope.go:122] "RemoveContainer" containerID="8df076bfb3c1061c01514a7b1ca3dea2b44f15c368ad4ba0a04c79824eeef0d2"
Oct 30 13:26:13.557152 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:13.557132    4544 scope.go:122] "RemoveContainer" containerID="724c0732e2f42ea495c2b006f5b2355b777a854b6038b6018f7d83354e1b82dc"
Oct 30 13:26:13.567275 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:13.567247    4544 scope.go:122] "RemoveContainer" containerID="a612aa2a754ff5add34ed1c4e1d39a378f8bbbd0f4e51a83c9558b0cd9728085"
Oct 30 13:26:13.657834 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:13.657801    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:13.698348 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:13.698123    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg" podUID="c6998926-73ac-4828-91b7-d453b5ebc33e" containerName="main" containerID="cri-o://6f2a250ed923317b1f0789e2c6550d0dfdfff9c56695f2628bff4e0a8dff05fe" gracePeriod=45
Oct 30 13:26:13.992169 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:13.992142    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg"
Oct 30 13:26:14.020666 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:14.020630    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6998926-73ac-4828-91b7-d453b5ebc33e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6998926-73ac-4828-91b7-d453b5ebc33e-tmp-dir\") pod \"c6998926-73ac-4828-91b7-d453b5ebc33e\" (UID: \"c6998926-73ac-4828-91b7-d453b5ebc33e\") "
Oct 30 13:26:14.020666 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:14.020667    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6998926-73ac-4828-91b7-d453b5ebc33e-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6998926-73ac-4828-91b7-d453b5ebc33e-tls-certs\") pod \"c6998926-73ac-4828-91b7-d453b5ebc33e\" (UID: \"c6998926-73ac-4828-91b7-d453b5ebc33e\") "
Oct 30 13:26:14.020927 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:14.020707    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6998926-73ac-4828-91b7-d453b5ebc33e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6998926-73ac-4828-91b7-d453b5ebc33e-model-cache\") pod \"c6998926-73ac-4828-91b7-d453b5ebc33e\" (UID: \"c6998926-73ac-4828-91b7-d453b5ebc33e\") "
Oct 30 13:26:14.020927 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:14.020750    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6998926-73ac-4828-91b7-d453b5ebc33e-kube-api-access-ghtpj\" (UniqueName: \"kubernetes.io/projected/c6998926-73ac-4828-91b7-d453b5ebc33e-kube-api-access-ghtpj\") pod \"c6998926-73ac-4828-91b7-d453b5ebc33e\" (UID: \"c6998926-73ac-4828-91b7-d453b5ebc33e\") "
Oct 30 13:26:14.020927 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:14.020805    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6998926-73ac-4828-91b7-d453b5ebc33e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6998926-73ac-4828-91b7-d453b5ebc33e-dshm\") pod \"c6998926-73ac-4828-91b7-d453b5ebc33e\" (UID: \"c6998926-73ac-4828-91b7-d453b5ebc33e\") "
Oct 30 13:26:14.020927 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:14.020849    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6998926-73ac-4828-91b7-d453b5ebc33e-home\" (UniqueName: \"kubernetes.io/empty-dir/c6998926-73ac-4828-91b7-d453b5ebc33e-home\") pod \"c6998926-73ac-4828-91b7-d453b5ebc33e\" (UID: \"c6998926-73ac-4828-91b7-d453b5ebc33e\") "
Oct 30 13:26:14.021140 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:14.020987    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6998926-73ac-4828-91b7-d453b5ebc33e-tmp-dir" pod "c6998926-73ac-4828-91b7-d453b5ebc33e" (UID: "c6998926-73ac-4828-91b7-d453b5ebc33e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:14.021140 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:14.021020    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6998926-73ac-4828-91b7-d453b5ebc33e-model-cache" pod "c6998926-73ac-4828-91b7-d453b5ebc33e" (UID: "c6998926-73ac-4828-91b7-d453b5ebc33e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:14.021290 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:14.021259    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6998926-73ac-4828-91b7-d453b5ebc33e-home" pod "c6998926-73ac-4828-91b7-d453b5ebc33e" (UID: "c6998926-73ac-4828-91b7-d453b5ebc33e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:14.021569 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:14.021550    4544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6998926-73ac-4828-91b7-d453b5ebc33e-tmp-dir\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:26:14.021639 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:14.021573    4544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6998926-73ac-4828-91b7-d453b5ebc33e-model-cache\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:26:14.021639 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:14.021587    4544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6998926-73ac-4828-91b7-d453b5ebc33e-home\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:26:14.023084 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:14.023058    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6998926-73ac-4828-91b7-d453b5ebc33e-dshm" pod "c6998926-73ac-4828-91b7-d453b5ebc33e" (UID: "c6998926-73ac-4828-91b7-d453b5ebc33e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:14.023403 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:14.023379    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6998926-73ac-4828-91b7-d453b5ebc33e-tls-certs" pod "c6998926-73ac-4828-91b7-d453b5ebc33e" (UID: "c6998926-73ac-4828-91b7-d453b5ebc33e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:26:14.023519 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:14.023410    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6998926-73ac-4828-91b7-d453b5ebc33e-kube-api-access-ghtpj" pod "c6998926-73ac-4828-91b7-d453b5ebc33e" (UID: "c6998926-73ac-4828-91b7-d453b5ebc33e"). InnerVolumeSpecName "kube-api-access-ghtpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:14.122955 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:14.122869    4544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6998926-73ac-4828-91b7-d453b5ebc33e-tls-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:26:14.122955 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:14.122894    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghtpj\" (UniqueName: \"kubernetes.io/projected/c6998926-73ac-4828-91b7-d453b5ebc33e-kube-api-access-ghtpj\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:26:14.122955 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:14.122904    4544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6998926-73ac-4828-91b7-d453b5ebc33e-dshm\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:26:14.457332 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:14.457234    4544 generic.go:356] "Generic (PLEG): container finished" podID="c6998926-73ac-4828-91b7-d453b5ebc33e" containerID="6f2a250ed923317b1f0789e2c6550d0dfdfff9c56695f2628bff4e0a8dff05fe" exitCode=0
Oct 30 13:26:14.457332 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:14.457277    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg"
Oct 30 13:26:14.457332 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:14.457300    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg" event={"ID":"c6998926-73ac-4828-91b7-d453b5ebc33e","Type":"ContainerDied","Data":"6f2a250ed923317b1f0789e2c6550d0dfdfff9c56695f2628bff4e0a8dff05fe"}
Oct 30 13:26:14.457593 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:14.457340    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg" event={"ID":"c6998926-73ac-4828-91b7-d453b5ebc33e","Type":"ContainerDied","Data":"05ca08ca5f50632937b3fa96cc57250bffe6d3c220f544e3424e209cf64881c7"}
Oct 30 13:26:14.457593 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:14.457360    4544 scope.go:122] "RemoveContainer" containerID="6f2a250ed923317b1f0789e2c6550d0dfdfff9c56695f2628bff4e0a8dff05fe"
Oct 30 13:26:14.468568 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:14.468550    4544 scope.go:122] "RemoveContainer" containerID="6f2a250ed923317b1f0789e2c6550d0dfdfff9c56695f2628bff4e0a8dff05fe"
Oct 30 13:26:14.468874 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:26:14.468842    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f2a250ed923317b1f0789e2c6550d0dfdfff9c56695f2628bff4e0a8dff05fe\": container with ID starting with 6f2a250ed923317b1f0789e2c6550d0dfdfff9c56695f2628bff4e0a8dff05fe not found: ID does not exist" containerID="6f2a250ed923317b1f0789e2c6550d0dfdfff9c56695f2628bff4e0a8dff05fe"
Oct 30 13:26:14.468924 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:14.468882    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f2a250ed923317b1f0789e2c6550d0dfdfff9c56695f2628bff4e0a8dff05fe"} err="failed to get container status \"6f2a250ed923317b1f0789e2c6550d0dfdfff9c56695f2628bff4e0a8dff05fe\": rpc error: code = NotFound desc = could not find container \"6f2a250ed923317b1f0789e2c6550d0dfdfff9c56695f2628bff4e0a8dff05fe\": container with ID starting with 6f2a250ed923317b1f0789e2c6550d0dfdfff9c56695f2628bff4e0a8dff05fe not found: ID does not exist"
Oct 30 13:26:14.511950 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:14.511915    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg"]
Oct 30 13:26:14.523493 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:14.523470    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f3338e/test-model-f3338e-kserve-86cff7f499-7lldg"]
Oct 30 13:26:15.090819 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:15.090786    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6998926-73ac-4828-91b7-d453b5ebc33e" path="/var/lib/kubelet/pods/c6998926-73ac-4828-91b7-d453b5ebc33e/volumes"
Oct 30 13:26:15.567345 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:15.567256    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:17.409746 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:17.409655    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:17.710881 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:17.710804    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj"]
Oct 30 13:26:17.867398 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:17.867362    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:20.012258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:20.010725    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:22.100486 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:22.100453    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:22.551879 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:22.551785    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv"]
Oct 30 13:26:22.572008 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:22.571975    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:24.033784 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:24.033750    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:25.601778 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:25.601732    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:27.487351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:27.487314    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:29.008152 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:26:29.008113    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:26:29.395215 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:29.395165    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:31.161687 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:31.161646    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:31.716298 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:31.716215    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:32.734192 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:32.734142    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj" podUID="21ffd071-46da-4940-879a-2877eca7d1fe" containerName="main" containerID="cri-o://358222b95bc3c2e17638279ed3172de130bab87c95c03cd8b32a785e788c0cfe" gracePeriod=45
Oct 30 13:26:32.873289 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:32.873244    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj" podUID="21ffd071-46da-4940-879a-2877eca7d1fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.229:8000/ready\": dial tcp 10.133.0.229:8000: connect: connection refused"
Oct 30 13:26:33.028260 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:33.028211    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj"
Oct 30 13:26:33.084831 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:33.084790    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21ffd071-46da-4940-879a-2877eca7d1fe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ffd071-46da-4940-879a-2877eca7d1fe-tmp-dir\") pod \"21ffd071-46da-4940-879a-2877eca7d1fe\" (UID: \"21ffd071-46da-4940-879a-2877eca7d1fe\") "
Oct 30 13:26:33.085029 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:33.084860    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21ffd071-46da-4940-879a-2877eca7d1fe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ffd071-46da-4940-879a-2877eca7d1fe-model-cache\") pod \"21ffd071-46da-4940-879a-2877eca7d1fe\" (UID: \"21ffd071-46da-4940-879a-2877eca7d1fe\") "
Oct 30 13:26:33.085029 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:33.084895    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21ffd071-46da-4940-879a-2877eca7d1fe-kube-api-access-hjg44\" (UniqueName: \"kubernetes.io/projected/21ffd071-46da-4940-879a-2877eca7d1fe-kube-api-access-hjg44\") pod \"21ffd071-46da-4940-879a-2877eca7d1fe\" (UID: \"21ffd071-46da-4940-879a-2877eca7d1fe\") "
Oct 30 13:26:33.085029 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:33.084971    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21ffd071-46da-4940-879a-2877eca7d1fe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/21ffd071-46da-4940-879a-2877eca7d1fe-dshm\") pod \"21ffd071-46da-4940-879a-2877eca7d1fe\" (UID: \"21ffd071-46da-4940-879a-2877eca7d1fe\") "
Oct 30 13:26:33.085323 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:33.085095    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21ffd071-46da-4940-879a-2877eca7d1fe-home\" (UniqueName: \"kubernetes.io/empty-dir/21ffd071-46da-4940-879a-2877eca7d1fe-home\") pod \"21ffd071-46da-4940-879a-2877eca7d1fe\" (UID: \"21ffd071-46da-4940-879a-2877eca7d1fe\") "
Oct 30 13:26:33.085323 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:33.085148    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21ffd071-46da-4940-879a-2877eca7d1fe-tls-certs\" (UniqueName: \"kubernetes.io/secret/21ffd071-46da-4940-879a-2877eca7d1fe-tls-certs\") pod \"21ffd071-46da-4940-879a-2877eca7d1fe\" (UID: \"21ffd071-46da-4940-879a-2877eca7d1fe\") "
Oct 30 13:26:33.086581 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:33.086553    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ffd071-46da-4940-879a-2877eca7d1fe-home" pod "21ffd071-46da-4940-879a-2877eca7d1fe" (UID: "21ffd071-46da-4940-879a-2877eca7d1fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:33.086959 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:33.086929    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ffd071-46da-4940-879a-2877eca7d1fe-tmp-dir" pod "21ffd071-46da-4940-879a-2877eca7d1fe" (UID: "21ffd071-46da-4940-879a-2877eca7d1fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:33.087128 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:33.086944    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ffd071-46da-4940-879a-2877eca7d1fe-model-cache" pod "21ffd071-46da-4940-879a-2877eca7d1fe" (UID: "21ffd071-46da-4940-879a-2877eca7d1fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:33.088736 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:33.088689    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ffd071-46da-4940-879a-2877eca7d1fe-dshm" pod "21ffd071-46da-4940-879a-2877eca7d1fe" (UID: "21ffd071-46da-4940-879a-2877eca7d1fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:33.088907 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:33.088743    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ffd071-46da-4940-879a-2877eca7d1fe-kube-api-access-hjg44" pod "21ffd071-46da-4940-879a-2877eca7d1fe" (UID: "21ffd071-46da-4940-879a-2877eca7d1fe"). InnerVolumeSpecName "kube-api-access-hjg44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:33.089959 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:33.089933    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21ffd071-46da-4940-879a-2877eca7d1fe-tls-certs" pod "21ffd071-46da-4940-879a-2877eca7d1fe" (UID: "21ffd071-46da-4940-879a-2877eca7d1fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:26:33.187560 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:33.187526    4544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ffd071-46da-4940-879a-2877eca7d1fe-tmp-dir\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:26:33.187560 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:33.187558    4544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ffd071-46da-4940-879a-2877eca7d1fe-model-cache\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:26:33.187805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:33.187574    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjg44\" (UniqueName: \"kubernetes.io/projected/21ffd071-46da-4940-879a-2877eca7d1fe-kube-api-access-hjg44\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:26:33.187805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:33.187587    4544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21ffd071-46da-4940-879a-2877eca7d1fe-dshm\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:26:33.187805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:33.187600    4544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21ffd071-46da-4940-879a-2877eca7d1fe-home\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:26:33.187805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:33.187611    4544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21ffd071-46da-4940-879a-2877eca7d1fe-tls-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:26:33.187958 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:33.187834    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:33.681622 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:33.681590    4544 generic.go:356] "Generic (PLEG): container finished" podID="21ffd071-46da-4940-879a-2877eca7d1fe" containerID="358222b95bc3c2e17638279ed3172de130bab87c95c03cd8b32a785e788c0cfe" exitCode=0
Oct 30 13:26:33.681849 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:33.681627    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj" event={"ID":"21ffd071-46da-4940-879a-2877eca7d1fe","Type":"ContainerDied","Data":"358222b95bc3c2e17638279ed3172de130bab87c95c03cd8b32a785e788c0cfe"}
Oct 30 13:26:33.681849 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:33.681650    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj"
Oct 30 13:26:33.681849 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:33.681685    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj" event={"ID":"21ffd071-46da-4940-879a-2877eca7d1fe","Type":"ContainerDied","Data":"3c88c84be01dc41f0f1d9db40f59e14d1f0200237b9a153086df54c07f67e94e"}
Oct 30 13:26:33.681849 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:33.681711    4544 scope.go:122] "RemoveContainer" containerID="358222b95bc3c2e17638279ed3172de130bab87c95c03cd8b32a785e788c0cfe"
Oct 30 13:26:33.695508 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:33.695483    4544 scope.go:122] "RemoveContainer" containerID="358222b95bc3c2e17638279ed3172de130bab87c95c03cd8b32a785e788c0cfe"
Oct 30 13:26:33.695804 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:26:33.695781    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"358222b95bc3c2e17638279ed3172de130bab87c95c03cd8b32a785e788c0cfe\": container with ID starting with 358222b95bc3c2e17638279ed3172de130bab87c95c03cd8b32a785e788c0cfe not found: ID does not exist" containerID="358222b95bc3c2e17638279ed3172de130bab87c95c03cd8b32a785e788c0cfe"
Oct 30 13:26:33.695875 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:33.695813    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"358222b95bc3c2e17638279ed3172de130bab87c95c03cd8b32a785e788c0cfe"} err="failed to get container status \"358222b95bc3c2e17638279ed3172de130bab87c95c03cd8b32a785e788c0cfe\": rpc error: code = NotFound desc = could not find container \"358222b95bc3c2e17638279ed3172de130bab87c95c03cd8b32a785e788c0cfe\": container with ID starting with 358222b95bc3c2e17638279ed3172de130bab87c95c03cd8b32a785e788c0cfe not found: ID does not exist"
Oct 30 13:26:33.735309 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:33.735275    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj"]
Oct 30 13:26:33.747363 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:33.747334    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-bcd76c/auth-test-model-bcd76c-kserve-665b758b77-bppbj"]
Oct 30 13:26:34.905853 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:34.905815    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:35.023096 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:35.018725    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:35.089980 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:35.089941    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21ffd071-46da-4940-879a-2877eca7d1fe" path="/var/lib/kubelet/pods/21ffd071-46da-4940-879a-2877eca7d1fe/volumes"
Oct 30 13:26:35.152737 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:35.152702    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:36.904586 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:36.904544    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:37.581054 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:37.581012    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv" podUID="7739d642-eb43-4056-ae8b-ace5f85e9155" containerName="main" containerID="cri-o://4d2ffd282929303e855066378909503f6c1f38695df2efb5c7b34e7b6bdf147b" gracePeriod=45
Oct 30 13:26:37.622150 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:37.622107    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv" podUID="7739d642-eb43-4056-ae8b-ace5f85e9155" containerName="main" probeResult="failure" output="Get \"https://10.133.0.237:8000/ready\": dial tcp 10.133.0.237:8000: connect: connection refused"
Oct 30 13:26:37.723732 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:37.723700    4544 generic.go:356] "Generic (PLEG): container finished" podID="7739d642-eb43-4056-ae8b-ace5f85e9155" containerID="4d2ffd282929303e855066378909503f6c1f38695df2efb5c7b34e7b6bdf147b" exitCode=0
Oct 30 13:26:37.723920 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:37.723775    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv" event={"ID":"7739d642-eb43-4056-ae8b-ace5f85e9155","Type":"ContainerDied","Data":"4d2ffd282929303e855066378909503f6c1f38695df2efb5c7b34e7b6bdf147b"}
Oct 30 13:26:37.875185 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:37.875153    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv"
Oct 30 13:26:37.975780 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:37.975725    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7739d642-eb43-4056-ae8b-ace5f85e9155-tls-certs\" (UniqueName: \"kubernetes.io/secret/7739d642-eb43-4056-ae8b-ace5f85e9155-tls-certs\") pod \"7739d642-eb43-4056-ae8b-ace5f85e9155\" (UID: \"7739d642-eb43-4056-ae8b-ace5f85e9155\") "
Oct 30 13:26:37.976355 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:37.975907    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7739d642-eb43-4056-ae8b-ace5f85e9155-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7739d642-eb43-4056-ae8b-ace5f85e9155-dshm\") pod \"7739d642-eb43-4056-ae8b-ace5f85e9155\" (UID: \"7739d642-eb43-4056-ae8b-ace5f85e9155\") "
Oct 30 13:26:37.976355 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:37.976056    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7739d642-eb43-4056-ae8b-ace5f85e9155-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7739d642-eb43-4056-ae8b-ace5f85e9155-model-cache\") pod \"7739d642-eb43-4056-ae8b-ace5f85e9155\" (UID: \"7739d642-eb43-4056-ae8b-ace5f85e9155\") "
Oct 30 13:26:37.976355 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:37.976122    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7739d642-eb43-4056-ae8b-ace5f85e9155-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7739d642-eb43-4056-ae8b-ace5f85e9155-tmp-dir\") pod \"7739d642-eb43-4056-ae8b-ace5f85e9155\" (UID: \"7739d642-eb43-4056-ae8b-ace5f85e9155\") "
Oct 30 13:26:37.976355 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:37.976178    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7739d642-eb43-4056-ae8b-ace5f85e9155-home\" (UniqueName: \"kubernetes.io/empty-dir/7739d642-eb43-4056-ae8b-ace5f85e9155-home\") pod \"7739d642-eb43-4056-ae8b-ace5f85e9155\" (UID: \"7739d642-eb43-4056-ae8b-ace5f85e9155\") "
Oct 30 13:26:37.976355 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:37.976252    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7739d642-eb43-4056-ae8b-ace5f85e9155-kube-api-access-dms82\" (UniqueName: \"kubernetes.io/projected/7739d642-eb43-4056-ae8b-ace5f85e9155-kube-api-access-dms82\") pod \"7739d642-eb43-4056-ae8b-ace5f85e9155\" (UID: \"7739d642-eb43-4056-ae8b-ace5f85e9155\") "
Oct 30 13:26:37.976682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:37.976363    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7739d642-eb43-4056-ae8b-ace5f85e9155-model-cache" pod "7739d642-eb43-4056-ae8b-ace5f85e9155" (UID: "7739d642-eb43-4056-ae8b-ace5f85e9155"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:37.976682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:37.976481    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7739d642-eb43-4056-ae8b-ace5f85e9155-tmp-dir" pod "7739d642-eb43-4056-ae8b-ace5f85e9155" (UID: "7739d642-eb43-4056-ae8b-ace5f85e9155"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:37.976803 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:37.976754    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7739d642-eb43-4056-ae8b-ace5f85e9155-home" pod "7739d642-eb43-4056-ae8b-ace5f85e9155" (UID: "7739d642-eb43-4056-ae8b-ace5f85e9155"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:37.977009 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:37.976988    4544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7739d642-eb43-4056-ae8b-ace5f85e9155-home\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:26:37.977113 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:37.977014    4544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7739d642-eb43-4056-ae8b-ace5f85e9155-model-cache\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:26:37.977113 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:37.977031    4544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7739d642-eb43-4056-ae8b-ace5f85e9155-tmp-dir\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:26:37.979281 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:37.979206    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7739d642-eb43-4056-ae8b-ace5f85e9155-dshm" pod "7739d642-eb43-4056-ae8b-ace5f85e9155" (UID: "7739d642-eb43-4056-ae8b-ace5f85e9155"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:37.979392 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:37.979272    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7739d642-eb43-4056-ae8b-ace5f85e9155-tls-certs" pod "7739d642-eb43-4056-ae8b-ace5f85e9155" (UID: "7739d642-eb43-4056-ae8b-ace5f85e9155"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:26:37.979392 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:37.979298    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7739d642-eb43-4056-ae8b-ace5f85e9155-kube-api-access-dms82" pod "7739d642-eb43-4056-ae8b-ace5f85e9155" (UID: "7739d642-eb43-4056-ae8b-ace5f85e9155"). InnerVolumeSpecName "kube-api-access-dms82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:38.078609 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:38.078568    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dms82\" (UniqueName: \"kubernetes.io/projected/7739d642-eb43-4056-ae8b-ace5f85e9155-kube-api-access-dms82\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:26:38.078609 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:38.078603    4544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7739d642-eb43-4056-ae8b-ace5f85e9155-tls-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:26:38.078609 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:38.078614    4544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7739d642-eb43-4056-ae8b-ace5f85e9155-dshm\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:26:38.734130 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:38.734097    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv"
Oct 30 13:26:38.734355 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:38.734137    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv" event={"ID":"7739d642-eb43-4056-ae8b-ace5f85e9155","Type":"ContainerDied","Data":"a6037d5de8c23879c1db1e81f4ec1b41c4a743625c7612b702b280c7a4a24ae0"}
Oct 30 13:26:38.734355 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:38.734196    4544 scope.go:122] "RemoveContainer" containerID="4d2ffd282929303e855066378909503f6c1f38695df2efb5c7b34e7b6bdf147b"
Oct 30 13:26:38.790539 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:38.790497    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv"]
Oct 30 13:26:38.801938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:38.801898    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-0e874e/auth-test-model-0e874e-kserve-b784968df-2wwfv"]
Oct 30 13:26:38.988107 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:38.988020    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:39.089339 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:39.089306    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7739d642-eb43-4056-ae8b-ace5f85e9155" path="/var/lib/kubelet/pods/7739d642-eb43-4056-ae8b-ace5f85e9155/volumes"
Oct 30 13:26:40.835108 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:40.835077    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:42.567575 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:42.567522    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:44.513092 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:44.513057    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:45.965505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:45.965462    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:46.600056 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:46.600019    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:46.662776 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:46.662737    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s"]
Oct 30 13:26:46.663600 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:46.663583    4544 cpu_manager.go:414] "containerMap: removing container" podUID="7739d642-eb43-4056-ae8b-ace5f85e9155" containerName="main"
Oct 30 13:26:46.663662 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:46.663602    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7739d642-eb43-4056-ae8b-ace5f85e9155" containerName="main"
Oct 30 13:26:46.663662 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:46.663627    4544 cpu_manager.go:414] "containerMap: removing container" podUID="c6998926-73ac-4828-91b7-d453b5ebc33e" containerName="main"
Oct 30 13:26:46.663662 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:46.663633    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6998926-73ac-4828-91b7-d453b5ebc33e" containerName="main"
Oct 30 13:26:46.663773 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:46.663670    4544 cpu_manager.go:414] "containerMap: removing container" podUID="21ffd071-46da-4940-879a-2877eca7d1fe" containerName="main"
Oct 30 13:26:46.663773 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:46.663676    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21ffd071-46da-4940-879a-2877eca7d1fe" containerName="main"
Oct 30 13:26:46.663844 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:46.663783    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="21ffd071-46da-4940-879a-2877eca7d1fe" containerName="main"
Oct 30 13:26:46.663844 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:46.663796    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="7739d642-eb43-4056-ae8b-ace5f85e9155" containerName="main"
Oct 30 13:26:46.663844 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:46.663805    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6998926-73ac-4828-91b7-d453b5ebc33e" containerName="main"
Oct 30 13:26:46.669165 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:46.669142    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s"
Oct 30 13:26:46.697799 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:46.697769    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-96ab73-serving-cert\""
Oct 30 13:26:46.702753 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:46.702716    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s"]
Oct 30 13:26:46.728376 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:46.728340    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:46.758136 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:46.758090    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f3ee96f-30cf-4afe-a5a7-0fb69beaa873-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s\" (UID: \"3f3ee96f-30cf-4afe-a5a7-0fb69beaa873\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s"
Oct 30 13:26:46.758360 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:46.758342    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s7wc\" (UniqueName: \"kubernetes.io/projected/3f3ee96f-30cf-4afe-a5a7-0fb69beaa873-kube-api-access-4s7wc\") pod \"maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s\" (UID: \"3f3ee96f-30cf-4afe-a5a7-0fb69beaa873\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s"
Oct 30 13:26:46.758417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:46.758378    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f3ee96f-30cf-4afe-a5a7-0fb69beaa873-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s\" (UID: \"3f3ee96f-30cf-4afe-a5a7-0fb69beaa873\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s"
Oct 30 13:26:46.758527 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:46.758509    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f3ee96f-30cf-4afe-a5a7-0fb69beaa873-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s\" (UID: \"3f3ee96f-30cf-4afe-a5a7-0fb69beaa873\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s"
Oct 30 13:26:46.872760 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:46.872648    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s7wc\" (UniqueName: \"kubernetes.io/projected/3f3ee96f-30cf-4afe-a5a7-0fb69beaa873-kube-api-access-4s7wc\") pod \"maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s\" (UID: \"3f3ee96f-30cf-4afe-a5a7-0fb69beaa873\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s"
Oct 30 13:26:46.891398 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:46.891363    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f3ee96f-30cf-4afe-a5a7-0fb69beaa873-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s\" (UID: \"3f3ee96f-30cf-4afe-a5a7-0fb69beaa873\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s"
Oct 30 13:26:46.891824 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:46.891796    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f3ee96f-30cf-4afe-a5a7-0fb69beaa873-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s\" (UID: \"3f3ee96f-30cf-4afe-a5a7-0fb69beaa873\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s"
Oct 30 13:26:46.892045 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:46.892021    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f3ee96f-30cf-4afe-a5a7-0fb69beaa873-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s\" (UID: \"3f3ee96f-30cf-4afe-a5a7-0fb69beaa873\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s"
Oct 30 13:26:46.903289 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:46.903023    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:46.907100 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:46.905305    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f3ee96f-30cf-4afe-a5a7-0fb69beaa873-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s\" (UID: \"3f3ee96f-30cf-4afe-a5a7-0fb69beaa873\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s"
Oct 30 13:26:46.923605 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:46.923477    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f3ee96f-30cf-4afe-a5a7-0fb69beaa873-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s\" (UID: \"3f3ee96f-30cf-4afe-a5a7-0fb69beaa873\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s"
Oct 30 13:26:46.925329 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:46.925300    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s7wc\" (UniqueName: \"kubernetes.io/projected/3f3ee96f-30cf-4afe-a5a7-0fb69beaa873-kube-api-access-4s7wc\") pod \"maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s\" (UID: \"3f3ee96f-30cf-4afe-a5a7-0fb69beaa873\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s"
Oct 30 13:26:46.925989 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:46.925960    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f3ee96f-30cf-4afe-a5a7-0fb69beaa873-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s\" (UID: \"3f3ee96f-30cf-4afe-a5a7-0fb69beaa873\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s"
Oct 30 13:26:46.984293 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:46.984206    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s"
Oct 30 13:26:47.283193 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:47.283102    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:47.305233 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:47.305182    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s"]
Oct 30 13:26:47.306919 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:26:47.306889    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f3ee96f_30cf_4afe_a5a7_0fb69beaa873.slice/crio-c3f69de9ac5f6d416a5de34bc8c6f05ff2a4bb4a3b2c29d2feb78a89a9b41d6f WatchSource:0}: Error finding container c3f69de9ac5f6d416a5de34bc8c6f05ff2a4bb4a3b2c29d2feb78a89a9b41d6f: Status 404 returned error can't find the container with id c3f69de9ac5f6d416a5de34bc8c6f05ff2a4bb4a3b2c29d2feb78a89a9b41d6f
Oct 30 13:26:47.827971 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:47.827882    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s" event={"ID":"3f3ee96f-30cf-4afe-a5a7-0fb69beaa873","Type":"ContainerStarted","Data":"d1c06a039a0f077cdaedeeff7456afd0d3b441a87950853445915b2ad18ff9a4"}
Oct 30 13:26:47.828184 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:47.828022    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s"
Oct 30 13:26:47.828184 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:47.828058    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s" event={"ID":"3f3ee96f-30cf-4afe-a5a7-0fb69beaa873","Type":"ContainerStarted","Data":"c3f69de9ac5f6d416a5de34bc8c6f05ff2a4bb4a3b2c29d2feb78a89a9b41d6f"}
Oct 30 13:26:47.896568 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:47.896504    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s" podStartSLOduration=1.498982235 podStartE2EDuration="1.896485818s" totalImagesPullingTime="397.503583ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:47.892318409 +0000 UTC m=+2502.939452360" watchObservedRunningTime="2026-10-30 13:26:47.896485818 +0000 UTC m=+2502.943619770"
Oct 30 13:26:48.904583 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:48.904545    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:49.781410 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:49.781360    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm"]
Oct 30 13:26:49.787664 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:49.787628    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm"
Oct 30 13:26:49.807598 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:49.807563    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm"]
Oct 30 13:26:49.817268 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:49.814507    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-e9d641\"/\"openshift-service-ca.crt\""
Oct 30 13:26:49.817268 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:49.814536    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-e9d641\"/\"kube-root-ca.crt\""
Oct 30 13:26:49.817268 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:49.815167    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-e9d641\"/\"default-dockercfg-rwklk\""
Oct 30 13:26:49.817268 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:49.815247    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-e9d641\"/\"sub-test-model-e9d641-kserve-self-signed-certs\""
Oct 30 13:26:49.831610 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:49.831574    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:49.832031 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:49.832005    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9af76539-ed44-4913-92cd-be61edf063ad-tls-certs\") pod \"sub-test-model-e9d641-kserve-594cb445dc-mc7jm\" (UID: \"9af76539-ed44-4913-92cd-be61edf063ad\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm"
Oct 30 13:26:49.832095 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:49.832042    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9af76539-ed44-4913-92cd-be61edf063ad-dshm\") pod \"sub-test-model-e9d641-kserve-594cb445dc-mc7jm\" (UID: \"9af76539-ed44-4913-92cd-be61edf063ad\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm"
Oct 30 13:26:49.832141 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:49.832119    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9af76539-ed44-4913-92cd-be61edf063ad-home\") pod \"sub-test-model-e9d641-kserve-594cb445dc-mc7jm\" (UID: \"9af76539-ed44-4913-92cd-be61edf063ad\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm"
Oct 30 13:26:49.832255 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:49.832215    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5wst\" (UniqueName: \"kubernetes.io/projected/9af76539-ed44-4913-92cd-be61edf063ad-kube-api-access-x5wst\") pod \"sub-test-model-e9d641-kserve-594cb445dc-mc7jm\" (UID: \"9af76539-ed44-4913-92cd-be61edf063ad\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm"
Oct 30 13:26:49.832312 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:49.832271    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9af76539-ed44-4913-92cd-be61edf063ad-model-cache\") pod \"sub-test-model-e9d641-kserve-594cb445dc-mc7jm\" (UID: \"9af76539-ed44-4913-92cd-be61edf063ad\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm"
Oct 30 13:26:49.832312 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:49.832300    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9af76539-ed44-4913-92cd-be61edf063ad-tmp-dir\") pod \"sub-test-model-e9d641-kserve-594cb445dc-mc7jm\" (UID: \"9af76539-ed44-4913-92cd-be61edf063ad\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm"
Oct 30 13:26:49.933833 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:49.933794    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9af76539-ed44-4913-92cd-be61edf063ad-tls-certs\") pod \"sub-test-model-e9d641-kserve-594cb445dc-mc7jm\" (UID: \"9af76539-ed44-4913-92cd-be61edf063ad\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm"
Oct 30 13:26:49.933833 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:49.933835    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9af76539-ed44-4913-92cd-be61edf063ad-dshm\") pod \"sub-test-model-e9d641-kserve-594cb445dc-mc7jm\" (UID: \"9af76539-ed44-4913-92cd-be61edf063ad\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm"
Oct 30 13:26:49.934467 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:49.934010    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9af76539-ed44-4913-92cd-be61edf063ad-home\") pod \"sub-test-model-e9d641-kserve-594cb445dc-mc7jm\" (UID: \"9af76539-ed44-4913-92cd-be61edf063ad\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm"
Oct 30 13:26:49.934467 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:49.934124    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5wst\" (UniqueName: \"kubernetes.io/projected/9af76539-ed44-4913-92cd-be61edf063ad-kube-api-access-x5wst\") pod \"sub-test-model-e9d641-kserve-594cb445dc-mc7jm\" (UID: \"9af76539-ed44-4913-92cd-be61edf063ad\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm"
Oct 30 13:26:49.934467 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:49.934163    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9af76539-ed44-4913-92cd-be61edf063ad-model-cache\") pod \"sub-test-model-e9d641-kserve-594cb445dc-mc7jm\" (UID: \"9af76539-ed44-4913-92cd-be61edf063ad\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm"
Oct 30 13:26:49.934467 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:49.934342    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9af76539-ed44-4913-92cd-be61edf063ad-tmp-dir\") pod \"sub-test-model-e9d641-kserve-594cb445dc-mc7jm\" (UID: \"9af76539-ed44-4913-92cd-be61edf063ad\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm"
Oct 30 13:26:49.934467 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:49.934410    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9af76539-ed44-4913-92cd-be61edf063ad-home\") pod \"sub-test-model-e9d641-kserve-594cb445dc-mc7jm\" (UID: \"9af76539-ed44-4913-92cd-be61edf063ad\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm"
Oct 30 13:26:49.934763 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:49.934482    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9af76539-ed44-4913-92cd-be61edf063ad-model-cache\") pod \"sub-test-model-e9d641-kserve-594cb445dc-mc7jm\" (UID: \"9af76539-ed44-4913-92cd-be61edf063ad\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm"
Oct 30 13:26:49.934763 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:49.934666    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9af76539-ed44-4913-92cd-be61edf063ad-tmp-dir\") pod \"sub-test-model-e9d641-kserve-594cb445dc-mc7jm\" (UID: \"9af76539-ed44-4913-92cd-be61edf063ad\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm"
Oct 30 13:26:49.936478 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:49.936453    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9af76539-ed44-4913-92cd-be61edf063ad-dshm\") pod \"sub-test-model-e9d641-kserve-594cb445dc-mc7jm\" (UID: \"9af76539-ed44-4913-92cd-be61edf063ad\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm"
Oct 30 13:26:49.936727 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:49.936708    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9af76539-ed44-4913-92cd-be61edf063ad-tls-certs\") pod \"sub-test-model-e9d641-kserve-594cb445dc-mc7jm\" (UID: \"9af76539-ed44-4913-92cd-be61edf063ad\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm"
Oct 30 13:26:49.957974 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:49.957934    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5wst\" (UniqueName: \"kubernetes.io/projected/9af76539-ed44-4913-92cd-be61edf063ad-kube-api-access-x5wst\") pod \"sub-test-model-e9d641-kserve-594cb445dc-mc7jm\" (UID: \"9af76539-ed44-4913-92cd-be61edf063ad\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm"
Oct 30 13:26:50.118241 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:50.116552    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm"
Oct 30 13:26:50.460074 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:50.459977    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm"]
Oct 30 13:26:50.471020 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:26:50.470985    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9af76539_ed44_4913_92cd_be61edf063ad.slice/crio-d90cba6968971d63caf215506aafd09f6ca376df9aedf0701dcd0c652cc5257b WatchSource:0}: Error finding container d90cba6968971d63caf215506aafd09f6ca376df9aedf0701dcd0c652cc5257b: Status 404 returned error can't find the container with id d90cba6968971d63caf215506aafd09f6ca376df9aedf0701dcd0c652cc5257b
Oct 30 13:26:50.853407 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:50.853362    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:50.863795 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:50.863751    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm" event={"ID":"9af76539-ed44-4913-92cd-be61edf063ad","Type":"ContainerStarted","Data":"7d449d4a016f4324d251233efa7e27663c0234c16046d426897be656605926cc"}
Oct 30 13:26:50.864038 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:50.863814    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm" event={"ID":"9af76539-ed44-4913-92cd-be61edf063ad","Type":"ContainerStarted","Data":"d90cba6968971d63caf215506aafd09f6ca376df9aedf0701dcd0c652cc5257b"}
Oct 30 13:26:50.933710 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:50.933637    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm" podStartSLOduration=1.9336181959999998 podStartE2EDuration="1.933618196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:50.925152096 +0000 UTC m=+2505.972286071" watchObservedRunningTime="2026-10-30 13:26:50.933618196 +0000 UTC m=+2505.980752149"
Oct 30 13:26:51.118399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:51.118302    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm"
Oct 30 13:26:52.707120 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:52.707082    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:53.844314 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:53.844276    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s"
Oct 30 13:26:54.279150 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:54.279049    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:54.431758 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:54.431717    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:54.677610 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:54.677560    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"]
Oct 30 13:26:54.677960 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:54.677930    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d" podUID="ed9d8ff2-2175-4747-a154-d803bb7b54f8" containerName="istio-proxy" containerID="cri-o://9e4dac145ea7dee8fe37327650f8acaa27afe024af64c862036c61dc36b57ac5" gracePeriod=30
Oct 30 13:26:54.696124 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:54.696085    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:54.827244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:54.826336    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:55.547310 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:55.547271    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:56.743471 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:56.743436    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:56.868267 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:56.865291    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:59.009268 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:26:59.009215    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:26:59.354397 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:59.354337    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:26:59.963407 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:59.963363    4544 generic.go:356] "Generic (PLEG): container finished" podID="ed9d8ff2-2175-4747-a154-d803bb7b54f8" containerID="9e4dac145ea7dee8fe37327650f8acaa27afe024af64c862036c61dc36b57ac5" exitCode=0
Oct 30 13:26:59.963633 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:59.963436    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d" event={"ID":"ed9d8ff2-2175-4747-a154-d803bb7b54f8","Type":"ContainerDied","Data":"9e4dac145ea7dee8fe37327650f8acaa27afe024af64c862036c61dc36b57ac5"}
Oct 30 13:26:59.963633 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:59.963494    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d" event={"ID":"ed9d8ff2-2175-4747-a154-d803bb7b54f8","Type":"ContainerDied","Data":"e6cafc150bff70d02143e927402efa952fc7cccbc42096464ec7884cd8709781"}
Oct 30 13:26:59.963633 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:59.963512    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6cafc150bff70d02143e927402efa952fc7cccbc42096464ec7884cd8709781"
Oct 30 13:26:59.963633 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:59.963528    4544 scope.go:122] "RemoveContainer" containerID="f856dbfc8cb97f7624945ddbe1fa45919223e6a1bc2abb87c9888021dfeee483"
Oct 30 13:26:59.979726 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:26:59.979695    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:27:00.057375 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:00.057319    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istio-podinfo\") pod \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") "
Oct 30 13:27:00.058479 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:00.057383    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-workload-socket\") pod \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") "
Oct 30 13:27:00.058479 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:00.057435    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istio-data\") pod \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") "
Oct 30 13:27:00.058479 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:00.057531    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istio-envoy\") pod \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") "
Oct 30 13:27:00.058479 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:00.057576    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istiod-ca-cert\") pod \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") "
Oct 30 13:27:00.058479 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:00.057694    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed9d8ff2-2175-4747-a154-d803bb7b54f8-kube-api-access-vccxv\" (UniqueName: \"kubernetes.io/projected/ed9d8ff2-2175-4747-a154-d803bb7b54f8-kube-api-access-vccxv\") pod \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") "
Oct 30 13:27:00.058479 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:00.057740    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-workload-certs\") pod \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") "
Oct 30 13:27:00.058479 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:00.057830    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-credential-socket\") pod \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") "
Oct 30 13:27:00.058479 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:00.057935    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istio-token\" (UniqueName: \"kubernetes.io/projected/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istio-token\") pod \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\" (UID: \"ed9d8ff2-2175-4747-a154-d803bb7b54f8\") "
Oct 30 13:27:00.059815 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:00.059391    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istio-data" pod "ed9d8ff2-2175-4747-a154-d803bb7b54f8" (UID: "ed9d8ff2-2175-4747-a154-d803bb7b54f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:00.059815 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:00.059622    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-workload-certs" pod "ed9d8ff2-2175-4747-a154-d803bb7b54f8" (UID: "ed9d8ff2-2175-4747-a154-d803bb7b54f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:00.059815 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:00.059739    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-credential-socket" pod "ed9d8ff2-2175-4747-a154-d803bb7b54f8" (UID: "ed9d8ff2-2175-4747-a154-d803bb7b54f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:00.060072 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:00.059939    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istiod-ca-cert" pod "ed9d8ff2-2175-4747-a154-d803bb7b54f8" (UID: "ed9d8ff2-2175-4747-a154-d803bb7b54f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:27:00.060447 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:00.060273    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-workload-socket" pod "ed9d8ff2-2175-4747-a154-d803bb7b54f8" (UID: "ed9d8ff2-2175-4747-a154-d803bb7b54f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:00.063814 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:00.063772    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istio-token" pod "ed9d8ff2-2175-4747-a154-d803bb7b54f8" (UID: "ed9d8ff2-2175-4747-a154-d803bb7b54f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:00.063960 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:00.063774    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istio-envoy" pod "ed9d8ff2-2175-4747-a154-d803bb7b54f8" (UID: "ed9d8ff2-2175-4747-a154-d803bb7b54f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:00.064328 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:00.064296    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istio-podinfo" pod "ed9d8ff2-2175-4747-a154-d803bb7b54f8" (UID: "ed9d8ff2-2175-4747-a154-d803bb7b54f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:27:00.064468 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:00.064348    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed9d8ff2-2175-4747-a154-d803bb7b54f8-kube-api-access-vccxv" pod "ed9d8ff2-2175-4747-a154-d803bb7b54f8" (UID: "ed9d8ff2-2175-4747-a154-d803bb7b54f8"). InnerVolumeSpecName "kube-api-access-vccxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:00.118299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:00.118254    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm"
Oct 30 13:27:00.132611 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:00.132575    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm"
Oct 30 13:27:00.160675 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:00.160643    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:27:00.160675 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:00.160670    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:27:00.160675 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:00.160682    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vccxv\" (UniqueName: \"kubernetes.io/projected/ed9d8ff2-2175-4747-a154-d803bb7b54f8-kube-api-access-vccxv\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:27:00.160943 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:00.160696    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:27:00.160943 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:00.160708    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:27:00.160943 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:00.160717    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:27:00.160943 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:00.160725    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:27:00.160943 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:00.160733    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:27:00.160943 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:00.160742    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9d8ff2-2175-4747-a154-d803bb7b54f8-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:27:00.993641 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:00.993607    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"
Oct 30 13:27:01.008594 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:01.008555    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm"
Oct 30 13:27:01.054963 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:01.054924    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-578524af-ht57r"]
Oct 30 13:27:01.073891 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:01.073854    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-578524af-ht57r"]
Oct 30 13:27:01.089003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:01.088959    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de9588c5-4d8b-48b1-a56b-f7d8b2a15463" path="/var/lib/kubelet/pods/de9588c5-4d8b-48b1-a56b-f7d8b2a15463/volumes"
Oct 30 13:27:01.146674 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:01.146635    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"]
Oct 30 13:27:01.149122 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:01.149080    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d"]
Oct 30 13:27:01.577292 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:01.577252    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:02.978673 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:02.978633    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:03.130710 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:03.130647    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed9d8ff2-2175-4747-a154-d803bb7b54f8" path="/var/lib/kubelet/pods/ed9d8ff2-2175-4747-a154-d803bb7b54f8/volumes"
Oct 30 13:27:03.445841 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:03.445804    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:05.391522 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:05.391478    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:05.775944 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:05.775862    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd"]
Oct 30 13:27:05.950933 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:05.950896    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:06.083196 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:06.083167    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:07.197694 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:07.197661    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:09.008429 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:09.008381    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:10.777200 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:10.777144    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd" podUID="05c58d32-2ee6-4197-a19c-8431668ca3eb" containerName="payload-processing" containerID="cri-o://6ed969f4bc2fa0939d9d0fe8be01690b2aa3ba9a9ad5fb6c0fb7c950c0ceed7a" gracePeriod=25
Oct 30 13:27:10.796966 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:10.796885    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:11.071795 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:11.071762    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd"
Oct 30 13:27:11.176032 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:11.175993    4544 generic.go:356] "Generic (PLEG): container finished" podID="05c58d32-2ee6-4197-a19c-8431668ca3eb" containerID="6ed969f4bc2fa0939d9d0fe8be01690b2aa3ba9a9ad5fb6c0fb7c950c0ceed7a" exitCode=0
Oct 30 13:27:11.176282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:11.176040    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd"
Oct 30 13:27:11.176282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:11.176077    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd" event={"ID":"05c58d32-2ee6-4197-a19c-8431668ca3eb","Type":"ContainerDied","Data":"6ed969f4bc2fa0939d9d0fe8be01690b2aa3ba9a9ad5fb6c0fb7c950c0ceed7a"}
Oct 30 13:27:11.176282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:11.176134    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd" event={"ID":"05c58d32-2ee6-4197-a19c-8431668ca3eb","Type":"ContainerDied","Data":"6b92ab558300e3fa64865863d44e2a3f5ecafa3f871d87b021032055c8db3c39"}
Oct 30 13:27:11.176282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:11.176160    4544 scope.go:122] "RemoveContainer" containerID="6ed969f4bc2fa0939d9d0fe8be01690b2aa3ba9a9ad5fb6c0fb7c950c0ceed7a"
Oct 30 13:27:11.177137 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:11.177114    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05c58d32-2ee6-4197-a19c-8431668ca3eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c58d32-2ee6-4197-a19c-8431668ca3eb-plugins-config-volume\") pod \"05c58d32-2ee6-4197-a19c-8431668ca3eb\" (UID: \"05c58d32-2ee6-4197-a19c-8431668ca3eb\") "
Oct 30 13:27:11.177318 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:11.177255    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05c58d32-2ee6-4197-a19c-8431668ca3eb-kube-api-access-qbrbt\" (UniqueName: \"kubernetes.io/projected/05c58d32-2ee6-4197-a19c-8431668ca3eb-kube-api-access-qbrbt\") pod \"05c58d32-2ee6-4197-a19c-8431668ca3eb\" (UID: \"05c58d32-2ee6-4197-a19c-8431668ca3eb\") "
Oct 30 13:27:11.177544 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:11.177524    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05c58d32-2ee6-4197-a19c-8431668ca3eb-plugins-config-volume" pod "05c58d32-2ee6-4197-a19c-8431668ca3eb" (UID: "05c58d32-2ee6-4197-a19c-8431668ca3eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:27:11.178087 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:11.178064    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c58d32-2ee6-4197-a19c-8431668ca3eb-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:27:11.179988 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:11.179965    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c58d32-2ee6-4197-a19c-8431668ca3eb-kube-api-access-qbrbt" pod "05c58d32-2ee6-4197-a19c-8431668ca3eb" (UID: "05c58d32-2ee6-4197-a19c-8431668ca3eb"). InnerVolumeSpecName "kube-api-access-qbrbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:11.200089 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:11.200055    4544 scope.go:122] "RemoveContainer" containerID="6ed969f4bc2fa0939d9d0fe8be01690b2aa3ba9a9ad5fb6c0fb7c950c0ceed7a"
Oct 30 13:27:11.200480 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:27:11.200452    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ed969f4bc2fa0939d9d0fe8be01690b2aa3ba9a9ad5fb6c0fb7c950c0ceed7a\": container with ID starting with 6ed969f4bc2fa0939d9d0fe8be01690b2aa3ba9a9ad5fb6c0fb7c950c0ceed7a not found: ID does not exist" containerID="6ed969f4bc2fa0939d9d0fe8be01690b2aa3ba9a9ad5fb6c0fb7c950c0ceed7a"
Oct 30 13:27:11.200561 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:11.200493    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ed969f4bc2fa0939d9d0fe8be01690b2aa3ba9a9ad5fb6c0fb7c950c0ceed7a"} err="failed to get container status \"6ed969f4bc2fa0939d9d0fe8be01690b2aa3ba9a9ad5fb6c0fb7c950c0ceed7a\": rpc error: code = NotFound desc = could not find container \"6ed969f4bc2fa0939d9d0fe8be01690b2aa3ba9a9ad5fb6c0fb7c950c0ceed7a\": container with ID starting with 6ed969f4bc2fa0939d9d0fe8be01690b2aa3ba9a9ad5fb6c0fb7c950c0ceed7a not found: ID does not exist"
Oct 30 13:27:11.279214 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:11.279148    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbrbt\" (UniqueName: \"kubernetes.io/projected/05c58d32-2ee6-4197-a19c-8431668ca3eb-kube-api-access-qbrbt\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:27:11.526864 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:11.526817    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd"]
Oct 30 13:27:11.538771 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:11.538731    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd"]
Oct 30 13:27:12.605062 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:12.605027    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:13.090716 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:13.090673    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05c58d32-2ee6-4197-a19c-8431668ca3eb" path="/var/lib/kubelet/pods/05c58d32-2ee6-4197-a19c-8431668ca3eb/volumes"
Oct 30 13:27:13.628248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:13.628189    4544 scope.go:122] "RemoveContainer" containerID="6e0aaf3cf0c7dd43cd3db6c159314a7eb63ea0d3a83fa2e5e67b2eaf4c028a94"
Oct 30 13:27:13.639762 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:13.639734    4544 scope.go:122] "RemoveContainer" containerID="2a3912b63efae2656d38f4f4aa0f39bdc8a89032d91c3a54e0da275ada1333c9"
Oct 30 13:27:13.650592 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:13.650567    4544 scope.go:122] "RemoveContainer" containerID="aaaeb82775ca79364dd59bb6104b50d7f433b28a057cbc9620133b3b1053a7e1"
Oct 30 13:27:13.661112 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:13.661090    4544 scope.go:122] "RemoveContainer" containerID="ccfd0efc4145503f874ce92d9810c3bee3aa12d124bae96772c5c12202dcf655"
Oct 30 13:27:13.672277 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:13.672202    4544 scope.go:122] "RemoveContainer" containerID="9e4dac145ea7dee8fe37327650f8acaa27afe024af64c862036c61dc36b57ac5"
Oct 30 13:27:14.033292 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:14.033191    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5b82fd8-trfr8"]
Oct 30 13:27:14.045979 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:14.045943    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5b82fd8-trfr8"]
Oct 30 13:27:14.378771 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:14.378730    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:15.095825 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:15.095783    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b68f691-8b13-4422-9f40-4461ee58128d" path="/var/lib/kubelet/pods/1b68f691-8b13-4422-9f40-4461ee58128d/volumes"
Oct 30 13:27:16.317347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:16.317314    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:17.335028 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:17.334986    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:18.430721 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:18.430685    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:20.532343 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:20.532304    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:22.535347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:22.535310    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:24.301612 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:24.301574    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:25.980843 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:25.980803    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:26.360960 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:26.360921    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:27.183501 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.183449    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:27.337893 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.337855    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:27.583878 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.583800    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"]
Oct 30 13:27:27.584742 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.584720    4544 cpu_manager.go:414] "containerMap: removing container" podUID="05c58d32-2ee6-4197-a19c-8431668ca3eb" containerName="payload-processing"
Oct 30 13:27:27.584742 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.584742    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05c58d32-2ee6-4197-a19c-8431668ca3eb" containerName="payload-processing"
Oct 30 13:27:27.584957 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.584763    4544 cpu_manager.go:414] "containerMap: removing container" podUID="ed9d8ff2-2175-4747-a154-d803bb7b54f8" containerName="istio-proxy"
Oct 30 13:27:27.584957 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.584770    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed9d8ff2-2175-4747-a154-d803bb7b54f8" containerName="istio-proxy"
Oct 30 13:27:27.584957 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.584788    4544 cpu_manager.go:414] "containerMap: removing container" podUID="ed9d8ff2-2175-4747-a154-d803bb7b54f8" containerName="istio-proxy"
Oct 30 13:27:27.584957 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.584796    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed9d8ff2-2175-4747-a154-d803bb7b54f8" containerName="istio-proxy"
Oct 30 13:27:27.585175 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.584984    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed9d8ff2-2175-4747-a154-d803bb7b54f8" containerName="istio-proxy"
Oct 30 13:27:27.585175 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.585001    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed9d8ff2-2175-4747-a154-d803bb7b54f8" containerName="istio-proxy"
Oct 30 13:27:27.585175 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.585025    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="05c58d32-2ee6-4197-a19c-8431668ca3eb" containerName="payload-processing"
Oct 30 13:27:27.589617 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.589596    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:27:27.610576 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.610529    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-7a71ca-openshift-default-dockercfg-n9nrg\""
Oct 30 13:27:27.654430 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.654393    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"]
Oct 30 13:27:27.696956 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.696906    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/347e2f12-0089-42aa-86b8-d42d02a56175-istio-token\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:27:27.697182 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.697050    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-credential-socket\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:27:27.697182 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.697112    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-istio-envoy\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:27:27.697346 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.697258    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwl2g\" (UniqueName: \"kubernetes.io/projected/347e2f12-0089-42aa-86b8-d42d02a56175-kube-api-access-gwl2g\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:27:27.697432 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.697411    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/347e2f12-0089-42aa-86b8-d42d02a56175-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:27:27.697485 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.697451    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/347e2f12-0089-42aa-86b8-d42d02a56175-istio-podinfo\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:27:27.697485 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.697480    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-istio-data\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:27:27.697627 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.697611    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-workload-socket\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:27:27.697671 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.697652    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-workload-certs\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:27:27.798895 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.798857    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-credential-socket\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:27:27.799109 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.798910    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-istio-envoy\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:27:27.799109 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.798943    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwl2g\" (UniqueName: \"kubernetes.io/projected/347e2f12-0089-42aa-86b8-d42d02a56175-kube-api-access-gwl2g\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:27:27.799109 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.799026    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/347e2f12-0089-42aa-86b8-d42d02a56175-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:27:27.799109 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.799050    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/347e2f12-0089-42aa-86b8-d42d02a56175-istio-podinfo\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:27:27.799109 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.799079    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-istio-data\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:27:27.799455 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.799161    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-workload-socket\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:27:27.799455 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.799208    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-workload-certs\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:27:27.799455 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.799279    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/347e2f12-0089-42aa-86b8-d42d02a56175-istio-token\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:27:27.799455 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.799386    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-credential-socket\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:27:27.799711 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.799687    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-workload-socket\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:27:27.799887 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.799803    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/347e2f12-0089-42aa-86b8-d42d02a56175-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:27:27.800106 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.800083    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-istio-data\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:27:27.800277 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.800259    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-workload-certs\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:27:27.801485 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.801463    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-istio-envoy\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:27:27.801889 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.801868    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/347e2f12-0089-42aa-86b8-d42d02a56175-istio-podinfo\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:27:27.848393 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.848309    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwl2g\" (UniqueName: \"kubernetes.io/projected/347e2f12-0089-42aa-86b8-d42d02a56175-kube-api-access-gwl2g\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:27:27.849957 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.849927    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/347e2f12-0089-42aa-86b8-d42d02a56175-istio-token\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:27:27.905050 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:27.905013    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:27:28.006947 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:28.006888    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp"]
Oct 30 13:27:28.059798 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:28.059756    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96"]
Oct 30 13:27:28.249563 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:28.249462    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:28.381295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:28.381165    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"]
Oct 30 13:27:28.384922 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:27:28.384894    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod347e2f12_0089_42aa_86b8_d42d02a56175.slice/crio-f9c209eac3ae95a3156acffcd481f20fc2feb2b56955675f01e9d44c1cad0122 WatchSource:0}: Error finding container f9c209eac3ae95a3156acffcd481f20fc2feb2b56955675f01e9d44c1cad0122: Status 404 returned error can't find the container with id f9c209eac3ae95a3156acffcd481f20fc2feb2b56955675f01e9d44c1cad0122
Oct 30 13:27:28.387773 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:28.387727    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:27:28.387884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:28.387830    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:27:28.387884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:28.387874    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:27:28.399168 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:28.398664    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm" event={"ID":"347e2f12-0089-42aa-86b8-d42d02a56175","Type":"ContainerStarted","Data":"f9c209eac3ae95a3156acffcd481f20fc2feb2b56955675f01e9d44c1cad0122"}
Oct 30 13:27:28.470779 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:28.470743    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:28.808934 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:28.808898    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:29.166294 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:27:29.164074    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:27:29.410378 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:29.410337    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm" event={"ID":"347e2f12-0089-42aa-86b8-d42d02a56175","Type":"ContainerStarted","Data":"33153249a7f6d8b9013d6842dd92d84c7c343ae48ab96cfc53ff0eba3e23fca1"}
Oct 30 13:27:29.475373 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:29.475080    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm" podStartSLOduration=2.475064317 podStartE2EDuration="2.475064317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:29.471679919 +0000 UTC m=+2544.518813873" watchObservedRunningTime="2026-10-30 13:27:29.475064317 +0000 UTC m=+2544.522198267"
Oct 30 13:27:29.905709 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:29.905674    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:27:29.907614 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:29.907586    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 13:27:29.907753 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:29.907659    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm" podUID="347e2f12-0089-42aa-86b8-d42d02a56175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 13:27:30.732248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:30.730139    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:30.905603 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:30.905567    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 13:27:30.905850 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:30.905627    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm" podUID="347e2f12-0089-42aa-86b8-d42d02a56175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 13:27:31.906194 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:31.906156    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 13:27:31.906729 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:31.906251    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm" podUID="347e2f12-0089-42aa-86b8-d42d02a56175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 13:27:32.694989 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:32.694951    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:32.905877 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:32.905841    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 13:27:32.906068 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:32.905917    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm" podUID="347e2f12-0089-42aa-86b8-d42d02a56175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 13:27:33.007321 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:33.007212    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp" podUID="9a9a0392-a7ea-4d31-add2-f4faf60c121b" containerName="payload-processing" containerID="cri-o://24d18cb96a8ece8aea454d2374a2cf341423ab88939b9a9c4bbab66d1e2fcc06" gracePeriod=25
Oct 30 13:27:33.061112 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:33.061062    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96" podUID="e443ff59-e6ec-444d-817a-1b7baaa5faf7" containerName="payload-pre-processing" containerID="cri-o://d9002011d7106ce68108af211ad6c4b8a3e5b73f1b2ee4b2f52077579674310e" gracePeriod=25
Oct 30 13:27:33.905427 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:33.905395    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 13:27:33.905617 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:33.905457    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm" podUID="347e2f12-0089-42aa-86b8-d42d02a56175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 13:27:34.905639 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:34.905599    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 13:27:34.906188 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:34.905669    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm" podUID="347e2f12-0089-42aa-86b8-d42d02a56175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 13:27:35.015480 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:35.015429    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:35.275155 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:35.275053    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:35.911435 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:35.911393    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:27:35.911935 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:35.911914    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:27:35.912870 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:35.912849    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:27:36.414411 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:36.414369    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:27:36.414616 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:36.414456    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt" podUID="4e66eb26-5727-4ad8-881f-64b6711fe440" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:27:36.820517 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:36.820427    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:37.626452 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:37.626360    4544 generic.go:356] "Generic (PLEG): container finished" podID="4e66eb26-5727-4ad8-881f-64b6711fe440" containerID="af461ba3c237d09537277a0321974f745885b1401311091e4ff9e15a5550f79a" exitCode=0
Oct 30 13:27:37.626452 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:37.626437    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt" event={"ID":"4e66eb26-5727-4ad8-881f-64b6711fe440","Type":"ContainerDied","Data":"af461ba3c237d09537277a0321974f745885b1401311091e4ff9e15a5550f79a"}
Oct 30 13:27:37.626972 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:37.626889    4544 scope.go:122] "RemoveContainer" containerID="af461ba3c237d09537277a0321974f745885b1401311091e4ff9e15a5550f79a"
Oct 30 13:27:37.628550 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:37.628515    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:27:37.628683 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:37.628608    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:27:37.628683 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:37.628659    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:27:37.982067 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:37.981986    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:38.296960 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:38.296879    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47"]
Oct 30 13:27:38.297119 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:38.297094    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47" podUID="6705f8d7-884a-4bf2-921b-e4656dfae77d" containerName="maas-api" containerID="cri-o://0aa5195bbce1b8ad4de5e90ef43c784f95a9b23770b6edcb1853ef7af94bd7fa" gracePeriod=30
Oct 30 13:27:38.396024 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:38.395991    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:38.509202 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:38.509167    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:38.673892 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:38.673564    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47"
Oct 30 13:27:38.680422 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:38.680388    4544 generic.go:356] "Generic (PLEG): container finished" podID="6705f8d7-884a-4bf2-921b-e4656dfae77d" containerID="0aa5195bbce1b8ad4de5e90ef43c784f95a9b23770b6edcb1853ef7af94bd7fa" exitCode=0
Oct 30 13:27:38.680636 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:38.680447    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47"
Oct 30 13:27:38.680636 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:38.680478    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47" event={"ID":"6705f8d7-884a-4bf2-921b-e4656dfae77d","Type":"ContainerDied","Data":"0aa5195bbce1b8ad4de5e90ef43c784f95a9b23770b6edcb1853ef7af94bd7fa"}
Oct 30 13:27:38.680636 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:38.680539    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47" event={"ID":"6705f8d7-884a-4bf2-921b-e4656dfae77d","Type":"ContainerDied","Data":"926c4a5f6754175914773d39b5708d87d8347b969e053406b14d4417e97da24f"}
Oct 30 13:27:38.680636 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:38.680573    4544 scope.go:122] "RemoveContainer" containerID="0aa5195bbce1b8ad4de5e90ef43c784f95a9b23770b6edcb1853ef7af94bd7fa"
Oct 30 13:27:38.683291 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:38.683255    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt" event={"ID":"4e66eb26-5727-4ad8-881f-64b6711fe440","Type":"ContainerStarted","Data":"b70ca727e64d4d8c689bfdf170699feeb415ee43f6b7336ed5fef90ec1d55285"}
Oct 30 13:27:38.693078 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:38.693051    4544 scope.go:122] "RemoveContainer" containerID="0aa5195bbce1b8ad4de5e90ef43c784f95a9b23770b6edcb1853ef7af94bd7fa"
Oct 30 13:27:38.693470 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:27:38.693447    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aa5195bbce1b8ad4de5e90ef43c784f95a9b23770b6edcb1853ef7af94bd7fa\": container with ID starting with 0aa5195bbce1b8ad4de5e90ef43c784f95a9b23770b6edcb1853ef7af94bd7fa not found: ID does not exist" containerID="0aa5195bbce1b8ad4de5e90ef43c784f95a9b23770b6edcb1853ef7af94bd7fa"
Oct 30 13:27:38.693539 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:38.693479    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aa5195bbce1b8ad4de5e90ef43c784f95a9b23770b6edcb1853ef7af94bd7fa"} err="failed to get container status \"0aa5195bbce1b8ad4de5e90ef43c784f95a9b23770b6edcb1853ef7af94bd7fa\": rpc error: code = NotFound desc = could not find container \"0aa5195bbce1b8ad4de5e90ef43c784f95a9b23770b6edcb1853ef7af94bd7fa\": container with ID starting with 0aa5195bbce1b8ad4de5e90ef43c784f95a9b23770b6edcb1853ef7af94bd7fa not found: ID does not exist"
Oct 30 13:27:38.736114 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:38.736077    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 13:27:38.736719 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:38.736388    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt" podUID="4e66eb26-5727-4ad8-881f-64b6711fe440" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 13:27:38.737403 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:38.737367    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:27:38.868439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:38.867786    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6705f8d7-884a-4bf2-921b-e4656dfae77d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6705f8d7-884a-4bf2-921b-e4656dfae77d-maas-api-tls\") pod \"6705f8d7-884a-4bf2-921b-e4656dfae77d\" (UID: \"6705f8d7-884a-4bf2-921b-e4656dfae77d\") "
Oct 30 13:27:38.868439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:38.867877    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6705f8d7-884a-4bf2-921b-e4656dfae77d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6705f8d7-884a-4bf2-921b-e4656dfae77d-openshift-service-ca\") pod \"6705f8d7-884a-4bf2-921b-e4656dfae77d\" (UID: \"6705f8d7-884a-4bf2-921b-e4656dfae77d\") "
Oct 30 13:27:38.868439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:38.867983    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6705f8d7-884a-4bf2-921b-e4656dfae77d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6705f8d7-884a-4bf2-921b-e4656dfae77d-metrics-certs\") pod \"6705f8d7-884a-4bf2-921b-e4656dfae77d\" (UID: \"6705f8d7-884a-4bf2-921b-e4656dfae77d\") "
Oct 30 13:27:38.868439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:38.868038    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6705f8d7-884a-4bf2-921b-e4656dfae77d-kube-api-access-jr8vc\" (UniqueName: \"kubernetes.io/projected/6705f8d7-884a-4bf2-921b-e4656dfae77d-kube-api-access-jr8vc\") pod \"6705f8d7-884a-4bf2-921b-e4656dfae77d\" (UID: \"6705f8d7-884a-4bf2-921b-e4656dfae77d\") "
Oct 30 13:27:38.869881 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:38.869851    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6705f8d7-884a-4bf2-921b-e4656dfae77d-openshift-service-ca" pod "6705f8d7-884a-4bf2-921b-e4656dfae77d" (UID: "6705f8d7-884a-4bf2-921b-e4656dfae77d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:27:38.873589 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:38.872181    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6705f8d7-884a-4bf2-921b-e4656dfae77d-metrics-certs" pod "6705f8d7-884a-4bf2-921b-e4656dfae77d" (UID: "6705f8d7-884a-4bf2-921b-e4656dfae77d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:27:38.873815 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:38.873785    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6705f8d7-884a-4bf2-921b-e4656dfae77d-kube-api-access-jr8vc" pod "6705f8d7-884a-4bf2-921b-e4656dfae77d" (UID: "6705f8d7-884a-4bf2-921b-e4656dfae77d"). InnerVolumeSpecName "kube-api-access-jr8vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:38.874281 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:38.874259    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6705f8d7-884a-4bf2-921b-e4656dfae77d-maas-api-tls" pod "6705f8d7-884a-4bf2-921b-e4656dfae77d" (UID: "6705f8d7-884a-4bf2-921b-e4656dfae77d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:27:38.972348 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:38.972310    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6705f8d7-884a-4bf2-921b-e4656dfae77d-maas-api-tls\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:27:38.972348 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:38.972345    4544 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6705f8d7-884a-4bf2-921b-e4656dfae77d-openshift-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:27:38.972616 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:38.972359    4544 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6705f8d7-884a-4bf2-921b-e4656dfae77d-metrics-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:27:38.972616 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:38.972371    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr8vc\" (UniqueName: \"kubernetes.io/projected/6705f8d7-884a-4bf2-921b-e4656dfae77d-kube-api-access-jr8vc\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:27:39.009870 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:39.009838    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:39.054832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:39.054765    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47"]
Oct 30 13:27:39.175770 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:39.175685    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f3338e-5cbf88f8bd-jkj47"]
Oct 30 13:27:39.194968 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:39.194931    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:39.332246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:39.327183    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:39.445306 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:39.445177    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:39.571670 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:39.571630    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:39.736309 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:39.735667    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 13:27:39.736309 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:39.735752    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt" podUID="4e66eb26-5727-4ad8-881f-64b6711fe440" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 13:27:39.878426 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:39.878387    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:40.821246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:40.786525    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 13:27:40.821246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:40.786606    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt" podUID="4e66eb26-5727-4ad8-881f-64b6711fe440" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 13:27:40.958986 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:40.958936    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96" podUID="e443ff59-e6ec-444d-817a-1b7baaa5faf7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:27:40.958986 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:40.958977    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp" podUID="9a9a0392-a7ea-4d31-add2-f4faf60c121b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:27:41.089148 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:41.089113    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6705f8d7-884a-4bf2-921b-e4656dfae77d" path="/var/lib/kubelet/pods/6705f8d7-884a-4bf2-921b-e4656dfae77d/volumes"
Oct 30 13:27:41.584673 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:41.584634    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:41.737736 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:41.737695    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 13:27:41.737950 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:41.737783    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt" podUID="4e66eb26-5727-4ad8-881f-64b6711fe440" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 13:27:42.738785 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:42.738741    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 13:27:42.739286 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:42.738814    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt" podUID="4e66eb26-5727-4ad8-881f-64b6711fe440" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 13:27:42.781868 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:42.781827    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:42.938380 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:42.938345    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:43.682205 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:43.682172    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:43.737736 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:43.737697    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 13:27:43.737939 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:43.737768    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt" podUID="4e66eb26-5727-4ad8-881f-64b6711fe440" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 13:27:44.045387 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:44.045300    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29501dc6-fbmrr"]
Oct 30 13:27:44.057573 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:44.057536    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29501dc6-fbmrr"]
Oct 30 13:27:44.556707 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:44.556665    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:44.682744 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:44.682707    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:44.738595 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:44.738561    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:27:44.738923 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:44.738893    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:27:44.739641 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:44.739620    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:27:44.808379 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:44.808301    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:44.884550 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:44.884517    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565"]
Oct 30 13:27:44.885452 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:44.885432    4544 cpu_manager.go:414] "containerMap: removing container" podUID="6705f8d7-884a-4bf2-921b-e4656dfae77d" containerName="maas-api"
Oct 30 13:27:44.885452 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:44.885453    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6705f8d7-884a-4bf2-921b-e4656dfae77d" containerName="maas-api"
Oct 30 13:27:44.885709 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:44.885694    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="6705f8d7-884a-4bf2-921b-e4656dfae77d" containerName="maas-api"
Oct 30 13:27:44.891194 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:44.891170    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565"
Oct 30 13:27:44.908070 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:44.906297    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565"]
Oct 30 13:27:44.908070 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:44.906552    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-8a5d5e-dockercfg-9h6tg\""
Oct 30 13:27:44.908070 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:44.906610    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-8a5d5e\""
Oct 30 13:27:44.943712 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:44.943672    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565\" (UID: \"e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565"
Oct 30 13:27:44.943913 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:44.943851    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8c2v\" (UniqueName: \"kubernetes.io/projected/e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6-kube-api-access-g8c2v\") pod \"payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565\" (UID: \"e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565"
Oct 30 13:27:45.070295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:45.069988    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss"]
Oct 30 13:27:45.082244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:45.078337    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565\" (UID: \"e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565"
Oct 30 13:27:45.082244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:45.078555    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8c2v\" (UniqueName: \"kubernetes.io/projected/e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6-kube-api-access-g8c2v\") pod \"payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565\" (UID: \"e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565"
Oct 30 13:27:45.082244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:45.080083    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565\" (UID: \"e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565"
Oct 30 13:27:45.141306 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:45.141193    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8c2v\" (UniqueName: \"kubernetes.io/projected/e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6-kube-api-access-g8c2v\") pod \"payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565\" (UID: \"e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565"
Oct 30 13:27:45.169647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:45.169528    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss"]
Oct 30 13:27:45.171305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:45.171206    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss"
Oct 30 13:27:45.197328 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:45.197293    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f132dc3a-c383-44dc-a5c0-39d18dbea7bf" path="/var/lib/kubelet/pods/f132dc3a-c383-44dc-a5c0-39d18dbea7bf/volumes"
Oct 30 13:27:45.209761 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:45.209731    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565"
Oct 30 13:27:45.259719 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:45.259681    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:45.283632 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:45.283602    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df66313-8135-4c91-a387-e7f654c305b3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss\" (UID: \"6df66313-8135-4c91-a387-e7f654c305b3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss"
Oct 30 13:27:45.283768 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:45.283750    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlh69\" (UniqueName: \"kubernetes.io/projected/6df66313-8135-4c91-a387-e7f654c305b3-kube-api-access-hlh69\") pod \"payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss\" (UID: \"6df66313-8135-4c91-a387-e7f654c305b3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss"
Oct 30 13:27:45.385156 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:45.385114    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlh69\" (UniqueName: \"kubernetes.io/projected/6df66313-8135-4c91-a387-e7f654c305b3-kube-api-access-hlh69\") pod \"payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss\" (UID: \"6df66313-8135-4c91-a387-e7f654c305b3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss"
Oct 30 13:27:45.385417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:45.385321    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df66313-8135-4c91-a387-e7f654c305b3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss\" (UID: \"6df66313-8135-4c91-a387-e7f654c305b3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss"
Oct 30 13:27:45.386023 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:45.386003    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df66313-8135-4c91-a387-e7f654c305b3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss\" (UID: \"6df66313-8135-4c91-a387-e7f654c305b3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss"
Oct 30 13:27:45.409284 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:45.409216    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlh69\" (UniqueName: \"kubernetes.io/projected/6df66313-8135-4c91-a387-e7f654c305b3-kube-api-access-hlh69\") pod \"payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss\" (UID: \"6df66313-8135-4c91-a387-e7f654c305b3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss"
Oct 30 13:27:45.507131 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:45.507094    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss"
Oct 30 13:27:45.521920 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:45.521887    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565"]
Oct 30 13:27:45.525072 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:27:45.525041    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c0f591_0ec0_47cd_b6cf_dc7cc295c9f6.slice/crio-1a724a64be5632fbab3512548e3435303327a1ebc6b123a21dfeb6cb707ffec6 WatchSource:0}: Error finding container 1a724a64be5632fbab3512548e3435303327a1ebc6b123a21dfeb6cb707ffec6: Status 404 returned error can't find the container with id 1a724a64be5632fbab3512548e3435303327a1ebc6b123a21dfeb6cb707ffec6
Oct 30 13:27:45.820736 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:45.820711    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss"]
Oct 30 13:27:45.823011 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:27:45.822981    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df66313_8135_4c91_a387_e7f654c305b3.slice/crio-abd5b09a1a47f441279ae55c32f94902579b5922c779ad403310bbe6a6f98be6 WatchSource:0}: Error finding container abd5b09a1a47f441279ae55c32f94902579b5922c779ad403310bbe6a6f98be6: Status 404 returned error can't find the container with id abd5b09a1a47f441279ae55c32f94902579b5922c779ad403310bbe6a6f98be6
Oct 30 13:27:45.928932 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:45.928898    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss" event={"ID":"6df66313-8135-4c91-a387-e7f654c305b3","Type":"ContainerStarted","Data":"6127dbb1dca4c9019a739a59b3933d00ee6b4c3b39138042277c4e7cbed1c58f"}
Oct 30 13:27:45.929098 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:45.928958    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss" event={"ID":"6df66313-8135-4c91-a387-e7f654c305b3","Type":"ContainerStarted","Data":"abd5b09a1a47f441279ae55c32f94902579b5922c779ad403310bbe6a6f98be6"}
Oct 30 13:27:45.929098 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:45.928987    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss"
Oct 30 13:27:45.930797 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:45.930771    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565" event={"ID":"e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6","Type":"ContainerStarted","Data":"cda9cfa5cedb6520e5c106b6c5dc7b1c48f0a02707aa8af26d1fdf7c2394d80e"}
Oct 30 13:27:45.930951 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:45.930815    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565" event={"ID":"e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6","Type":"ContainerStarted","Data":"1a724a64be5632fbab3512548e3435303327a1ebc6b123a21dfeb6cb707ffec6"}
Oct 30 13:27:45.930951 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:45.930837    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565"
Oct 30 13:27:45.972940 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:45.972878    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss" podStartSLOduration=1.972863553 podStartE2EDuration="1.972863553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:45.970325751 +0000 UTC m=+2561.017459704" watchObservedRunningTime="2026-10-30 13:27:45.972863553 +0000 UTC m=+2561.019997505"
Oct 30 13:27:46.011691 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:46.011642    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565" podStartSLOduration=2.011629326 podStartE2EDuration="2.011629326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:46.0104093 +0000 UTC m=+2561.057543254" watchObservedRunningTime="2026-10-30 13:27:46.011629326 +0000 UTC m=+2561.058763278"
Oct 30 13:27:46.654892 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:46.654855    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:46.925604 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:46.925495    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:47.180462 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:47.180373    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:47.463340 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:47.460112    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:49.034681 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:49.034643    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:50.877140 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:50.877058    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:50.959292 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:50.959254    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96" podUID="e443ff59-e6ec-444d-817a-1b7baaa5faf7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:27:50.959455 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:50.959296    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp" podUID="9a9a0392-a7ea-4d31-add2-f4faf60c121b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:27:51.194339 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:51.194252    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4"]
Oct 30 13:27:51.199760 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:51.199735    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4"
Oct 30 13:27:51.220133 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:51.220100    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4"]
Oct 30 13:27:51.226142 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:51.226114    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-51288b\"/\"default-dockercfg-vqgb5\""
Oct 30 13:27:51.226320 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:51.226202    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-51288b\"/\"auth-test-model-51288b-kserve-self-signed-certs\""
Oct 30 13:27:51.228584 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:51.228563    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-51288b\"/\"openshift-service-ca.crt\""
Oct 30 13:27:51.228819 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:51.228801    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-51288b\"/\"kube-root-ca.crt\""
Oct 30 13:27:51.257418 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:51.257386    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:51.279910 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:51.279876    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bb9d35b-2245-418e-b133-e9007ee9ba66-dshm\") pod \"auth-test-model-51288b-kserve-5844f5b57c-fm8h4\" (UID: \"7bb9d35b-2245-418e-b133-e9007ee9ba66\") " pod="ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4"
Oct 30 13:27:51.280074 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:51.279993    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bb9d35b-2245-418e-b133-e9007ee9ba66-home\") pod \"auth-test-model-51288b-kserve-5844f5b57c-fm8h4\" (UID: \"7bb9d35b-2245-418e-b133-e9007ee9ba66\") " pod="ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4"
Oct 30 13:27:51.280157 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:51.280107    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bb9d35b-2245-418e-b133-e9007ee9ba66-tls-certs\") pod \"auth-test-model-51288b-kserve-5844f5b57c-fm8h4\" (UID: \"7bb9d35b-2245-418e-b133-e9007ee9ba66\") " pod="ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4"
Oct 30 13:27:51.280210 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:51.280185    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swdkv\" (UniqueName: \"kubernetes.io/projected/7bb9d35b-2245-418e-b133-e9007ee9ba66-kube-api-access-swdkv\") pod \"auth-test-model-51288b-kserve-5844f5b57c-fm8h4\" (UID: \"7bb9d35b-2245-418e-b133-e9007ee9ba66\") " pod="ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4"
Oct 30 13:27:51.280289 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:51.280212    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bb9d35b-2245-418e-b133-e9007ee9ba66-model-cache\") pod \"auth-test-model-51288b-kserve-5844f5b57c-fm8h4\" (UID: \"7bb9d35b-2245-418e-b133-e9007ee9ba66\") " pod="ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4"
Oct 30 13:27:51.280289 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:51.280258    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bb9d35b-2245-418e-b133-e9007ee9ba66-tmp-dir\") pod \"auth-test-model-51288b-kserve-5844f5b57c-fm8h4\" (UID: \"7bb9d35b-2245-418e-b133-e9007ee9ba66\") " pod="ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4"
Oct 30 13:27:51.382089 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:51.382047    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bb9d35b-2245-418e-b133-e9007ee9ba66-tls-certs\") pod \"auth-test-model-51288b-kserve-5844f5b57c-fm8h4\" (UID: \"7bb9d35b-2245-418e-b133-e9007ee9ba66\") " pod="ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4"
Oct 30 13:27:51.382333 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:51.382149    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swdkv\" (UniqueName: \"kubernetes.io/projected/7bb9d35b-2245-418e-b133-e9007ee9ba66-kube-api-access-swdkv\") pod \"auth-test-model-51288b-kserve-5844f5b57c-fm8h4\" (UID: \"7bb9d35b-2245-418e-b133-e9007ee9ba66\") " pod="ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4"
Oct 30 13:27:51.382333 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:51.382188    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bb9d35b-2245-418e-b133-e9007ee9ba66-model-cache\") pod \"auth-test-model-51288b-kserve-5844f5b57c-fm8h4\" (UID: \"7bb9d35b-2245-418e-b133-e9007ee9ba66\") " pod="ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4"
Oct 30 13:27:51.382333 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:51.382277    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bb9d35b-2245-418e-b133-e9007ee9ba66-tmp-dir\") pod \"auth-test-model-51288b-kserve-5844f5b57c-fm8h4\" (UID: \"7bb9d35b-2245-418e-b133-e9007ee9ba66\") " pod="ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4"
Oct 30 13:27:51.382533 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:51.382399    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bb9d35b-2245-418e-b133-e9007ee9ba66-dshm\") pod \"auth-test-model-51288b-kserve-5844f5b57c-fm8h4\" (UID: \"7bb9d35b-2245-418e-b133-e9007ee9ba66\") " pod="ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4"
Oct 30 13:27:51.382533 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:51.382478    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bb9d35b-2245-418e-b133-e9007ee9ba66-home\") pod \"auth-test-model-51288b-kserve-5844f5b57c-fm8h4\" (UID: \"7bb9d35b-2245-418e-b133-e9007ee9ba66\") " pod="ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4"
Oct 30 13:27:51.382726 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:51.382705    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bb9d35b-2245-418e-b133-e9007ee9ba66-tmp-dir\") pod \"auth-test-model-51288b-kserve-5844f5b57c-fm8h4\" (UID: \"7bb9d35b-2245-418e-b133-e9007ee9ba66\") " pod="ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4"
Oct 30 13:27:51.382795 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:51.382747    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bb9d35b-2245-418e-b133-e9007ee9ba66-model-cache\") pod \"auth-test-model-51288b-kserve-5844f5b57c-fm8h4\" (UID: \"7bb9d35b-2245-418e-b133-e9007ee9ba66\") " pod="ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4"
Oct 30 13:27:51.382886 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:51.382865    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bb9d35b-2245-418e-b133-e9007ee9ba66-home\") pod \"auth-test-model-51288b-kserve-5844f5b57c-fm8h4\" (UID: \"7bb9d35b-2245-418e-b133-e9007ee9ba66\") " pod="ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4"
Oct 30 13:27:51.384604 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:51.384578    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bb9d35b-2245-418e-b133-e9007ee9ba66-dshm\") pod \"auth-test-model-51288b-kserve-5844f5b57c-fm8h4\" (UID: \"7bb9d35b-2245-418e-b133-e9007ee9ba66\") " pod="ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4"
Oct 30 13:27:51.384847 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:51.384826    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bb9d35b-2245-418e-b133-e9007ee9ba66-tls-certs\") pod \"auth-test-model-51288b-kserve-5844f5b57c-fm8h4\" (UID: \"7bb9d35b-2245-418e-b133-e9007ee9ba66\") " pod="ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4"
Oct 30 13:27:51.405535 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:51.405510    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swdkv\" (UniqueName: \"kubernetes.io/projected/7bb9d35b-2245-418e-b133-e9007ee9ba66-kube-api-access-swdkv\") pod \"auth-test-model-51288b-kserve-5844f5b57c-fm8h4\" (UID: \"7bb9d35b-2245-418e-b133-e9007ee9ba66\") " pod="ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4"
Oct 30 13:27:51.525297 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:51.521764    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4"
Oct 30 13:27:51.843032 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:51.843004    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4"]
Oct 30 13:27:51.845392 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:27:51.845360    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7bb9d35b_2245_418e_b133_e9007ee9ba66.slice/crio-612786a172faef8a8425a7b91fa953115f0480aa3100cd5e081c59e5002743ff WatchSource:0}: Error finding container 612786a172faef8a8425a7b91fa953115f0480aa3100cd5e081c59e5002743ff: Status 404 returned error can't find the container with id 612786a172faef8a8425a7b91fa953115f0480aa3100cd5e081c59e5002743ff
Oct 30 13:27:51.982112 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:51.982067    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4" event={"ID":"7bb9d35b-2245-418e-b133-e9007ee9ba66","Type":"ContainerStarted","Data":"250b243f654a6d11ffa191bf9a673b51475999fafa5356351fc940f198d89e05"}
Oct 30 13:27:51.982625 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:51.982124    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4" event={"ID":"7bb9d35b-2245-418e-b133-e9007ee9ba66","Type":"ContainerStarted","Data":"612786a172faef8a8425a7b91fa953115f0480aa3100cd5e081c59e5002743ff"}
Oct 30 13:27:52.025264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:52.025183    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4" podStartSLOduration=1.02516706 podStartE2EDuration="1.02516706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:52.022789987 +0000 UTC m=+2567.069923941" watchObservedRunningTime="2026-10-30 13:27:52.02516706 +0000 UTC m=+2567.072301017"
Oct 30 13:27:52.350804 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:52.350770    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:52.472047 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:52.472014    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:52.523212 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:52.523182    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4"
Oct 30 13:27:52.646901 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:52.646794    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:52.693068 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:52.693033    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"]
Oct 30 13:27:52.693409 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:52.693329    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7" podUID="f08cecf0-f7ec-4d03-8168-8b35a8446ba4" containerName="istio-proxy" containerID="cri-o://0759e23d488af4acd49ab4173f96c35fe2f1e3d2bad81a2d4a6afb92535ab652" gracePeriod=30
Oct 30 13:27:52.800583 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:52.797529    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:53.363581 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:53.363147    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp"
Oct 30 13:27:53.410711 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:53.410682    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96"
Oct 30 13:27:53.465464 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:53.465429    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e443ff59-e6ec-444d-817a-1b7baaa5faf7-kube-api-access-59m4v\" (UniqueName: \"kubernetes.io/projected/e443ff59-e6ec-444d-817a-1b7baaa5faf7-kube-api-access-59m4v\") pod \"e443ff59-e6ec-444d-817a-1b7baaa5faf7\" (UID: \"e443ff59-e6ec-444d-817a-1b7baaa5faf7\") "
Oct 30 13:27:53.465679 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:53.465499    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e443ff59-e6ec-444d-817a-1b7baaa5faf7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e443ff59-e6ec-444d-817a-1b7baaa5faf7-plugins-config-volume\") pod \"e443ff59-e6ec-444d-817a-1b7baaa5faf7\" (UID: \"e443ff59-e6ec-444d-817a-1b7baaa5faf7\") "
Oct 30 13:27:53.465756 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:53.465697    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a9a0392-a7ea-4d31-add2-f4faf60c121b-kube-api-access-xt44z\" (UniqueName: \"kubernetes.io/projected/9a9a0392-a7ea-4d31-add2-f4faf60c121b-kube-api-access-xt44z\") pod \"9a9a0392-a7ea-4d31-add2-f4faf60c121b\" (UID: \"9a9a0392-a7ea-4d31-add2-f4faf60c121b\") "
Oct 30 13:27:53.465756 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:53.465739    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a9a0392-a7ea-4d31-add2-f4faf60c121b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a9a0392-a7ea-4d31-add2-f4faf60c121b-plugins-config-volume\") pod \"9a9a0392-a7ea-4d31-add2-f4faf60c121b\" (UID: \"9a9a0392-a7ea-4d31-add2-f4faf60c121b\") "
Oct 30 13:27:53.467417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:53.467378    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e443ff59-e6ec-444d-817a-1b7baaa5faf7-plugins-config-volume" pod "e443ff59-e6ec-444d-817a-1b7baaa5faf7" (UID: "e443ff59-e6ec-444d-817a-1b7baaa5faf7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:27:53.470849 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:53.470068    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e443ff59-e6ec-444d-817a-1b7baaa5faf7-kube-api-access-59m4v" pod "e443ff59-e6ec-444d-817a-1b7baaa5faf7" (UID: "e443ff59-e6ec-444d-817a-1b7baaa5faf7"). InnerVolumeSpecName "kube-api-access-59m4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:53.473370 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:53.473131    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a9a0392-a7ea-4d31-add2-f4faf60c121b-plugins-config-volume" pod "9a9a0392-a7ea-4d31-add2-f4faf60c121b" (UID: "9a9a0392-a7ea-4d31-add2-f4faf60c121b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:27:53.474690 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:53.474663    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a9a0392-a7ea-4d31-add2-f4faf60c121b-kube-api-access-xt44z" pod "9a9a0392-a7ea-4d31-add2-f4faf60c121b" (UID: "9a9a0392-a7ea-4d31-add2-f4faf60c121b"). InnerVolumeSpecName "kube-api-access-xt44z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:53.567277 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:53.567171    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59m4v\" (UniqueName: \"kubernetes.io/projected/e443ff59-e6ec-444d-817a-1b7baaa5faf7-kube-api-access-59m4v\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:27:53.567277 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:53.567201    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e443ff59-e6ec-444d-817a-1b7baaa5faf7-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:27:53.567277 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:53.567213    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt44z\" (UniqueName: \"kubernetes.io/projected/9a9a0392-a7ea-4d31-add2-f4faf60c121b-kube-api-access-xt44z\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:27:53.567277 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:53.567253    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a9a0392-a7ea-4d31-add2-f4faf60c121b-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:27:54.000166 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:54.000128    4544 generic.go:356] "Generic (PLEG): container finished" podID="e443ff59-e6ec-444d-817a-1b7baaa5faf7" containerID="d9002011d7106ce68108af211ad6c4b8a3e5b73f1b2ee4b2f52077579674310e" exitCode=0
Oct 30 13:27:54.000166 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:54.000168    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96"
Oct 30 13:27:54.000490 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:54.000209    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96" event={"ID":"e443ff59-e6ec-444d-817a-1b7baaa5faf7","Type":"ContainerDied","Data":"d9002011d7106ce68108af211ad6c4b8a3e5b73f1b2ee4b2f52077579674310e"}
Oct 30 13:27:54.000490 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:54.000299    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96" event={"ID":"e443ff59-e6ec-444d-817a-1b7baaa5faf7","Type":"ContainerDied","Data":"f098e396907497c6478cc6e52ce9a4f73ce40bab1fd7621b80e2a52d5a30eca7"}
Oct 30 13:27:54.000490 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:54.000334    4544 scope.go:122] "RemoveContainer" containerID="d9002011d7106ce68108af211ad6c4b8a3e5b73f1b2ee4b2f52077579674310e"
Oct 30 13:27:54.001929 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:54.001903    4544 generic.go:356] "Generic (PLEG): container finished" podID="9a9a0392-a7ea-4d31-add2-f4faf60c121b" containerID="24d18cb96a8ece8aea454d2374a2cf341423ab88939b9a9c4bbab66d1e2fcc06" exitCode=0
Oct 30 13:27:54.002061 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:54.001956    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp"
Oct 30 13:27:54.002061 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:54.001977    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp" event={"ID":"9a9a0392-a7ea-4d31-add2-f4faf60c121b","Type":"ContainerDied","Data":"24d18cb96a8ece8aea454d2374a2cf341423ab88939b9a9c4bbab66d1e2fcc06"}
Oct 30 13:27:54.002061 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:54.002030    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp" event={"ID":"9a9a0392-a7ea-4d31-add2-f4faf60c121b","Type":"ContainerDied","Data":"5a7deae3f22606a66de62363c74fce150a647829726e5403b09d2e2539b8b676"}
Oct 30 13:27:54.014916 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:54.014896    4544 scope.go:122] "RemoveContainer" containerID="d9002011d7106ce68108af211ad6c4b8a3e5b73f1b2ee4b2f52077579674310e"
Oct 30 13:27:54.015287 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:27:54.015257    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9002011d7106ce68108af211ad6c4b8a3e5b73f1b2ee4b2f52077579674310e\": container with ID starting with d9002011d7106ce68108af211ad6c4b8a3e5b73f1b2ee4b2f52077579674310e not found: ID does not exist" containerID="d9002011d7106ce68108af211ad6c4b8a3e5b73f1b2ee4b2f52077579674310e"
Oct 30 13:27:54.015401 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:54.015300    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9002011d7106ce68108af211ad6c4b8a3e5b73f1b2ee4b2f52077579674310e"} err="failed to get container status \"d9002011d7106ce68108af211ad6c4b8a3e5b73f1b2ee4b2f52077579674310e\": rpc error: code = NotFound desc = could not find container \"d9002011d7106ce68108af211ad6c4b8a3e5b73f1b2ee4b2f52077579674310e\": container with ID starting with d9002011d7106ce68108af211ad6c4b8a3e5b73f1b2ee4b2f52077579674310e not found: ID does not exist"
Oct 30 13:27:54.015401 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:54.015327    4544 scope.go:122] "RemoveContainer" containerID="24d18cb96a8ece8aea454d2374a2cf341423ab88939b9a9c4bbab66d1e2fcc06"
Oct 30 13:27:54.028620 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:54.028599    4544 scope.go:122] "RemoveContainer" containerID="24d18cb96a8ece8aea454d2374a2cf341423ab88939b9a9c4bbab66d1e2fcc06"
Oct 30 13:27:54.028903 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:27:54.028883    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24d18cb96a8ece8aea454d2374a2cf341423ab88939b9a9c4bbab66d1e2fcc06\": container with ID starting with 24d18cb96a8ece8aea454d2374a2cf341423ab88939b9a9c4bbab66d1e2fcc06 not found: ID does not exist" containerID="24d18cb96a8ece8aea454d2374a2cf341423ab88939b9a9c4bbab66d1e2fcc06"
Oct 30 13:27:54.028955 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:54.028912    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24d18cb96a8ece8aea454d2374a2cf341423ab88939b9a9c4bbab66d1e2fcc06"} err="failed to get container status \"24d18cb96a8ece8aea454d2374a2cf341423ab88939b9a9c4bbab66d1e2fcc06\": rpc error: code = NotFound desc = could not find container \"24d18cb96a8ece8aea454d2374a2cf341423ab88939b9a9c4bbab66d1e2fcc06\": container with ID starting with 24d18cb96a8ece8aea454d2374a2cf341423ab88939b9a9c4bbab66d1e2fcc06 not found: ID does not exist"
Oct 30 13:27:54.075034 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:54.075006    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp"]
Oct 30 13:27:54.086560 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:54.086531    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp"]
Oct 30 13:27:54.112989 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:54.112962    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96"]
Oct 30 13:27:54.116756 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:54.116730    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96"]
Oct 30 13:27:54.717789 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:54.717743    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:55.089541 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:55.089504    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a9a0392-a7ea-4d31-add2-f4faf60c121b" path="/var/lib/kubelet/pods/9a9a0392-a7ea-4d31-add2-f4faf60c121b/volumes"
Oct 30 13:27:55.090072 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:55.090053    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e443ff59-e6ec-444d-817a-1b7baaa5faf7" path="/var/lib/kubelet/pods/e443ff59-e6ec-444d-817a-1b7baaa5faf7/volumes"
Oct 30 13:27:56.776945 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:56.776911    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:56.885803 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:56.885749    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:56.942267 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:56.942211    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss"
Oct 30 13:27:56.942684 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:56.942663    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565"
Oct 30 13:27:57.192815 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:57.192776    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:57.989136 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:57.989107    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:27:58.037521 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.037480    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istio-podinfo\") pod \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") "
Oct 30 13:27:58.037730 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.037562    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-workload-socket\") pod \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") "
Oct 30 13:27:58.037730 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.037632    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istiod-ca-cert\") pod \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") "
Oct 30 13:27:58.037730 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.037687    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istio-envoy\") pod \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") "
Oct 30 13:27:58.038052 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.037783    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-workload-certs\") pod \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") "
Oct 30 13:27:58.038052 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.037847    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-workload-socket" pod "f08cecf0-f7ec-4d03-8168-8b35a8446ba4" (UID: "f08cecf0-f7ec-4d03-8168-8b35a8446ba4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:58.038052 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.037951    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-credential-socket\") pod \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") "
Oct 30 13:27:58.038052 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.037988    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istio-data\") pod \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") "
Oct 30 13:27:58.038052 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.038018    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istio-token\" (UniqueName: \"kubernetes.io/projected/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istio-token\") pod \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") "
Oct 30 13:27:58.038349 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.038074    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istiod-ca-cert" pod "f08cecf0-f7ec-4d03-8168-8b35a8446ba4" (UID: "f08cecf0-f7ec-4d03-8168-8b35a8446ba4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:27:58.038349 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.038202    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-kube-api-access-lmbxz\" (UniqueName: \"kubernetes.io/projected/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-kube-api-access-lmbxz\") pod \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\" (UID: \"f08cecf0-f7ec-4d03-8168-8b35a8446ba4\") "
Oct 30 13:27:58.038463 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.038380    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-credential-socket" pod "f08cecf0-f7ec-4d03-8168-8b35a8446ba4" (UID: "f08cecf0-f7ec-4d03-8168-8b35a8446ba4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:58.038891 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.038868    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:27:58.039093 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.038893    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:27:58.039093 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.038910    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:27:58.039594 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.039571    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istio-data" pod "f08cecf0-f7ec-4d03-8168-8b35a8446ba4" (UID: "f08cecf0-f7ec-4d03-8168-8b35a8446ba4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:58.039819 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.039776    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-workload-certs" pod "f08cecf0-f7ec-4d03-8168-8b35a8446ba4" (UID: "f08cecf0-f7ec-4d03-8168-8b35a8446ba4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:58.043631 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.040711    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istio-envoy" pod "f08cecf0-f7ec-4d03-8168-8b35a8446ba4" (UID: "f08cecf0-f7ec-4d03-8168-8b35a8446ba4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:58.043631 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.041200    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istio-podinfo" pod "f08cecf0-f7ec-4d03-8168-8b35a8446ba4" (UID: "f08cecf0-f7ec-4d03-8168-8b35a8446ba4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:27:58.043631 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.041289    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-kube-api-access-lmbxz" pod "f08cecf0-f7ec-4d03-8168-8b35a8446ba4" (UID: "f08cecf0-f7ec-4d03-8168-8b35a8446ba4"). InnerVolumeSpecName "kube-api-access-lmbxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:58.043631 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.041611    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istio-token" pod "f08cecf0-f7ec-4d03-8168-8b35a8446ba4" (UID: "f08cecf0-f7ec-4d03-8168-8b35a8446ba4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:58.046845 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.046817    4544 generic.go:356] "Generic (PLEG): container finished" podID="f08cecf0-f7ec-4d03-8168-8b35a8446ba4" containerID="0759e23d488af4acd49ab4173f96c35fe2f1e3d2bad81a2d4a6afb92535ab652" exitCode=0
Oct 30 13:27:58.047055 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.046887    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"
Oct 30 13:27:58.047150 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.046898    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7" event={"ID":"f08cecf0-f7ec-4d03-8168-8b35a8446ba4","Type":"ContainerDied","Data":"0759e23d488af4acd49ab4173f96c35fe2f1e3d2bad81a2d4a6afb92535ab652"}
Oct 30 13:27:58.047294 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.047271    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7" event={"ID":"f08cecf0-f7ec-4d03-8168-8b35a8446ba4","Type":"ContainerDied","Data":"140b3e1e52bb27d2ad593563954c3c186893579da0f9fe0d1493fff2e226377f"}
Oct 30 13:27:58.047373 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.047324    4544 scope.go:122] "RemoveContainer" containerID="0759e23d488af4acd49ab4173f96c35fe2f1e3d2bad81a2d4a6afb92535ab652"
Oct 30 13:27:58.095847 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.095822    4544 scope.go:122] "RemoveContainer" containerID="0759e23d488af4acd49ab4173f96c35fe2f1e3d2bad81a2d4a6afb92535ab652"
Oct 30 13:27:58.096213 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:27:58.096185    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0759e23d488af4acd49ab4173f96c35fe2f1e3d2bad81a2d4a6afb92535ab652\": container with ID starting with 0759e23d488af4acd49ab4173f96c35fe2f1e3d2bad81a2d4a6afb92535ab652 not found: ID does not exist" containerID="0759e23d488af4acd49ab4173f96c35fe2f1e3d2bad81a2d4a6afb92535ab652"
Oct 30 13:27:58.096350 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.096239    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0759e23d488af4acd49ab4173f96c35fe2f1e3d2bad81a2d4a6afb92535ab652"} err="failed to get container status \"0759e23d488af4acd49ab4173f96c35fe2f1e3d2bad81a2d4a6afb92535ab652\": rpc error: code = NotFound desc = could not find container \"0759e23d488af4acd49ab4173f96c35fe2f1e3d2bad81a2d4a6afb92535ab652\": container with ID starting with 0759e23d488af4acd49ab4173f96c35fe2f1e3d2bad81a2d4a6afb92535ab652 not found: ID does not exist"
Oct 30 13:27:58.136208 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.136165    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"]
Oct 30 13:27:58.140404 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.140320    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:27:58.140404 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.140350    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:27:58.140404 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.140366    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmbxz\" (UniqueName: \"kubernetes.io/projected/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-kube-api-access-lmbxz\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:27:58.140404 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.140382    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:27:58.140404 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.140397    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:27:58.140404 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.140408    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f08cecf0-f7ec-4d03-8168-8b35a8446ba4-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:27:58.140800 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.140780    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7"]
Oct 30 13:27:58.608631 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:58.608593    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:27:59.009091 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:27:59.009003    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:27:59.089392 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:27:59.089359    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f08cecf0-f7ec-4d03-8168-8b35a8446ba4" path="/var/lib/kubelet/pods/f08cecf0-f7ec-4d03-8168-8b35a8446ba4/volumes"
Oct 30 13:28:00.330256 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:00.330192    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:01.523309 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:01.523259    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4"
Oct 30 13:28:01.536091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:01.536067    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4"
Oct 30 13:28:02.095034 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:02.095002    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4"
Oct 30 13:28:02.181053 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:02.181017    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:03.804961 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:03.804925    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj"]
Oct 30 13:28:03.952394 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:03.952359    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9"]
Oct 30 13:28:04.058971 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:04.058776    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:04.199107 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:04.199073    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:05.828055 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:05.828020    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:06.727959 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.727921    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s"]
Oct 30 13:28:06.728790 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.728765    4544 cpu_manager.go:414] "containerMap: removing container" podUID="e443ff59-e6ec-444d-817a-1b7baaa5faf7" containerName="payload-pre-processing"
Oct 30 13:28:06.728790 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.728785    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e443ff59-e6ec-444d-817a-1b7baaa5faf7" containerName="payload-pre-processing"
Oct 30 13:28:06.728970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.728799    4544 cpu_manager.go:414] "containerMap: removing container" podUID="f08cecf0-f7ec-4d03-8168-8b35a8446ba4" containerName="istio-proxy"
Oct 30 13:28:06.728970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.728804    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f08cecf0-f7ec-4d03-8168-8b35a8446ba4" containerName="istio-proxy"
Oct 30 13:28:06.728970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.728825    4544 cpu_manager.go:414] "containerMap: removing container" podUID="9a9a0392-a7ea-4d31-add2-f4faf60c121b" containerName="payload-processing"
Oct 30 13:28:06.728970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.728836    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a9a0392-a7ea-4d31-add2-f4faf60c121b" containerName="payload-processing"
Oct 30 13:28:06.729167 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.729094    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="e443ff59-e6ec-444d-817a-1b7baaa5faf7" containerName="payload-pre-processing"
Oct 30 13:28:06.729167 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.729109    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="f08cecf0-f7ec-4d03-8168-8b35a8446ba4" containerName="istio-proxy"
Oct 30 13:28:06.729167 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.729119    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a9a0392-a7ea-4d31-add2-f4faf60c121b" containerName="payload-processing"
Oct 30 13:28:06.733894 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.733862    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s"
Oct 30 13:28:06.764569 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.764540    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-0b2c92\"/\"default-dockercfg-pk2ws\""
Oct 30 13:28:06.764934 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.764915    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-0b2c92\"/\"auth-test-model-0b2c92-kserve-self-signed-certs\""
Oct 30 13:28:06.765143 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.765128    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-0b2c92\"/\"kube-root-ca.crt\""
Oct 30 13:28:06.765376 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.765361    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-0b2c92\"/\"openshift-service-ca.crt\""
Oct 30 13:28:06.784144 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.784111    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s"]
Oct 30 13:28:06.800182 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.800136    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-tls-certs\") pod \"auth-test-model-0b2c92-kserve-9855b8f97-lqt6s\" (UID: \"3f87a8dd-dd6e-4e00-942d-cfe614d28e6f\") " pod="ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s"
Oct 30 13:28:06.800530 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.800285    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-tmp-dir\") pod \"auth-test-model-0b2c92-kserve-9855b8f97-lqt6s\" (UID: \"3f87a8dd-dd6e-4e00-942d-cfe614d28e6f\") " pod="ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s"
Oct 30 13:28:06.800530 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.800446    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pzfq\" (UniqueName: \"kubernetes.io/projected/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-kube-api-access-6pzfq\") pod \"auth-test-model-0b2c92-kserve-9855b8f97-lqt6s\" (UID: \"3f87a8dd-dd6e-4e00-942d-cfe614d28e6f\") " pod="ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s"
Oct 30 13:28:06.800530 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.800479    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-dshm\") pod \"auth-test-model-0b2c92-kserve-9855b8f97-lqt6s\" (UID: \"3f87a8dd-dd6e-4e00-942d-cfe614d28e6f\") " pod="ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s"
Oct 30 13:28:06.800682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.800533    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-home\") pod \"auth-test-model-0b2c92-kserve-9855b8f97-lqt6s\" (UID: \"3f87a8dd-dd6e-4e00-942d-cfe614d28e6f\") " pod="ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s"
Oct 30 13:28:06.800682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.800577    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-model-cache\") pod \"auth-test-model-0b2c92-kserve-9855b8f97-lqt6s\" (UID: \"3f87a8dd-dd6e-4e00-942d-cfe614d28e6f\") " pod="ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s"
Oct 30 13:28:06.811207 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.811172    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:06.901862 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.901823    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-tmp-dir\") pod \"auth-test-model-0b2c92-kserve-9855b8f97-lqt6s\" (UID: \"3f87a8dd-dd6e-4e00-942d-cfe614d28e6f\") " pod="ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s"
Oct 30 13:28:06.902351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.901949    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pzfq\" (UniqueName: \"kubernetes.io/projected/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-kube-api-access-6pzfq\") pod \"auth-test-model-0b2c92-kserve-9855b8f97-lqt6s\" (UID: \"3f87a8dd-dd6e-4e00-942d-cfe614d28e6f\") " pod="ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s"
Oct 30 13:28:06.902351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.901978    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-dshm\") pod \"auth-test-model-0b2c92-kserve-9855b8f97-lqt6s\" (UID: \"3f87a8dd-dd6e-4e00-942d-cfe614d28e6f\") " pod="ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s"
Oct 30 13:28:06.902351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.902012    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-home\") pod \"auth-test-model-0b2c92-kserve-9855b8f97-lqt6s\" (UID: \"3f87a8dd-dd6e-4e00-942d-cfe614d28e6f\") " pod="ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s"
Oct 30 13:28:06.902351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.902054    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-model-cache\") pod \"auth-test-model-0b2c92-kserve-9855b8f97-lqt6s\" (UID: \"3f87a8dd-dd6e-4e00-942d-cfe614d28e6f\") " pod="ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s"
Oct 30 13:28:06.902351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.902154    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-tls-certs\") pod \"auth-test-model-0b2c92-kserve-9855b8f97-lqt6s\" (UID: \"3f87a8dd-dd6e-4e00-942d-cfe614d28e6f\") " pod="ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s"
Oct 30 13:28:06.902351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.902308    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-tmp-dir\") pod \"auth-test-model-0b2c92-kserve-9855b8f97-lqt6s\" (UID: \"3f87a8dd-dd6e-4e00-942d-cfe614d28e6f\") " pod="ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s"
Oct 30 13:28:06.902690 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.902460    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-model-cache\") pod \"auth-test-model-0b2c92-kserve-9855b8f97-lqt6s\" (UID: \"3f87a8dd-dd6e-4e00-942d-cfe614d28e6f\") " pod="ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s"
Oct 30 13:28:06.902690 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.902454    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-home\") pod \"auth-test-model-0b2c92-kserve-9855b8f97-lqt6s\" (UID: \"3f87a8dd-dd6e-4e00-942d-cfe614d28e6f\") " pod="ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s"
Oct 30 13:28:06.904458 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.904431    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-dshm\") pod \"auth-test-model-0b2c92-kserve-9855b8f97-lqt6s\" (UID: \"3f87a8dd-dd6e-4e00-942d-cfe614d28e6f\") " pod="ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s"
Oct 30 13:28:06.904798 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.904778    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-tls-certs\") pod \"auth-test-model-0b2c92-kserve-9855b8f97-lqt6s\" (UID: \"3f87a8dd-dd6e-4e00-942d-cfe614d28e6f\") " pod="ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s"
Oct 30 13:28:06.926891 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:06.926828    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pzfq\" (UniqueName: \"kubernetes.io/projected/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-kube-api-access-6pzfq\") pod \"auth-test-model-0b2c92-kserve-9855b8f97-lqt6s\" (UID: \"3f87a8dd-dd6e-4e00-942d-cfe614d28e6f\") " pod="ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s"
Oct 30 13:28:07.125243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:07.106862    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s"
Oct 30 13:28:07.624288 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:07.612432    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s"]
Oct 30 13:28:07.654437 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:28:07.654350    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f87a8dd_dd6e_4e00_942d_cfe614d28e6f.slice/crio-0aa64d90973d76bf10e3dbcc3eb3201e62f4f78ebad6073779b45bcccdceed25 WatchSource:0}: Error finding container 0aa64d90973d76bf10e3dbcc3eb3201e62f4f78ebad6073779b45bcccdceed25: Status 404 returned error can't find the container with id 0aa64d90973d76bf10e3dbcc3eb3201e62f4f78ebad6073779b45bcccdceed25
Oct 30 13:28:08.163933 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:08.163896    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s" event={"ID":"3f87a8dd-dd6e-4e00-942d-cfe614d28e6f","Type":"ContainerStarted","Data":"2359691669bd197d08cc073f232481502857cd8c20251d498caadf3e2a98edae"}
Oct 30 13:28:08.163933 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:08.163938    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s" event={"ID":"3f87a8dd-dd6e-4e00-942d-cfe614d28e6f","Type":"ContainerStarted","Data":"0aa64d90973d76bf10e3dbcc3eb3201e62f4f78ebad6073779b45bcccdceed25"}
Oct 30 13:28:08.212488 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:08.212435    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s" podStartSLOduration=2.212421417 podStartE2EDuration="2.212421417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:08.210800897 +0000 UTC m=+2583.257934853" watchObservedRunningTime="2026-10-30 13:28:08.212421417 +0000 UTC m=+2583.259555370"
Oct 30 13:28:08.300945 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:08.300906    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:08.805557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:08.805513    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj" podUID="2a9b6789-4b42-43b8-a815-bc9f835452c7" containerName="payload-processing" containerID="cri-o://61c150bc17cff9173ff5d6c5f32dc9c1b8b4a8b8c4486fef770420a3af8e8bbc" gracePeriod=25
Oct 30 13:28:08.953313 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:08.953267    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9" podUID="872e259b-51ea-477d-b898-e8d4bbb086f5" containerName="payload-pre-processing" containerID="cri-o://81a19b3c7be5e67ceb8a4d553f6de5ea437599f43f81dc99e970905d9a2df212" gracePeriod=25
Oct 30 13:28:09.110713 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:09.110685    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s"
Oct 30 13:28:10.038264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:10.038202    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:11.926980 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:11.926939    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:13.716541 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:13.716501    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:13.759637 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:13.759597    4544 scope.go:122] "RemoveContainer" containerID="b99220279d32c63d51786be6dc6fe616fc1b19b92f67a72ad56dfbe0f284d7f7"
Oct 30 13:28:13.783394 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:13.783348    4544 scope.go:122] "RemoveContainer" containerID="0182b2d67ba8785b0e530622213cbd0c6b600f2acd52433c11c29076b3cfcc23"
Oct 30 13:28:15.159835 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:15.159791    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj" podUID="2a9b6789-4b42-43b8-a815-bc9f835452c7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:28:15.160241 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:15.159842    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9" podUID="872e259b-51ea-477d-b898-e8d4bbb086f5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:28:15.436422 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:15.436336    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:16.042412 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:16.042377    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7db9402-5nkgf"]
Oct 30 13:28:16.054213 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:16.054182    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7db9402-5nkgf"]
Oct 30 13:28:17.089293 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:17.089259    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="806e56d4-e04c-4b18-b18f-da7f5b44189c" path="/var/lib/kubelet/pods/806e56d4-e04c-4b18-b18f-da7f5b44189c/volumes"
Oct 30 13:28:17.110265 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:17.110232    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s"
Oct 30 13:28:17.122999 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:17.122980    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s"
Oct 30 13:28:17.250108 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:17.250071    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s"
Oct 30 13:28:17.264896 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:17.264861    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:18.989917 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:18.989871    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:20.710416 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:20.710382    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:20.835253 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:20.835205    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:22.864723 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:22.864678    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:24.748053 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:24.748005    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:25.159824 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:25.159787    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9" podUID="872e259b-51ea-477d-b898-e8d4bbb086f5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:28:25.159824 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:25.159807    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj" podUID="2a9b6789-4b42-43b8-a815-bc9f835452c7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:28:26.630605 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:26.630563    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:28.487303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:28.487215    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:29.028505 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:28:29.028466    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:28:29.099933 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:29.099902    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj"
Oct 30 13:28:29.217302 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:29.217252    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a9b6789-4b42-43b8-a815-bc9f835452c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a9b6789-4b42-43b8-a815-bc9f835452c7-plugins-config-volume\") pod \"2a9b6789-4b42-43b8-a815-bc9f835452c7\" (UID: \"2a9b6789-4b42-43b8-a815-bc9f835452c7\") "
Oct 30 13:28:29.217510 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:29.217367    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a9b6789-4b42-43b8-a815-bc9f835452c7-kube-api-access-vt9t2\" (UniqueName: \"kubernetes.io/projected/2a9b6789-4b42-43b8-a815-bc9f835452c7-kube-api-access-vt9t2\") pod \"2a9b6789-4b42-43b8-a815-bc9f835452c7\" (UID: \"2a9b6789-4b42-43b8-a815-bc9f835452c7\") "
Oct 30 13:28:29.217725 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:29.217684    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a9b6789-4b42-43b8-a815-bc9f835452c7-plugins-config-volume" pod "2a9b6789-4b42-43b8-a815-bc9f835452c7" (UID: "2a9b6789-4b42-43b8-a815-bc9f835452c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:29.218048 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:29.218016    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a9b6789-4b42-43b8-a815-bc9f835452c7-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:28:29.221793 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:29.221755    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a9b6789-4b42-43b8-a815-bc9f835452c7-kube-api-access-vt9t2" pod "2a9b6789-4b42-43b8-a815-bc9f835452c7" (UID: "2a9b6789-4b42-43b8-a815-bc9f835452c7"). InnerVolumeSpecName "kube-api-access-vt9t2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:29.242297 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:29.242271    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9"
Oct 30 13:28:29.319425 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:29.319325    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/872e259b-51ea-477d-b898-e8d4bbb086f5-kube-api-access-hbhms\" (UniqueName: \"kubernetes.io/projected/872e259b-51ea-477d-b898-e8d4bbb086f5-kube-api-access-hbhms\") pod \"872e259b-51ea-477d-b898-e8d4bbb086f5\" (UID: \"872e259b-51ea-477d-b898-e8d4bbb086f5\") "
Oct 30 13:28:29.319606 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:29.319527    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/872e259b-51ea-477d-b898-e8d4bbb086f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872e259b-51ea-477d-b898-e8d4bbb086f5-plugins-config-volume\") pod \"872e259b-51ea-477d-b898-e8d4bbb086f5\" (UID: \"872e259b-51ea-477d-b898-e8d4bbb086f5\") "
Oct 30 13:28:29.319967 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:29.319940    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/872e259b-51ea-477d-b898-e8d4bbb086f5-plugins-config-volume" pod "872e259b-51ea-477d-b898-e8d4bbb086f5" (UID: "872e259b-51ea-477d-b898-e8d4bbb086f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:29.320077 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:29.320039    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt9t2\" (UniqueName: \"kubernetes.io/projected/2a9b6789-4b42-43b8-a815-bc9f835452c7-kube-api-access-vt9t2\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:28:29.321656 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:29.321628    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872e259b-51ea-477d-b898-e8d4bbb086f5-kube-api-access-hbhms" pod "872e259b-51ea-477d-b898-e8d4bbb086f5" (UID: "872e259b-51ea-477d-b898-e8d4bbb086f5"). InnerVolumeSpecName "kube-api-access-hbhms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:29.338923 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:29.338884    4544 generic.go:356] "Generic (PLEG): container finished" podID="2a9b6789-4b42-43b8-a815-bc9f835452c7" containerID="61c150bc17cff9173ff5d6c5f32dc9c1b8b4a8b8c4486fef770420a3af8e8bbc" exitCode=0
Oct 30 13:28:29.339145 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:29.338953    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj"
Oct 30 13:28:29.339278 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:29.339014    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj" event={"ID":"2a9b6789-4b42-43b8-a815-bc9f835452c7","Type":"ContainerDied","Data":"61c150bc17cff9173ff5d6c5f32dc9c1b8b4a8b8c4486fef770420a3af8e8bbc"}
Oct 30 13:28:29.339508 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:29.339462    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj" event={"ID":"2a9b6789-4b42-43b8-a815-bc9f835452c7","Type":"ContainerDied","Data":"3a919582a6ddecb2b134e2f3a580cc2c1f7adbc662f085eb9dcc7d095e1ba5f3"}
Oct 30 13:28:29.339642 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:29.339501    4544 scope.go:122] "RemoveContainer" containerID="61c150bc17cff9173ff5d6c5f32dc9c1b8b4a8b8c4486fef770420a3af8e8bbc"
Oct 30 13:28:29.350059 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:29.341439    4544 generic.go:356] "Generic (PLEG): container finished" podID="872e259b-51ea-477d-b898-e8d4bbb086f5" containerID="81a19b3c7be5e67ceb8a4d553f6de5ea437599f43f81dc99e970905d9a2df212" exitCode=0
Oct 30 13:28:29.350059 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:29.341632    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9" event={"ID":"872e259b-51ea-477d-b898-e8d4bbb086f5","Type":"ContainerDied","Data":"81a19b3c7be5e67ceb8a4d553f6de5ea437599f43f81dc99e970905d9a2df212"}
Oct 30 13:28:29.350059 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:29.341677    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9" event={"ID":"872e259b-51ea-477d-b898-e8d4bbb086f5","Type":"ContainerDied","Data":"2c784cc29ccabda944e84812729fb6638cc069abd16111450f8280a641f58de8"}
Oct 30 13:28:29.350059 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:29.341764    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9"
Oct 30 13:28:29.354385 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:29.354360    4544 scope.go:122] "RemoveContainer" containerID="61c150bc17cff9173ff5d6c5f32dc9c1b8b4a8b8c4486fef770420a3af8e8bbc"
Oct 30 13:28:29.354688 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:28:29.354666    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61c150bc17cff9173ff5d6c5f32dc9c1b8b4a8b8c4486fef770420a3af8e8bbc\": container with ID starting with 61c150bc17cff9173ff5d6c5f32dc9c1b8b4a8b8c4486fef770420a3af8e8bbc not found: ID does not exist" containerID="61c150bc17cff9173ff5d6c5f32dc9c1b8b4a8b8c4486fef770420a3af8e8bbc"
Oct 30 13:28:29.354750 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:29.354697    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61c150bc17cff9173ff5d6c5f32dc9c1b8b4a8b8c4486fef770420a3af8e8bbc"} err="failed to get container status \"61c150bc17cff9173ff5d6c5f32dc9c1b8b4a8b8c4486fef770420a3af8e8bbc\": rpc error: code = NotFound desc = could not find container \"61c150bc17cff9173ff5d6c5f32dc9c1b8b4a8b8c4486fef770420a3af8e8bbc\": container with ID starting with 61c150bc17cff9173ff5d6c5f32dc9c1b8b4a8b8c4486fef770420a3af8e8bbc not found: ID does not exist"
Oct 30 13:28:29.354750 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:29.354715    4544 scope.go:122] "RemoveContainer" containerID="81a19b3c7be5e67ceb8a4d553f6de5ea437599f43f81dc99e970905d9a2df212"
Oct 30 13:28:29.365527 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:29.365501    4544 scope.go:122] "RemoveContainer" containerID="81a19b3c7be5e67ceb8a4d553f6de5ea437599f43f81dc99e970905d9a2df212"
Oct 30 13:28:29.365903 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:28:29.365878    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81a19b3c7be5e67ceb8a4d553f6de5ea437599f43f81dc99e970905d9a2df212\": container with ID starting with 81a19b3c7be5e67ceb8a4d553f6de5ea437599f43f81dc99e970905d9a2df212 not found: ID does not exist" containerID="81a19b3c7be5e67ceb8a4d553f6de5ea437599f43f81dc99e970905d9a2df212"
Oct 30 13:28:29.365971 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:29.365915    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81a19b3c7be5e67ceb8a4d553f6de5ea437599f43f81dc99e970905d9a2df212"} err="failed to get container status \"81a19b3c7be5e67ceb8a4d553f6de5ea437599f43f81dc99e970905d9a2df212\": rpc error: code = NotFound desc = could not find container \"81a19b3c7be5e67ceb8a4d553f6de5ea437599f43f81dc99e970905d9a2df212\": container with ID starting with 81a19b3c7be5e67ceb8a4d553f6de5ea437599f43f81dc99e970905d9a2df212 not found: ID does not exist"
Oct 30 13:28:29.395121 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:29.395080    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9"]
Oct 30 13:28:29.400908 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:29.400869    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9"]
Oct 30 13:28:29.421547 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:29.421504    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872e259b-51ea-477d-b898-e8d4bbb086f5-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:28:29.421547 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:29.421541    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbhms\" (UniqueName: \"kubernetes.io/projected/872e259b-51ea-477d-b898-e8d4bbb086f5-kube-api-access-hbhms\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:28:29.435076 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:29.435036    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj"]
Oct 30 13:28:29.439918 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:29.439880    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj"]
Oct 30 13:28:29.960913 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:29.960881    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:30.531245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:30.519157    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:30.756023 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:30.755983    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:30.899696 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:30.899660    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:31.093997 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:31.093317    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a9b6789-4b42-43b8-a815-bc9f835452c7" path="/var/lib/kubelet/pods/2a9b6789-4b42-43b8-a815-bc9f835452c7/volumes"
Oct 30 13:28:31.104240 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:31.100471    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="872e259b-51ea-477d-b898-e8d4bbb086f5" path="/var/lib/kubelet/pods/872e259b-51ea-477d-b898-e8d4bbb086f5/volumes"
Oct 30 13:28:31.250164 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:31.250066    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:32.429533 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:32.429490    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:32.794079 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:32.793980    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:34.586481 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:34.586443    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:36.470371 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:36.470328    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:36.478292 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:36.478252    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd"]
Oct 30 13:28:36.479514 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:36.479493    4544 cpu_manager.go:414] "containerMap: removing container" podUID="2a9b6789-4b42-43b8-a815-bc9f835452c7" containerName="payload-processing"
Oct 30 13:28:36.479598 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:36.479517    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a9b6789-4b42-43b8-a815-bc9f835452c7" containerName="payload-processing"
Oct 30 13:28:36.479598 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:36.479575    4544 cpu_manager.go:414] "containerMap: removing container" podUID="872e259b-51ea-477d-b898-e8d4bbb086f5" containerName="payload-pre-processing"
Oct 30 13:28:36.479598 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:36.479584    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="872e259b-51ea-477d-b898-e8d4bbb086f5" containerName="payload-pre-processing"
Oct 30 13:28:36.479748 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:36.479734    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a9b6789-4b42-43b8-a815-bc9f835452c7" containerName="payload-processing"
Oct 30 13:28:36.479782 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:36.479758    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="872e259b-51ea-477d-b898-e8d4bbb086f5" containerName="payload-pre-processing"
Oct 30 13:28:36.484270 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:36.484208    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd"
Oct 30 13:28:36.502600 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:36.502565    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd"]
Oct 30 13:28:36.508636 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:36.508603    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-3e8ee4\"/\"kube-root-ca.crt\""
Oct 30 13:28:36.508878 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:36.508714    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-3e8ee4\"/\"openshift-service-ca.crt\""
Oct 30 13:28:36.512497 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:36.512469    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-3e8ee4\"/\"auth-test-model-3e8ee4-kserve-self-signed-certs\""
Oct 30 13:28:36.512497 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:36.512473    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-3e8ee4\"/\"default-dockercfg-s5s6r\""
Oct 30 13:28:36.608776 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:36.608729    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:36.649216 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:36.649179    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf7d6\" (UniqueName: \"kubernetes.io/projected/84804195-56b8-4aae-99a4-73ee1e589fed-kube-api-access-lf7d6\") pod \"auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd\" (UID: \"84804195-56b8-4aae-99a4-73ee1e589fed\") " pod="ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd"
Oct 30 13:28:36.649428 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:36.649410    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84804195-56b8-4aae-99a4-73ee1e589fed-tls-certs\") pod \"auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd\" (UID: \"84804195-56b8-4aae-99a4-73ee1e589fed\") " pod="ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd"
Oct 30 13:28:36.649476 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:36.649442    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84804195-56b8-4aae-99a4-73ee1e589fed-model-cache\") pod \"auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd\" (UID: \"84804195-56b8-4aae-99a4-73ee1e589fed\") " pod="ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd"
Oct 30 13:28:36.649514 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:36.649469    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84804195-56b8-4aae-99a4-73ee1e589fed-dshm\") pod \"auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd\" (UID: \"84804195-56b8-4aae-99a4-73ee1e589fed\") " pod="ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd"
Oct 30 13:28:36.649634 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:36.649610    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84804195-56b8-4aae-99a4-73ee1e589fed-tmp-dir\") pod \"auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd\" (UID: \"84804195-56b8-4aae-99a4-73ee1e589fed\") " pod="ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd"
Oct 30 13:28:36.649672 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:36.649659    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84804195-56b8-4aae-99a4-73ee1e589fed-home\") pod \"auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd\" (UID: \"84804195-56b8-4aae-99a4-73ee1e589fed\") " pod="ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd"
Oct 30 13:28:36.776297 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:36.768572    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84804195-56b8-4aae-99a4-73ee1e589fed-tls-certs\") pod \"auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd\" (UID: \"84804195-56b8-4aae-99a4-73ee1e589fed\") " pod="ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd"
Oct 30 13:28:36.776297 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:36.768633    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84804195-56b8-4aae-99a4-73ee1e589fed-model-cache\") pod \"auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd\" (UID: \"84804195-56b8-4aae-99a4-73ee1e589fed\") " pod="ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd"
Oct 30 13:28:36.776297 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:36.768674    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84804195-56b8-4aae-99a4-73ee1e589fed-dshm\") pod \"auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd\" (UID: \"84804195-56b8-4aae-99a4-73ee1e589fed\") " pod="ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd"
Oct 30 13:28:36.776297 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:36.768738    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84804195-56b8-4aae-99a4-73ee1e589fed-tmp-dir\") pod \"auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd\" (UID: \"84804195-56b8-4aae-99a4-73ee1e589fed\") " pod="ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd"
Oct 30 13:28:36.776297 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:36.768787    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84804195-56b8-4aae-99a4-73ee1e589fed-home\") pod \"auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd\" (UID: \"84804195-56b8-4aae-99a4-73ee1e589fed\") " pod="ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd"
Oct 30 13:28:36.776297 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:36.768869    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf7d6\" (UniqueName: \"kubernetes.io/projected/84804195-56b8-4aae-99a4-73ee1e589fed-kube-api-access-lf7d6\") pod \"auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd\" (UID: \"84804195-56b8-4aae-99a4-73ee1e589fed\") " pod="ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd"
Oct 30 13:28:36.776297 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:36.769784    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84804195-56b8-4aae-99a4-73ee1e589fed-tmp-dir\") pod \"auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd\" (UID: \"84804195-56b8-4aae-99a4-73ee1e589fed\") " pod="ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd"
Oct 30 13:28:36.776297 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:36.770043    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84804195-56b8-4aae-99a4-73ee1e589fed-home\") pod \"auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd\" (UID: \"84804195-56b8-4aae-99a4-73ee1e589fed\") " pod="ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd"
Oct 30 13:28:36.776297 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:36.770739    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84804195-56b8-4aae-99a4-73ee1e589fed-model-cache\") pod \"auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd\" (UID: \"84804195-56b8-4aae-99a4-73ee1e589fed\") " pod="ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd"
Oct 30 13:28:36.785277 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:36.785177    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84804195-56b8-4aae-99a4-73ee1e589fed-tls-certs\") pod \"auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd\" (UID: \"84804195-56b8-4aae-99a4-73ee1e589fed\") " pod="ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd"
Oct 30 13:28:36.808257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:36.798114    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf7d6\" (UniqueName: \"kubernetes.io/projected/84804195-56b8-4aae-99a4-73ee1e589fed-kube-api-access-lf7d6\") pod \"auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd\" (UID: \"84804195-56b8-4aae-99a4-73ee1e589fed\") " pod="ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd"
Oct 30 13:28:36.812657 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:36.810642    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84804195-56b8-4aae-99a4-73ee1e589fed-dshm\") pod \"auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd\" (UID: \"84804195-56b8-4aae-99a4-73ee1e589fed\") " pod="ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd"
Oct 30 13:28:37.099890 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:37.099845    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd"
Oct 30 13:28:37.413270 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:37.413237    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd"]
Oct 30 13:28:37.414487 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:28:37.414454    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84804195_56b8_4aae_99a4_73ee1e589fed.slice/crio-e8336e870279968a99bbabe0de142a7b40e1e63c80b37a529801ea2cac49423b WatchSource:0}: Error finding container e8336e870279968a99bbabe0de142a7b40e1e63c80b37a529801ea2cac49423b: Status 404 returned error can't find the container with id e8336e870279968a99bbabe0de142a7b40e1e63c80b37a529801ea2cac49423b
Oct 30 13:28:38.230886 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:38.230856    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:38.358898 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:38.357804    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:38.422592 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:38.422548    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd" event={"ID":"84804195-56b8-4aae-99a4-73ee1e589fed","Type":"ContainerStarted","Data":"ef9463b3088302cd0c552038a5ce5ad70eddd2c1f059aa2f27e20b11040de16d"}
Oct 30 13:28:38.422592 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:38.422600    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd" event={"ID":"84804195-56b8-4aae-99a4-73ee1e589fed","Type":"ContainerStarted","Data":"e8336e870279968a99bbabe0de142a7b40e1e63c80b37a529801ea2cac49423b"}
Oct 30 13:28:38.467317 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:38.467264    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd" podStartSLOduration=2.467247379 podStartE2EDuration="2.467247379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:38.467017674 +0000 UTC m=+2613.514151626" watchObservedRunningTime="2026-10-30 13:28:38.467247379 +0000 UTC m=+2613.514381333"
Oct 30 13:28:38.565422 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:38.565325    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"]
Oct 30 13:28:38.565654 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:38.565627    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl" podUID="bd4c038f-d0d4-4fd5-ad51-2dbae210cb26" containerName="istio-proxy" containerID="cri-o://ad78e25267fb73c05e7e65d6e5d7fb1a098a6d5988b0134d3255e6b807b0d621" gracePeriod=30
Oct 30 13:28:38.606456 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:38.606423    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:39.100665 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:39.100630    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd"
Oct 30 13:28:39.380171 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:39.380056    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:39.945666 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:39.945630    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:40.092260 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:40.080191    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:40.240351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:40.238118    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:42.175968 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:42.175919    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:43.381670 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:43.381634    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:43.593523 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:43.593480    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:43.916351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:43.916275    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:28:43.997328 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:43.997290    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istio-podinfo\") pod \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") "
Oct 30 13:28:43.997545 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:43.997458    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-credential-socket\") pod \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") "
Oct 30 13:28:43.997545 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:43.997518    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istio-token\" (UniqueName: \"kubernetes.io/projected/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istio-token\") pod \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") "
Oct 30 13:28:43.997720 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:43.997587    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-workload-certs\") pod \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") "
Oct 30 13:28:43.997720 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:43.997695    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istio-data\") pod \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") "
Oct 30 13:28:43.997841 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:43.997749    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-workload-socket\") pod \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") "
Oct 30 13:28:43.997841 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:43.997813    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-kube-api-access-kkjwf\" (UniqueName: \"kubernetes.io/projected/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-kube-api-access-kkjwf\") pod \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") "
Oct 30 13:28:43.998019 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:43.997843    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istiod-ca-cert\") pod \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") "
Oct 30 13:28:43.998019 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:43.997870    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istio-envoy\") pod \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\" (UID: \"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26\") "
Oct 30 13:28:43.999265 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:43.999186    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-workload-certs" pod "bd4c038f-d0d4-4fd5-ad51-2dbae210cb26" (UID: "bd4c038f-d0d4-4fd5-ad51-2dbae210cb26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:43.999265 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:43.999251    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-credential-socket" pod "bd4c038f-d0d4-4fd5-ad51-2dbae210cb26" (UID: "bd4c038f-d0d4-4fd5-ad51-2dbae210cb26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:44.000289 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:44.000070    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istiod-ca-cert" pod "bd4c038f-d0d4-4fd5-ad51-2dbae210cb26" (UID: "bd4c038f-d0d4-4fd5-ad51-2dbae210cb26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:44.000486 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:44.000444    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istio-data" pod "bd4c038f-d0d4-4fd5-ad51-2dbae210cb26" (UID: "bd4c038f-d0d4-4fd5-ad51-2dbae210cb26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:44.000798 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:44.000781    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-workload-socket" pod "bd4c038f-d0d4-4fd5-ad51-2dbae210cb26" (UID: "bd4c038f-d0d4-4fd5-ad51-2dbae210cb26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:44.001252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:44.001176    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istio-envoy" pod "bd4c038f-d0d4-4fd5-ad51-2dbae210cb26" (UID: "bd4c038f-d0d4-4fd5-ad51-2dbae210cb26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:44.002491 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:44.002466    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istio-token" pod "bd4c038f-d0d4-4fd5-ad51-2dbae210cb26" (UID: "bd4c038f-d0d4-4fd5-ad51-2dbae210cb26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:44.002577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:44.002559    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-kube-api-access-kkjwf" pod "bd4c038f-d0d4-4fd5-ad51-2dbae210cb26" (UID: "bd4c038f-d0d4-4fd5-ad51-2dbae210cb26"). InnerVolumeSpecName "kube-api-access-kkjwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:44.012092 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:44.012054    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istio-podinfo" pod "bd4c038f-d0d4-4fd5-ad51-2dbae210cb26" (UID: "bd4c038f-d0d4-4fd5-ad51-2dbae210cb26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:28:44.099851 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:44.099810    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkjwf\" (UniqueName: \"kubernetes.io/projected/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-kube-api-access-kkjwf\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:28:44.099851 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:44.099843    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:28:44.099851 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:44.099853    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:28:44.099851 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:44.099861    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:28:44.100170 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:44.099869    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:28:44.100170 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:44.099878    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:28:44.100170 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:44.099886    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:28:44.100170 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:44.099893    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:28:44.100170 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:44.099901    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:28:44.392547 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:44.392508    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:44.475890 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:44.475855    4544 generic.go:356] "Generic (PLEG): container finished" podID="bd4c038f-d0d4-4fd5-ad51-2dbae210cb26" containerID="ad78e25267fb73c05e7e65d6e5d7fb1a098a6d5988b0134d3255e6b807b0d621" exitCode=0
Oct 30 13:28:44.476090 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:44.475903    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"
Oct 30 13:28:44.476090 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:44.475940    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl" event={"ID":"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26","Type":"ContainerDied","Data":"ad78e25267fb73c05e7e65d6e5d7fb1a098a6d5988b0134d3255e6b807b0d621"}
Oct 30 13:28:44.476090 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:44.475999    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl" event={"ID":"bd4c038f-d0d4-4fd5-ad51-2dbae210cb26","Type":"ContainerDied","Data":"e29c90e0a3d4cbbd2119a8cace01f4847a5139579ae2224c7c92fbaa6cde9381"}
Oct 30 13:28:44.476090 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:44.476025    4544 scope.go:122] "RemoveContainer" containerID="ad78e25267fb73c05e7e65d6e5d7fb1a098a6d5988b0134d3255e6b807b0d621"
Oct 30 13:28:44.487378 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:44.487329    4544 scope.go:122] "RemoveContainer" containerID="8065e62d66aaf050576444ef5dd78cae779752f4f4e2c60b2f551ced1d07dd5d"
Oct 30 13:28:44.498246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:44.498198    4544 scope.go:122] "RemoveContainer" containerID="ad78e25267fb73c05e7e65d6e5d7fb1a098a6d5988b0134d3255e6b807b0d621"
Oct 30 13:28:44.498559 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:28:44.498536    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad78e25267fb73c05e7e65d6e5d7fb1a098a6d5988b0134d3255e6b807b0d621\": container with ID starting with ad78e25267fb73c05e7e65d6e5d7fb1a098a6d5988b0134d3255e6b807b0d621 not found: ID does not exist" containerID="ad78e25267fb73c05e7e65d6e5d7fb1a098a6d5988b0134d3255e6b807b0d621"
Oct 30 13:28:44.498615 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:44.498567    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad78e25267fb73c05e7e65d6e5d7fb1a098a6d5988b0134d3255e6b807b0d621"} err="failed to get container status \"ad78e25267fb73c05e7e65d6e5d7fb1a098a6d5988b0134d3255e6b807b0d621\": rpc error: code = NotFound desc = could not find container \"ad78e25267fb73c05e7e65d6e5d7fb1a098a6d5988b0134d3255e6b807b0d621\": container with ID starting with ad78e25267fb73c05e7e65d6e5d7fb1a098a6d5988b0134d3255e6b807b0d621 not found: ID does not exist"
Oct 30 13:28:44.498615 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:44.498585    4544 scope.go:122] "RemoveContainer" containerID="8065e62d66aaf050576444ef5dd78cae779752f4f4e2c60b2f551ced1d07dd5d"
Oct 30 13:28:44.498841 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:28:44.498819    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8065e62d66aaf050576444ef5dd78cae779752f4f4e2c60b2f551ced1d07dd5d\": container with ID starting with 8065e62d66aaf050576444ef5dd78cae779752f4f4e2c60b2f551ced1d07dd5d not found: ID does not exist" containerID="8065e62d66aaf050576444ef5dd78cae779752f4f4e2c60b2f551ced1d07dd5d"
Oct 30 13:28:44.498888 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:44.498849    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8065e62d66aaf050576444ef5dd78cae779752f4f4e2c60b2f551ced1d07dd5d"} err="failed to get container status \"8065e62d66aaf050576444ef5dd78cae779752f4f4e2c60b2f551ced1d07dd5d\": rpc error: code = NotFound desc = could not find container \"8065e62d66aaf050576444ef5dd78cae779752f4f4e2c60b2f551ced1d07dd5d\": container with ID starting with 8065e62d66aaf050576444ef5dd78cae779752f4f4e2c60b2f551ced1d07dd5d not found: ID does not exist"
Oct 30 13:28:44.535274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:44.535234    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"]
Oct 30 13:28:44.556246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:44.551580    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl"]
Oct 30 13:28:45.260244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:45.248642    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:45.289926 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:45.284955    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd4c038f-d0d4-4fd5-ad51-2dbae210cb26" path="/var/lib/kubelet/pods/bd4c038f-d0d4-4fd5-ad51-2dbae210cb26/volumes"
Oct 30 13:28:45.365846 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:45.365808    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:45.420876 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:45.420831    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl"]
Oct 30 13:28:45.421852 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:45.421827    4544 cpu_manager.go:414] "containerMap: removing container" podUID="bd4c038f-d0d4-4fd5-ad51-2dbae210cb26" containerName="istio-proxy"
Oct 30 13:28:45.421852 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:45.421851    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd4c038f-d0d4-4fd5-ad51-2dbae210cb26" containerName="istio-proxy"
Oct 30 13:28:45.422032 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:45.421872    4544 cpu_manager.go:414] "containerMap: removing container" podUID="bd4c038f-d0d4-4fd5-ad51-2dbae210cb26" containerName="istio-proxy"
Oct 30 13:28:45.422032 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:45.421878    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd4c038f-d0d4-4fd5-ad51-2dbae210cb26" containerName="istio-proxy"
Oct 30 13:28:45.422135 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:45.422074    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd4c038f-d0d4-4fd5-ad51-2dbae210cb26" containerName="istio-proxy"
Oct 30 13:28:45.422135 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:45.422091    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd4c038f-d0d4-4fd5-ad51-2dbae210cb26" containerName="istio-proxy"
Oct 30 13:28:45.427420 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:45.427396    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl"
Oct 30 13:28:45.449322 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:45.449286    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl"]
Oct 30 13:28:45.456617 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:45.456584    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-1ba901-dockercfg-ccvkx\""
Oct 30 13:28:45.456874 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:45.456706    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-1ba901\""
Oct 30 13:28:45.473848 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:45.473814    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk"]
Oct 30 13:28:45.479177 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:45.479152    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk"
Oct 30 13:28:45.554387 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:45.539446    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b72rv\" (UniqueName: \"kubernetes.io/projected/63baae32-04f3-47e2-9b88-b04768e3f762-kube-api-access-b72rv\") pod \"payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl\" (UID: \"63baae32-04f3-47e2-9b88-b04768e3f762\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl"
Oct 30 13:28:45.554387 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:45.539568    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63baae32-04f3-47e2-9b88-b04768e3f762-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl\" (UID: \"63baae32-04f3-47e2-9b88-b04768e3f762\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl"
Oct 30 13:28:45.608247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:45.598466    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk"]
Oct 30 13:28:45.651246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:45.642010    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b72rv\" (UniqueName: \"kubernetes.io/projected/63baae32-04f3-47e2-9b88-b04768e3f762-kube-api-access-b72rv\") pod \"payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl\" (UID: \"63baae32-04f3-47e2-9b88-b04768e3f762\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl"
Oct 30 13:28:45.651246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:45.642111    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e6a87f-6868-4a77-b75f-c3e1b0b00018-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk\" (UID: \"a0e6a87f-6868-4a77-b75f-c3e1b0b00018\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk"
Oct 30 13:28:45.651246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:45.642140    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63baae32-04f3-47e2-9b88-b04768e3f762-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl\" (UID: \"63baae32-04f3-47e2-9b88-b04768e3f762\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl"
Oct 30 13:28:45.651246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:45.642193    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbv9s\" (UniqueName: \"kubernetes.io/projected/a0e6a87f-6868-4a77-b75f-c3e1b0b00018-kube-api-access-zbv9s\") pod \"payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk\" (UID: \"a0e6a87f-6868-4a77-b75f-c3e1b0b00018\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk"
Oct 30 13:28:45.651246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:45.643319    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63baae32-04f3-47e2-9b88-b04768e3f762-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl\" (UID: \"63baae32-04f3-47e2-9b88-b04768e3f762\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl"
Oct 30 13:28:45.705260 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:45.703788    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b72rv\" (UniqueName: \"kubernetes.io/projected/63baae32-04f3-47e2-9b88-b04768e3f762-kube-api-access-b72rv\") pod \"payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl\" (UID: \"63baae32-04f3-47e2-9b88-b04768e3f762\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl"
Oct 30 13:28:45.745164 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:45.743730    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl"
Oct 30 13:28:45.745164 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:45.744426    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e6a87f-6868-4a77-b75f-c3e1b0b00018-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk\" (UID: \"a0e6a87f-6868-4a77-b75f-c3e1b0b00018\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk"
Oct 30 13:28:45.745164 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:45.744578    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbv9s\" (UniqueName: \"kubernetes.io/projected/a0e6a87f-6868-4a77-b75f-c3e1b0b00018-kube-api-access-zbv9s\") pod \"payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk\" (UID: \"a0e6a87f-6868-4a77-b75f-c3e1b0b00018\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk"
Oct 30 13:28:45.746734 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:45.746507    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e6a87f-6868-4a77-b75f-c3e1b0b00018-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk\" (UID: \"a0e6a87f-6868-4a77-b75f-c3e1b0b00018\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk"
Oct 30 13:28:45.809596 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:45.809496    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbv9s\" (UniqueName: \"kubernetes.io/projected/a0e6a87f-6868-4a77-b75f-c3e1b0b00018-kube-api-access-zbv9s\") pod \"payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk\" (UID: \"a0e6a87f-6868-4a77-b75f-c3e1b0b00018\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk"
Oct 30 13:28:45.809978 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:45.809941    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk"
Oct 30 13:28:45.902824 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:45.902584    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"]
Oct 30 13:28:45.903034 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:45.902860    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj" podUID="287ea8cd-7fc1-4967-a291-adf6973f5967" containerName="istio-proxy" containerID="cri-o://8811fa1f9883d0d3b4048ba57103e1e34881d03683a3c5ee38215dfba4b6fc35" gracePeriod=30
Oct 30 13:28:46.251012 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:46.250980    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk"]
Oct 30 13:28:46.252549 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:28:46.252519    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e6a87f_6868_4a77_b75f_c3e1b0b00018.slice/crio-59e40b20a719722c01434a166be10294f6fbd8f94f7f29fe8fcab17c4891f87e WatchSource:0}: Error finding container 59e40b20a719722c01434a166be10294f6fbd8f94f7f29fe8fcab17c4891f87e: Status 404 returned error can't find the container with id 59e40b20a719722c01434a166be10294f6fbd8f94f7f29fe8fcab17c4891f87e
Oct 30 13:28:46.315998 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:46.315964    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl"]
Oct 30 13:28:46.509357 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:46.509264    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl" event={"ID":"63baae32-04f3-47e2-9b88-b04768e3f762","Type":"ContainerStarted","Data":"e205f35122b58e091e3688de4c77f65ab10f4234f5dca2d7e9341679a3682e9a"}
Oct 30 13:28:46.509357 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:46.509327    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl" event={"ID":"63baae32-04f3-47e2-9b88-b04768e3f762","Type":"ContainerStarted","Data":"221dbbdb33f4ee4fb333425bd096240f6afd41d047bbf0f4d32eaf18b4fe5126"}
Oct 30 13:28:46.509875 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:46.509410    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl"
Oct 30 13:28:46.511157 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:46.511129    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk" event={"ID":"a0e6a87f-6868-4a77-b75f-c3e1b0b00018","Type":"ContainerStarted","Data":"f22e25bd952e208ebf7fbfb99328b067de736c4827ba2eef4b5d3cc2922f7f67"}
Oct 30 13:28:46.511297 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:46.511177    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk" event={"ID":"a0e6a87f-6868-4a77-b75f-c3e1b0b00018","Type":"ContainerStarted","Data":"59e40b20a719722c01434a166be10294f6fbd8f94f7f29fe8fcab17c4891f87e"}
Oct 30 13:28:46.511297 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:46.511209    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk"
Oct 30 13:28:46.584561 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:46.584504    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl" podStartSLOduration=1.584489652 podStartE2EDuration="1.584489652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:46.582812673 +0000 UTC m=+2621.629946626" watchObservedRunningTime="2026-10-30 13:28:46.584489652 +0000 UTC m=+2621.631623604"
Oct 30 13:28:46.632662 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:46.632604    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk" podStartSLOduration=1.6325908789999999 podStartE2EDuration="1.632590879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:46.631953646 +0000 UTC m=+2621.679087600" watchObservedRunningTime="2026-10-30 13:28:46.632590879 +0000 UTC m=+2621.679724831"
Oct 30 13:28:46.692027 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:46.691988    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:47.101055 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:47.101014    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd"
Oct 30 13:28:47.114667 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:47.114637    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd"
Oct 30 13:28:47.531490 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:47.531402    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd"
Oct 30 13:28:48.493573 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:48.493527    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-w27kj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:28:48.493778 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:48.493588    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj" podUID="287ea8cd-7fc1-4967-a291-adf6973f5967" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:28:48.688963 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:48.688929    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:50.426643 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:50.426607    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:51.197012 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.196943    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 13:28:51.264856 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.264820    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-credential-socket\") pod \"287ea8cd-7fc1-4967-a291-adf6973f5967\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") "
Oct 30 13:28:51.265151 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.264892    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/287ea8cd-7fc1-4967-a291-adf6973f5967-istio-token\" (UniqueName: \"kubernetes.io/projected/287ea8cd-7fc1-4967-a291-adf6973f5967-istio-token\") pod \"287ea8cd-7fc1-4967-a291-adf6973f5967\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") "
Oct 30 13:28:51.265151 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.264925    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-istio-envoy\") pod \"287ea8cd-7fc1-4967-a291-adf6973f5967\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") "
Oct 30 13:28:51.265151 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.265006    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/287ea8cd-7fc1-4967-a291-adf6973f5967-kube-api-access-k76nq\" (UniqueName: \"kubernetes.io/projected/287ea8cd-7fc1-4967-a291-adf6973f5967-kube-api-access-k76nq\") pod \"287ea8cd-7fc1-4967-a291-adf6973f5967\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") "
Oct 30 13:28:51.265151 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.265088    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/287ea8cd-7fc1-4967-a291-adf6973f5967-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/287ea8cd-7fc1-4967-a291-adf6973f5967-istio-podinfo\") pod \"287ea8cd-7fc1-4967-a291-adf6973f5967\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") "
Oct 30 13:28:51.265465 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.265177    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-workload-socket\") pod \"287ea8cd-7fc1-4967-a291-adf6973f5967\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") "
Oct 30 13:28:51.265465 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.265332    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/287ea8cd-7fc1-4967-a291-adf6973f5967-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/287ea8cd-7fc1-4967-a291-adf6973f5967-istiod-ca-cert\") pod \"287ea8cd-7fc1-4967-a291-adf6973f5967\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") "
Oct 30 13:28:51.265580 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.265567    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-workload-certs\") pod \"287ea8cd-7fc1-4967-a291-adf6973f5967\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") "
Oct 30 13:28:51.265648 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.265602    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-istio-data\") pod \"287ea8cd-7fc1-4967-a291-adf6973f5967\" (UID: \"287ea8cd-7fc1-4967-a291-adf6973f5967\") "
Oct 30 13:28:51.265712 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.265642    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-credential-socket" pod "287ea8cd-7fc1-4967-a291-adf6973f5967" (UID: "287ea8cd-7fc1-4967-a291-adf6973f5967"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:51.265982 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.265951    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-istio-data" pod "287ea8cd-7fc1-4967-a291-adf6973f5967" (UID: "287ea8cd-7fc1-4967-a291-adf6973f5967"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:51.266396 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.266373    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:28:51.266396 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.266404    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:28:51.266655 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.266520    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-workload-certs" pod "287ea8cd-7fc1-4967-a291-adf6973f5967" (UID: "287ea8cd-7fc1-4967-a291-adf6973f5967"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:51.266915 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.266893    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/287ea8cd-7fc1-4967-a291-adf6973f5967-istiod-ca-cert" pod "287ea8cd-7fc1-4967-a291-adf6973f5967" (UID: "287ea8cd-7fc1-4967-a291-adf6973f5967"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:51.267608 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.267582    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-workload-socket" pod "287ea8cd-7fc1-4967-a291-adf6973f5967" (UID: "287ea8cd-7fc1-4967-a291-adf6973f5967"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:51.269966 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.269175    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-istio-envoy" pod "287ea8cd-7fc1-4967-a291-adf6973f5967" (UID: "287ea8cd-7fc1-4967-a291-adf6973f5967"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:51.269966 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.269817    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/287ea8cd-7fc1-4967-a291-adf6973f5967-istio-token" pod "287ea8cd-7fc1-4967-a291-adf6973f5967" (UID: "287ea8cd-7fc1-4967-a291-adf6973f5967"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:51.271070 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.271048    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/287ea8cd-7fc1-4967-a291-adf6973f5967-istio-podinfo" pod "287ea8cd-7fc1-4967-a291-adf6973f5967" (UID: "287ea8cd-7fc1-4967-a291-adf6973f5967"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:28:51.271631 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.271611    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/287ea8cd-7fc1-4967-a291-adf6973f5967-kube-api-access-k76nq" pod "287ea8cd-7fc1-4967-a291-adf6973f5967" (UID: "287ea8cd-7fc1-4967-a291-adf6973f5967"). InnerVolumeSpecName "kube-api-access-k76nq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:51.372607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.368245    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:28:51.372607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.368285    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/287ea8cd-7fc1-4967-a291-adf6973f5967-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:28:51.372607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.368299    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:28:51.372607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.368317    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k76nq\" (UniqueName: \"kubernetes.io/projected/287ea8cd-7fc1-4967-a291-adf6973f5967-kube-api-access-k76nq\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:28:51.372607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.368332    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/287ea8cd-7fc1-4967-a291-adf6973f5967-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:28:51.372607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.368346    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/287ea8cd-7fc1-4967-a291-adf6973f5967-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:28:51.372607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.368360    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/287ea8cd-7fc1-4967-a291-adf6973f5967-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:28:51.554920 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.554833    4544 generic.go:356] "Generic (PLEG): container finished" podID="287ea8cd-7fc1-4967-a291-adf6973f5967" containerID="8811fa1f9883d0d3b4048ba57103e1e34881d03683a3c5ee38215dfba4b6fc35" exitCode=0
Oct 30 13:28:51.554920 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.554890    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"
Oct 30 13:28:51.555485 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.554918    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj" event={"ID":"287ea8cd-7fc1-4967-a291-adf6973f5967","Type":"ContainerDied","Data":"8811fa1f9883d0d3b4048ba57103e1e34881d03683a3c5ee38215dfba4b6fc35"}
Oct 30 13:28:51.555485 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.554987    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj" event={"ID":"287ea8cd-7fc1-4967-a291-adf6973f5967","Type":"ContainerDied","Data":"6e9651e2d28d618d0ad35d88509b54c9b88a3192341d0e5a1c199f7b9ac2beb6"}
Oct 30 13:28:51.555485 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.555018    4544 scope.go:122] "RemoveContainer" containerID="8811fa1f9883d0d3b4048ba57103e1e34881d03683a3c5ee38215dfba4b6fc35"
Oct 30 13:28:51.567898 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.567876    4544 scope.go:122] "RemoveContainer" containerID="8811fa1f9883d0d3b4048ba57103e1e34881d03683a3c5ee38215dfba4b6fc35"
Oct 30 13:28:51.568258 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:28:51.568213    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8811fa1f9883d0d3b4048ba57103e1e34881d03683a3c5ee38215dfba4b6fc35\": container with ID starting with 8811fa1f9883d0d3b4048ba57103e1e34881d03683a3c5ee38215dfba4b6fc35 not found: ID does not exist" containerID="8811fa1f9883d0d3b4048ba57103e1e34881d03683a3c5ee38215dfba4b6fc35"
Oct 30 13:28:51.568320 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.568260    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8811fa1f9883d0d3b4048ba57103e1e34881d03683a3c5ee38215dfba4b6fc35"} err="failed to get container status \"8811fa1f9883d0d3b4048ba57103e1e34881d03683a3c5ee38215dfba4b6fc35\": rpc error: code = NotFound desc = could not find container \"8811fa1f9883d0d3b4048ba57103e1e34881d03683a3c5ee38215dfba4b6fc35\": container with ID starting with 8811fa1f9883d0d3b4048ba57103e1e34881d03683a3c5ee38215dfba4b6fc35 not found: ID does not exist"
Oct 30 13:28:51.611701 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.611665    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"]
Oct 30 13:28:51.624068 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:51.624028    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj"]
Oct 30 13:28:52.030034 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:52.029998    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-547c0f1d-mq9j4"]
Oct 30 13:28:52.034070 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:52.034043    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-547c0f1d-mq9j4"]
Oct 30 13:28:52.185873 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:52.185837    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:53.088567 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:53.088531    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="287ea8cd-7fc1-4967-a291-adf6973f5967" path="/var/lib/kubelet/pods/287ea8cd-7fc1-4967-a291-adf6973f5967/volumes"
Oct 30 13:28:53.089088 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:53.089035    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8403cb99-bff9-4129-b35b-ff6fd0af6f19" path="/var/lib/kubelet/pods/8403cb99-bff9-4129-b35b-ff6fd0af6f19/volumes"
Oct 30 13:28:54.073481 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.073426    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:54.233564 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.233526    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"]
Oct 30 13:28:54.234330 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.234314    4544 cpu_manager.go:414] "containerMap: removing container" podUID="287ea8cd-7fc1-4967-a291-adf6973f5967" containerName="istio-proxy"
Oct 30 13:28:54.234330 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.234330    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="287ea8cd-7fc1-4967-a291-adf6973f5967" containerName="istio-proxy"
Oct 30 13:28:54.234480 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.234470    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="287ea8cd-7fc1-4967-a291-adf6973f5967" containerName="istio-proxy"
Oct 30 13:28:54.239481 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.239453    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:28:54.261065 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.261029    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-8a5d5e-openshift-default-dockercfg-t586t\""
Oct 30 13:28:54.275600 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.275561    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"]
Oct 30 13:28:54.313818 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.313780    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-istio-envoy\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:28:54.313818 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.313820    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f03ae5-629b-4a35-8698-0043f221428d-istiod-ca-cert\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:28:54.314095 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.313841    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09f03ae5-629b-4a35-8698-0043f221428d-istio-token\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:28:54.314170 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.314089    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-credential-socket\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:28:54.314170 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.314138    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f03ae5-629b-4a35-8698-0043f221428d-istio-podinfo\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:28:54.314284 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.314266    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-istio-data\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:28:54.314330 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.314314    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-workload-socket\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:28:54.314439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.314423    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkbbv\" (UniqueName: \"kubernetes.io/projected/09f03ae5-629b-4a35-8698-0043f221428d-kube-api-access-mkbbv\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:28:54.314479 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.314453    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-workload-certs\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:28:54.415505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.415410    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-istio-envoy\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:28:54.415505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.415461    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f03ae5-629b-4a35-8698-0043f221428d-istiod-ca-cert\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:28:54.415505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.415498    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09f03ae5-629b-4a35-8698-0043f221428d-istio-token\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:28:54.415811 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.415597    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-credential-socket\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:28:54.415811 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.415630    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f03ae5-629b-4a35-8698-0043f221428d-istio-podinfo\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:28:54.415811 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.415718    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-istio-data\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:28:54.415811 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.415778    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-workload-socket\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:28:54.416026 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.415864    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkbbv\" (UniqueName: \"kubernetes.io/projected/09f03ae5-629b-4a35-8698-0043f221428d-kube-api-access-mkbbv\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:28:54.416026 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.415900    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-workload-certs\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:28:54.416149 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.416087    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-istio-data\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:28:54.416277 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.416251    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-workload-socket\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:28:54.416351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.416292    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f03ae5-629b-4a35-8698-0043f221428d-istiod-ca-cert\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:28:54.416400 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.416357    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-credential-socket\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:28:54.416400 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.416386    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-workload-certs\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:28:54.417829 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.417806    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-istio-envoy\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:28:54.418142 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.418124    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f03ae5-629b-4a35-8698-0043f221428d-istio-podinfo\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:28:54.438953 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.438917    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09f03ae5-629b-4a35-8698-0043f221428d-istio-token\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:28:54.439503 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.439476    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkbbv\" (UniqueName: \"kubernetes.io/projected/09f03ae5-629b-4a35-8698-0043f221428d-kube-api-access-mkbbv\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:28:54.558064 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.558021    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:28:54.889083 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.889049    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"]
Oct 30 13:28:54.891028 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:28:54.890999    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f03ae5_629b_4a35_8698_0043f221428d.slice/crio-2600921f1c88052af48eb51fd648b4f2bd607ec2f88026e2b0fad98be656b4af WatchSource:0}: Error finding container 2600921f1c88052af48eb51fd648b4f2bd607ec2f88026e2b0fad98be656b4af: Status 404 returned error can't find the container with id 2600921f1c88052af48eb51fd648b4f2bd607ec2f88026e2b0fad98be656b4af
Oct 30 13:28:54.894046 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.894006    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:28:54.894145 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.894087    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:28:54.894145 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:54.894122    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:28:55.598080 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:55.598031    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz" event={"ID":"09f03ae5-629b-4a35-8698-0043f221428d","Type":"ContainerStarted","Data":"731b2617b6db797a5b6f2dd2610ba69d6340c307926fee8f2c2c00599bf2cecc"}
Oct 30 13:28:55.598554 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:55.598102    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz" event={"ID":"09f03ae5-629b-4a35-8698-0043f221428d","Type":"ContainerStarted","Data":"2600921f1c88052af48eb51fd648b4f2bd607ec2f88026e2b0fad98be656b4af"}
Oct 30 13:28:55.680883 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:55.680820    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz" podStartSLOduration=1.680805222 podStartE2EDuration="1.680805222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:55.665792976 +0000 UTC m=+2630.712926931" watchObservedRunningTime="2026-10-30 13:28:55.680805222 +0000 UTC m=+2630.727939174"
Oct 30 13:28:55.868178 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:55.868077    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:56.558902 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:56.558865    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:28:56.560576 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:56.560547    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Oct 30 13:28:56.560700 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:56.560603    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz" podUID="09f03ae5-629b-4a35-8698-0043f221428d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Oct 30 13:28:56.985862 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:56.985821    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:57.203779 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:57.203745    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:57.539285 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:57.539257    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk"
Oct 30 13:28:57.539570 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:57.539556    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl"
Oct 30 13:28:57.558546 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:57.558514    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Oct 30 13:28:57.558724 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:57.558564    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz" podUID="09f03ae5-629b-4a35-8698-0043f221428d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Oct 30 13:28:57.786035 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:57.785989    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:28:58.559053 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:58.558955    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Oct 30 13:28:58.559053 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:58.559018    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz" podUID="09f03ae5-629b-4a35-8698-0043f221428d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Oct 30 13:28:59.008731 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:28:59.008697    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:28:59.559028 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:59.558991    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Oct 30 13:28:59.559248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:59.559044    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz" podUID="09f03ae5-629b-4a35-8698-0043f221428d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Oct 30 13:28:59.739260 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:28:59.735697    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:00.559283 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:00.559244    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Oct 30 13:29:00.559716 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:00.559309    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz" podUID="09f03ae5-629b-4a35-8698-0043f221428d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Oct 30 13:29:01.622743 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:01.622708    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:29:01.623280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:01.623213    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:29:01.623998 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:01.623978    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:29:01.807860 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:01.807825    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:02.114137 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:02.114096    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:02.581030 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:02.580998    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:03.949816 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:03.949776    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:04.065751 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:04.065716    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:04.682025 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:04.681988    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:05.731707 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:05.731675    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:07.570118 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:07.570064    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:09.363793 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:09.363755    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:10.031296 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:10.031258    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e318c72d-htcc6"]
Oct 30 13:29:10.043536 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:10.043504    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e318c72d-htcc6"]
Oct 30 13:29:11.089128 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:11.089088    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95b60838-fbc5-41fe-be77-97c4101e6109" path="/var/lib/kubelet/pods/95b60838-fbc5-41fe-be77-97c4101e6109/volumes"
Oct 30 13:29:11.121988 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:11.121948    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:12.939685 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:12.939650    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:13.863793 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:13.863760    4544 scope.go:122] "RemoveContainer" containerID="8cc767c52b53d99484cf704e793c8d70899ac29782298a414bb938de35546d61"
Oct 30 13:29:13.874428 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:13.874407    4544 scope.go:122] "RemoveContainer" containerID="a4fe84d90cb5fbb0c1ea3e625479f530b7214a0948353b81ec20bf213122ba10"
Oct 30 13:29:13.885356 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:13.885330    4544 scope.go:122] "RemoveContainer" containerID="6a788f8c95b3baba1affc198e9443c9273445a04e4ab3ff795099857fb60d511"
Oct 30 13:29:13.900390 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:13.900364    4544 scope.go:122] "RemoveContainer" containerID="e45b706db4fb5d8c7b23fba6a6030c0d31ff3a15e2c7f55d23323d901c7c2051"
Oct 30 13:29:13.960434 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:13.960407    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm"]
Oct 30 13:29:14.052570 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:14.051643    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:15.177494 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:15.177430    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:17.061185 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:17.061098    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:18.442811 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:18.442769    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:19.156597 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.156552    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:19.326049 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.325980    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:19.655089 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.655043    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:19.665316 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.665285    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"]
Oct 30 13:29:19.672107 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.672079    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:29:19.687805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.687776    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-76405c-openshift-default-dockercfg-xgccv\""
Oct 30 13:29:19.719503 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.719463    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"]
Oct 30 13:29:19.731939 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.731895    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-istio-data\") pod \"e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") " pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:29:19.732130 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.731973    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-istio-envoy\") pod \"e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") " pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:29:19.732130 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.732121    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-credential-socket\") pod \"e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") " pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:29:19.732280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.732240    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-workload-certs\") pod \"e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") " pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:29:19.732346 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.732316    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/820333af-678e-40c7-a1b7-24cab9591ea2-istio-podinfo\") pod \"e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") " pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:29:19.732430 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.732405    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhzhv\" (UniqueName: \"kubernetes.io/projected/820333af-678e-40c7-a1b7-24cab9591ea2-kube-api-access-vhzhv\") pod \"e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") " pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:29:19.732523 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.732496    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/820333af-678e-40c7-a1b7-24cab9591ea2-istio-token\") pod \"e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") " pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:29:19.732672 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.732652    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/820333af-678e-40c7-a1b7-24cab9591ea2-istiod-ca-cert\") pod \"e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") " pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:29:19.732738 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.732723    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-workload-socket\") pod \"e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") " pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:29:19.847242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.836862    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/820333af-678e-40c7-a1b7-24cab9591ea2-istiod-ca-cert\") pod \"e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") " pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:29:19.847242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.836963    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-workload-socket\") pod \"e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") " pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:29:19.847242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.837004    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-istio-data\") pod \"e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") " pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:29:19.847242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.837057    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-istio-envoy\") pod \"e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") " pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:29:19.847242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.837162    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-credential-socket\") pod \"e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") " pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:29:19.847242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.837248    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-workload-certs\") pod \"e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") " pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:29:19.847242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.837296    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/820333af-678e-40c7-a1b7-24cab9591ea2-istio-podinfo\") pod \"e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") " pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:29:19.847242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.837363    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhzhv\" (UniqueName: \"kubernetes.io/projected/820333af-678e-40c7-a1b7-24cab9591ea2-kube-api-access-vhzhv\") pod \"e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") " pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:29:19.847242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.837402    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/820333af-678e-40c7-a1b7-24cab9591ea2-istio-token\") pod \"e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") " pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:29:19.847242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.838730    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/820333af-678e-40c7-a1b7-24cab9591ea2-istiod-ca-cert\") pod \"e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") " pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:29:19.847242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.839039    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-workload-socket\") pod \"e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") " pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:29:19.916293 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.912432    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-istio-data\") pod \"e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") " pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:29:19.921414 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.916553    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-credential-socket\") pod \"e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") " pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:29:19.921414 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.919879    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-istio-envoy\") pod \"e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") " pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:29:19.921414 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.920503    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/820333af-678e-40c7-a1b7-24cab9591ea2-istio-token\") pod \"e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") " pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:29:19.921414 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.920638    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/820333af-678e-40c7-a1b7-24cab9591ea2-istio-podinfo\") pod \"e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") " pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:29:19.922166 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.922143    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-workload-certs\") pod \"e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") " pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:29:19.928096 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.928045    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhzhv\" (UniqueName: \"kubernetes.io/projected/820333af-678e-40c7-a1b7-24cab9591ea2-kube-api-access-vhzhv\") pod \"e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") " pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:29:19.962917 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.962795    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:19.988490 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:19.988452    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:29:20.131167 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:20.131128    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:20.347858 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:20.347826    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"]
Oct 30 13:29:20.350390 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:29:20.350358    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod820333af_678e_40c7_a1b7_24cab9591ea2.slice/crio-22fffec9aab71f0764f5f003683a7a98eae5a03441d5469b21a96dff657ea312 WatchSource:0}: Error finding container 22fffec9aab71f0764f5f003683a7a98eae5a03441d5469b21a96dff657ea312: Status 404 returned error can't find the container with id 22fffec9aab71f0764f5f003683a7a98eae5a03441d5469b21a96dff657ea312
Oct 30 13:29:20.353550 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:20.353504    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:29:20.353646 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:20.353618    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:29:20.353755 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:20.353663    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:29:20.953933 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:20.953893    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7" event={"ID":"820333af-678e-40c7-a1b7-24cab9591ea2","Type":"ContainerStarted","Data":"2c60a75236026d1d1f0f915292164c351adeebb4e0101cae02351b5b402b083f"}
Oct 30 13:29:20.954334 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:20.953960    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7" event={"ID":"820333af-678e-40c7-a1b7-24cab9591ea2","Type":"ContainerStarted","Data":"22fffec9aab71f0764f5f003683a7a98eae5a03441d5469b21a96dff657ea312"}
Oct 30 13:29:20.988810 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:20.988773    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:29:21.028498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:21.028441    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7" podStartSLOduration=2.028426554 podStartE2EDuration="2.028426554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:21.028375524 +0000 UTC m=+2656.075509478" watchObservedRunningTime="2026-10-30 13:29:21.028426554 +0000 UTC m=+2656.075560506"
Oct 30 13:29:21.175478 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:21.169525    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns"]
Oct 30 13:29:21.187243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:21.184683    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns"
Oct 30 13:29:21.215310 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:21.200875    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-1ba901-serving-cert\""
Oct 30 13:29:21.219076 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:21.219038    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns"]
Oct 30 13:29:21.312569 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:21.312532    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:21.379246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:21.375913    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6cbt\" (UniqueName: \"kubernetes.io/projected/501f27bf-f8e6-4e51-97be-9383291b04ff-kube-api-access-x6cbt\") pod \"maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns\" (UID: \"501f27bf-f8e6-4e51-97be-9383291b04ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns"
Oct 30 13:29:21.379246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:21.376072    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/501f27bf-f8e6-4e51-97be-9383291b04ff-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns\" (UID: \"501f27bf-f8e6-4e51-97be-9383291b04ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns"
Oct 30 13:29:21.379246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:21.376238    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/501f27bf-f8e6-4e51-97be-9383291b04ff-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns\" (UID: \"501f27bf-f8e6-4e51-97be-9383291b04ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns"
Oct 30 13:29:21.379246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:21.376275    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/501f27bf-f8e6-4e51-97be-9383291b04ff-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns\" (UID: \"501f27bf-f8e6-4e51-97be-9383291b04ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns"
Oct 30 13:29:21.431813 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:21.431774    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:21.477377 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:21.477291    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6cbt\" (UniqueName: \"kubernetes.io/projected/501f27bf-f8e6-4e51-97be-9383291b04ff-kube-api-access-x6cbt\") pod \"maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns\" (UID: \"501f27bf-f8e6-4e51-97be-9383291b04ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns"
Oct 30 13:29:21.477548 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:21.477429    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/501f27bf-f8e6-4e51-97be-9383291b04ff-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns\" (UID: \"501f27bf-f8e6-4e51-97be-9383291b04ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns"
Oct 30 13:29:21.477596 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:21.477560    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/501f27bf-f8e6-4e51-97be-9383291b04ff-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns\" (UID: \"501f27bf-f8e6-4e51-97be-9383291b04ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns"
Oct 30 13:29:21.477596 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:21.477588    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/501f27bf-f8e6-4e51-97be-9383291b04ff-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns\" (UID: \"501f27bf-f8e6-4e51-97be-9383291b04ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns"
Oct 30 13:29:21.478230 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:21.478194    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/501f27bf-f8e6-4e51-97be-9383291b04ff-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns\" (UID: \"501f27bf-f8e6-4e51-97be-9383291b04ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns"
Oct 30 13:29:21.480138 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:21.480106    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/501f27bf-f8e6-4e51-97be-9383291b04ff-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns\" (UID: \"501f27bf-f8e6-4e51-97be-9383291b04ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns"
Oct 30 13:29:21.480259 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:21.480183    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/501f27bf-f8e6-4e51-97be-9383291b04ff-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns\" (UID: \"501f27bf-f8e6-4e51-97be-9383291b04ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns"
Oct 30 13:29:21.506871 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:21.506838    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6cbt\" (UniqueName: \"kubernetes.io/projected/501f27bf-f8e6-4e51-97be-9383291b04ff-kube-api-access-x6cbt\") pod \"maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns\" (UID: \"501f27bf-f8e6-4e51-97be-9383291b04ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns"
Oct 30 13:29:21.514776 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:21.514744    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns"
Oct 30 13:29:21.825555 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:21.825523    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns"]
Oct 30 13:29:21.828967 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:29:21.828935    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod501f27bf_f8e6_4e51_97be_9383291b04ff.slice/crio-43055af08234a41a1d0d776fcdfe413626f6c2cffe663345a7cace37afc7a44d WatchSource:0}: Error finding container 43055af08234a41a1d0d776fcdfe413626f6c2cffe663345a7cace37afc7a44d: Status 404 returned error can't find the container with id 43055af08234a41a1d0d776fcdfe413626f6c2cffe663345a7cace37afc7a44d
Oct 30 13:29:21.963283 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:21.963248    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns" event={"ID":"501f27bf-f8e6-4e51-97be-9383291b04ff","Type":"ContainerStarted","Data":"43055af08234a41a1d0d776fcdfe413626f6c2cffe663345a7cace37afc7a44d"}
Oct 30 13:29:21.996470 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:21.996436    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:29:22.973231 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:22.973192    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns" event={"ID":"501f27bf-f8e6-4e51-97be-9383291b04ff","Type":"ContainerStarted","Data":"3ee5798a02c755b6b56c847b265d54a61920bc6ad0a842540b12d90780fc40e0"}
Oct 30 13:29:22.973750 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:22.973418    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns"
Oct 30 13:29:22.973750 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:22.973459    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:29:22.974412 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:22.974391    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:29:23.021186 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:23.021115    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns" podStartSLOduration=1.703287558 podStartE2EDuration="2.021095075s" totalImagesPullingTime="317.807517ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:23.015920403 +0000 UTC m=+2658.063054358" watchObservedRunningTime="2026-10-30 13:29:23.021095075 +0000 UTC m=+2658.068229031"
Oct 30 13:29:23.440019 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:23.439976    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:23.584130 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:23.584081    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:23.711988 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:23.711909    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:25.573454 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:25.573413    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:26.171117 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:26.171082    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:26.297164 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:26.297114    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:26.353197 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:26.353163    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58"]
Oct 30 13:29:26.358391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:26.358369    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58"
Oct 30 13:29:26.373648 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:26.373624    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-76405c\""
Oct 30 13:29:26.391620 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:26.391580    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58"]
Oct 30 13:29:26.394365 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:26.394331    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-76405c-dockercfg-gvtnv\""
Oct 30 13:29:26.495300 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:26.481316    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p67cc\" (UniqueName: \"kubernetes.io/projected/7dddf370-d1a3-44af-bb88-aad50dccee04-kube-api-access-p67cc\") pod \"payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58\" (UID: \"7dddf370-d1a3-44af-bb88-aad50dccee04\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58"
Oct 30 13:29:26.495300 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:26.481427    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dddf370-d1a3-44af-bb88-aad50dccee04-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58\" (UID: \"7dddf370-d1a3-44af-bb88-aad50dccee04\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58"
Oct 30 13:29:26.608060 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:26.602337    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p67cc\" (UniqueName: \"kubernetes.io/projected/7dddf370-d1a3-44af-bb88-aad50dccee04-kube-api-access-p67cc\") pod \"payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58\" (UID: \"7dddf370-d1a3-44af-bb88-aad50dccee04\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58"
Oct 30 13:29:26.608060 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:26.602477    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dddf370-d1a3-44af-bb88-aad50dccee04-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58\" (UID: \"7dddf370-d1a3-44af-bb88-aad50dccee04\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58"
Oct 30 13:29:26.608060 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:26.603796    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dddf370-d1a3-44af-bb88-aad50dccee04-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58\" (UID: \"7dddf370-d1a3-44af-bb88-aad50dccee04\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58"
Oct 30 13:29:26.614170 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:26.614142    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p67cc\" (UniqueName: \"kubernetes.io/projected/7dddf370-d1a3-44af-bb88-aad50dccee04-kube-api-access-p67cc\") pod \"payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58\" (UID: \"7dddf370-d1a3-44af-bb88-aad50dccee04\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58"
Oct 30 13:29:26.673702 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:26.673667    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58"
Oct 30 13:29:26.994670 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:26.994636    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58"]
Oct 30 13:29:26.995941 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:29:26.995911    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dddf370_d1a3_44af_bb88_aad50dccee04.slice/crio-d5e916ad52ff063ad2db47359c58968f62a61ba23dfd10b9d30ee8c97d1d3fca WatchSource:0}: Error finding container d5e916ad52ff063ad2db47359c58968f62a61ba23dfd10b9d30ee8c97d1d3fca: Status 404 returned error can't find the container with id d5e916ad52ff063ad2db47359c58968f62a61ba23dfd10b9d30ee8c97d1d3fca
Oct 30 13:29:27.007735 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:27.007706    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58" event={"ID":"7dddf370-d1a3-44af-bb88-aad50dccee04","Type":"ContainerStarted","Data":"d5e916ad52ff063ad2db47359c58968f62a61ba23dfd10b9d30ee8c97d1d3fca"}
Oct 30 13:29:27.679130 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:27.679095    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:28.017965 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:28.017878    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58" event={"ID":"7dddf370-d1a3-44af-bb88-aad50dccee04","Type":"ContainerStarted","Data":"7195dbca2013e4d9b52302ffda80742fe2152416d6da696a00c54c7fd0042f7a"}
Oct 30 13:29:28.018146 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:28.017983    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58"
Oct 30 13:29:28.062280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:28.062213    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58" podStartSLOduration=2.062197462 podStartE2EDuration="2.062197462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:28.05842202 +0000 UTC m=+2663.105555966" watchObservedRunningTime="2026-10-30 13:29:28.062197462 +0000 UTC m=+2663.109331415"
Oct 30 13:29:28.985657 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:28.985617    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm" podUID="9af76539-ed44-4913-92cd-be61edf063ad" containerName="main" containerID="cri-o://7d449d4a016f4324d251233efa7e27663c0234c16046d426897be656605926cc" gracePeriod=45
Oct 30 13:29:28.986595 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:28.986575    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns"
Oct 30 13:29:28.994521 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:28.994487    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm" podUID="9af76539-ed44-4913-92cd-be61edf063ad" containerName="main" probeResult="failure" output="Get \"https://10.133.1.0:8000/ready\": dial tcp 10.133.1.0:8000: connect: connection refused"
Oct 30 13:29:29.009412 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:29:29.009386    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:29:29.283402 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:29.283376    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm"
Oct 30 13:29:29.462516 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:29.462479    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9af76539-ed44-4913-92cd-be61edf063ad-home\" (UniqueName: \"kubernetes.io/empty-dir/9af76539-ed44-4913-92cd-be61edf063ad-home\") pod \"9af76539-ed44-4913-92cd-be61edf063ad\" (UID: \"9af76539-ed44-4913-92cd-be61edf063ad\") "
Oct 30 13:29:29.462729 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:29.462561    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9af76539-ed44-4913-92cd-be61edf063ad-tls-certs\" (UniqueName: \"kubernetes.io/secret/9af76539-ed44-4913-92cd-be61edf063ad-tls-certs\") pod \"9af76539-ed44-4913-92cd-be61edf063ad\" (UID: \"9af76539-ed44-4913-92cd-be61edf063ad\") "
Oct 30 13:29:29.462729 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:29.462595    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9af76539-ed44-4913-92cd-be61edf063ad-kube-api-access-x5wst\" (UniqueName: \"kubernetes.io/projected/9af76539-ed44-4913-92cd-be61edf063ad-kube-api-access-x5wst\") pod \"9af76539-ed44-4913-92cd-be61edf063ad\" (UID: \"9af76539-ed44-4913-92cd-be61edf063ad\") "
Oct 30 13:29:29.462729 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:29.462682    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9af76539-ed44-4913-92cd-be61edf063ad-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9af76539-ed44-4913-92cd-be61edf063ad-model-cache\") pod \"9af76539-ed44-4913-92cd-be61edf063ad\" (UID: \"9af76539-ed44-4913-92cd-be61edf063ad\") "
Oct 30 13:29:29.462937 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:29.462770    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9af76539-ed44-4913-92cd-be61edf063ad-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9af76539-ed44-4913-92cd-be61edf063ad-dshm\") pod \"9af76539-ed44-4913-92cd-be61edf063ad\" (UID: \"9af76539-ed44-4913-92cd-be61edf063ad\") "
Oct 30 13:29:29.462937 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:29.462816    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9af76539-ed44-4913-92cd-be61edf063ad-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9af76539-ed44-4913-92cd-be61edf063ad-tmp-dir\") pod \"9af76539-ed44-4913-92cd-be61edf063ad\" (UID: \"9af76539-ed44-4913-92cd-be61edf063ad\") "
Oct 30 13:29:29.462937 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:29.462823    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af76539-ed44-4913-92cd-be61edf063ad-home" pod "9af76539-ed44-4913-92cd-be61edf063ad" (UID: "9af76539-ed44-4913-92cd-be61edf063ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:29.463080 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:29.463014    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af76539-ed44-4913-92cd-be61edf063ad-model-cache" pod "9af76539-ed44-4913-92cd-be61edf063ad" (UID: "9af76539-ed44-4913-92cd-be61edf063ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:29.463205 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:29.463173    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af76539-ed44-4913-92cd-be61edf063ad-tmp-dir" pod "9af76539-ed44-4913-92cd-be61edf063ad" (UID: "9af76539-ed44-4913-92cd-be61edf063ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:29.463757 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:29.463733    4544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9af76539-ed44-4913-92cd-be61edf063ad-home\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:29:29.463954 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:29.463760    4544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9af76539-ed44-4913-92cd-be61edf063ad-model-cache\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:29:29.464046 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:29.463956    4544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9af76539-ed44-4913-92cd-be61edf063ad-tmp-dir\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:29:29.465257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:29.465212    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9af76539-ed44-4913-92cd-be61edf063ad-tls-certs" pod "9af76539-ed44-4913-92cd-be61edf063ad" (UID: "9af76539-ed44-4913-92cd-be61edf063ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:29:29.465355 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:29.465303    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af76539-ed44-4913-92cd-be61edf063ad-dshm" pod "9af76539-ed44-4913-92cd-be61edf063ad" (UID: "9af76539-ed44-4913-92cd-be61edf063ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:29.465404 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:29.465370    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9af76539-ed44-4913-92cd-be61edf063ad-kube-api-access-x5wst" pod "9af76539-ed44-4913-92cd-be61edf063ad" (UID: "9af76539-ed44-4913-92cd-be61edf063ad"). InnerVolumeSpecName "kube-api-access-x5wst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:29.474605 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:29.474574    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:29.565960 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:29.565860    4544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9af76539-ed44-4913-92cd-be61edf063ad-tls-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:29:29.565960 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:29.565900    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5wst\" (UniqueName: \"kubernetes.io/projected/9af76539-ed44-4913-92cd-be61edf063ad-kube-api-access-x5wst\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:29:29.565960 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:29.565913    4544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9af76539-ed44-4913-92cd-be61edf063ad-dshm\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:29:30.035062 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:30.035026    4544 generic.go:356] "Generic (PLEG): container finished" podID="9af76539-ed44-4913-92cd-be61edf063ad" containerID="7d449d4a016f4324d251233efa7e27663c0234c16046d426897be656605926cc" exitCode=0
Oct 30 13:29:30.035548 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:30.035087    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm"
Oct 30 13:29:30.035548 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:30.035089    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm" event={"ID":"9af76539-ed44-4913-92cd-be61edf063ad","Type":"ContainerDied","Data":"7d449d4a016f4324d251233efa7e27663c0234c16046d426897be656605926cc"}
Oct 30 13:29:30.035548 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:30.035193    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm" event={"ID":"9af76539-ed44-4913-92cd-be61edf063ad","Type":"ContainerDied","Data":"d90cba6968971d63caf215506aafd09f6ca376df9aedf0701dcd0c652cc5257b"}
Oct 30 13:29:30.035548 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:30.035209    4544 scope.go:122] "RemoveContainer" containerID="7d449d4a016f4324d251233efa7e27663c0234c16046d426897be656605926cc"
Oct 30 13:29:30.046908 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:30.046879    4544 scope.go:122] "RemoveContainer" containerID="7d449d4a016f4324d251233efa7e27663c0234c16046d426897be656605926cc"
Oct 30 13:29:30.047230 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:29:30.047197    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d449d4a016f4324d251233efa7e27663c0234c16046d426897be656605926cc\": container with ID starting with 7d449d4a016f4324d251233efa7e27663c0234c16046d426897be656605926cc not found: ID does not exist" containerID="7d449d4a016f4324d251233efa7e27663c0234c16046d426897be656605926cc"
Oct 30 13:29:30.047285 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:30.047244    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d449d4a016f4324d251233efa7e27663c0234c16046d426897be656605926cc"} err="failed to get container status \"7d449d4a016f4324d251233efa7e27663c0234c16046d426897be656605926cc\": rpc error: code = NotFound desc = could not find container \"7d449d4a016f4324d251233efa7e27663c0234c16046d426897be656605926cc\": container with ID starting with 7d449d4a016f4324d251233efa7e27663c0234c16046d426897be656605926cc not found: ID does not exist"
Oct 30 13:29:30.089076 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:30.089043    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm"]
Oct 30 13:29:30.094417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:30.094388    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-e9d641/sub-test-model-e9d641-kserve-594cb445dc-mc7jm"]
Oct 30 13:29:31.089438 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:31.089399    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9af76539-ed44-4913-92cd-be61edf063ad" path="/var/lib/kubelet/pods/9af76539-ed44-4913-92cd-be61edf063ad/volumes"
Oct 30 13:29:31.216959 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:31.216920    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:32.988038 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:32.987996    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:34.463174 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:34.463129    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:34.720030 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:34.719946    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:36.538523 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:36.538485    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:38.386006 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.385966    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:38.662261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.662145    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"]
Oct 30 13:29:38.663012 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.662987    4544 cpu_manager.go:414] "containerMap: removing container" podUID="9af76539-ed44-4913-92cd-be61edf063ad" containerName="main"
Oct 30 13:29:38.663012 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.663004    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9af76539-ed44-4913-92cd-be61edf063ad" containerName="main"
Oct 30 13:29:38.663124 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.663115    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="9af76539-ed44-4913-92cd-be61edf063ad" containerName="main"
Oct 30 13:29:38.666386 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.666369    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:29:38.709002 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.708960    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"]
Oct 30 13:29:38.796006 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.795968    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f559800f-587f-4681-8e8c-5694a16cff06-istio-token\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:29:38.796211 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.796041    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f559800f-587f-4681-8e8c-5694a16cff06-istiod-ca-cert\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:29:38.796211 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.796159    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-istio-envoy\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:29:38.796370 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.796260    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-workload-socket\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:29:38.796370 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.796355    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-credential-socket\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:29:38.796449 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.796388    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f559800f-587f-4681-8e8c-5694a16cff06-istio-podinfo\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:29:38.796449 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.796426    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2v4m\" (UniqueName: \"kubernetes.io/projected/f559800f-587f-4681-8e8c-5694a16cff06-kube-api-access-d2v4m\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:29:38.796526 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.796493    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-istio-data\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:29:38.796563 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.796531    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-workload-certs\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:29:38.898420 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.898379    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f559800f-587f-4681-8e8c-5694a16cff06-istio-token\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:29:38.898621 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.898487    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f559800f-587f-4681-8e8c-5694a16cff06-istiod-ca-cert\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:29:38.898621 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.898558    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-istio-envoy\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:29:38.898621 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.898619    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-workload-socket\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:29:38.898817 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.898690    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-credential-socket\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:29:38.898817 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.898726    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f559800f-587f-4681-8e8c-5694a16cff06-istio-podinfo\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:29:38.898817 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.898761    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2v4m\" (UniqueName: \"kubernetes.io/projected/f559800f-587f-4681-8e8c-5694a16cff06-kube-api-access-d2v4m\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:29:38.898983 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.898840    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-istio-data\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:29:38.898983 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.898868    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-workload-certs\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:29:38.899406 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.899371    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f559800f-587f-4681-8e8c-5694a16cff06-istiod-ca-cert\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:29:38.899625 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.899604    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-workload-socket\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:29:38.899749 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.899643    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-istio-data\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:29:38.899749 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.899653    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-workload-certs\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:29:38.899962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.899940    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-credential-socket\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:29:38.901745 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.901719    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-istio-envoy\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:29:38.902052 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.902032    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f559800f-587f-4681-8e8c-5694a16cff06-istio-podinfo\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:29:38.924524 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.924412    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f559800f-587f-4681-8e8c-5694a16cff06-istio-token\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:29:38.924697 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.924624    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2v4m\" (UniqueName: \"kubernetes.io/projected/f559800f-587f-4681-8e8c-5694a16cff06-kube-api-access-d2v4m\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:29:38.983142 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:38.983039    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:29:39.041013 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:39.040978    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58"
Oct 30 13:29:39.317193 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:39.317167    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"]
Oct 30 13:29:39.318950 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:29:39.318924    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf559800f_587f_4681_8e8c_5694a16cff06.slice/crio-e8e46c27855b810d8767ae03c15f701e7257110e12f5af93fcac4066a351ab16 WatchSource:0}: Error finding container e8e46c27855b810d8767ae03c15f701e7257110e12f5af93fcac4066a351ab16: Status 404 returned error can't find the container with id e8e46c27855b810d8767ae03c15f701e7257110e12f5af93fcac4066a351ab16
Oct 30 13:29:39.322014 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:39.321974    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:29:39.322103 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:39.322051    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:29:39.322103 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:39.322090    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:29:39.826425 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:39.826383    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s"]
Oct 30 13:29:39.880149 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:39.880105    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:40.160593 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:40.160491    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd" event={"ID":"f559800f-587f-4681-8e8c-5694a16cff06","Type":"ContainerStarted","Data":"b0d00d6817f6ed909780497e2a109113004a2963504083cb982d9268a9040d99"}
Oct 30 13:29:40.160593 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:40.160559    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd" event={"ID":"f559800f-587f-4681-8e8c-5694a16cff06","Type":"ContainerStarted","Data":"e8e46c27855b810d8767ae03c15f701e7257110e12f5af93fcac4066a351ab16"}
Oct 30 13:29:40.208904 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:40.208835    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd" podStartSLOduration=2.208815675 podStartE2EDuration="2.208815675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:40.205407597 +0000 UTC m=+2675.252541549" watchObservedRunningTime="2026-10-30 13:29:40.208815675 +0000 UTC m=+2675.255949627"
Oct 30 13:29:40.257568 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:40.257530    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:40.983866 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:40.983827    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:29:40.985847 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:40.985819    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 30 13:29:40.985969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:40.985872    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd" podUID="f559800f-587f-4681-8e8c-5694a16cff06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 30 13:29:41.384783 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:41.384747    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"]
Oct 30 13:29:41.389825 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:41.389802    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:29:41.405462 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:41.405435    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-3e8ee4-openshift-default-dockercfg-hpcvm\""
Oct 30 13:29:41.444888 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:41.444854    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"]
Oct 30 13:29:41.456669 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:41.456620    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-credential-socket\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:29:41.456873 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:41.456685    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f44f3385-bd49-41d7-9add-718b415008f2-istio-podinfo\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:29:41.456873 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:41.456813    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z24b\" (UniqueName: \"kubernetes.io/projected/f44f3385-bd49-41d7-9add-718b415008f2-kube-api-access-5z24b\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:29:41.457003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:41.456899    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-workload-socket\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:29:41.457003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:41.456948    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-workload-certs\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:29:41.457120 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:41.457006    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f44f3385-bd49-41d7-9add-718b415008f2-istio-token\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:29:41.457120 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:41.457078    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-istio-envoy\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:29:41.457251 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:41.457116    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f44f3385-bd49-41d7-9add-718b415008f2-istiod-ca-cert\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:29:41.457251 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:41.457158    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-istio-data\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:29:41.558691 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:41.558654    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-istio-envoy\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:29:41.558878 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:41.558701    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f44f3385-bd49-41d7-9add-718b415008f2-istiod-ca-cert\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:29:41.558878 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:41.558733    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-istio-data\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:29:41.558878 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:41.558795    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-credential-socket\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:29:41.559034 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:41.558910    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f44f3385-bd49-41d7-9add-718b415008f2-istio-podinfo\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:29:41.559097 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:41.559042    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z24b\" (UniqueName: \"kubernetes.io/projected/f44f3385-bd49-41d7-9add-718b415008f2-kube-api-access-5z24b\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:29:41.559154 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:41.559134    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-workload-socket\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:29:41.559210 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:41.559186    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-workload-certs\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:29:41.559329 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:41.559269    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-credential-socket\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:29:41.559329 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:41.559273    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-istio-data\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:29:41.559329 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:41.559291    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f44f3385-bd49-41d7-9add-718b415008f2-istio-token\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:29:41.559547 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:41.559525    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f44f3385-bd49-41d7-9add-718b415008f2-istiod-ca-cert\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:29:41.559547 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:41.559540    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-workload-certs\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:29:41.559663 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:41.559585    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-workload-socket\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:29:41.561173 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:41.561153    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-istio-envoy\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:29:41.561440 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:41.561424    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f44f3385-bd49-41d7-9add-718b415008f2-istio-podinfo\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:29:41.582930 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:41.582903    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f44f3385-bd49-41d7-9add-718b415008f2-istio-token\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:29:41.583150 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:41.583126    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z24b\" (UniqueName: \"kubernetes.io/projected/f44f3385-bd49-41d7-9add-718b415008f2-kube-api-access-5z24b\") pod \"e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") " pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:29:41.705271 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:41.705158    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:29:41.984401 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:41.984308    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 30 13:29:41.984401 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:41.984369    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd" podUID="f559800f-587f-4681-8e8c-5694a16cff06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 30 13:29:42.078640 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:42.078603    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"]
Oct 30 13:29:42.094016 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:29:42.093981    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf44f3385_bd49_41d7_9add_718b415008f2.slice/crio-929a30ee4b707616f8bd7ab501442973e855518173c49a90a436415afafd72dd WatchSource:0}: Error finding container 929a30ee4b707616f8bd7ab501442973e855518173c49a90a436415afafd72dd: Status 404 returned error can't find the container with id 929a30ee4b707616f8bd7ab501442973e855518173c49a90a436415afafd72dd
Oct 30 13:29:42.097367 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:42.097329    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:29:42.097471 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:42.097401    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:29:42.097471 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:42.097432    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:29:42.178914 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:42.178873    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf" event={"ID":"f44f3385-bd49-41d7-9add-718b415008f2","Type":"ContainerStarted","Data":"929a30ee4b707616f8bd7ab501442973e855518173c49a90a436415afafd72dd"}
Oct 30 13:29:42.268914 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:42.268166    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:42.983474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:42.983425    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 30 13:29:42.983664 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:42.983583    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd" podUID="f559800f-587f-4681-8e8c-5694a16cff06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 30 13:29:43.188487 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:43.188442    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf" event={"ID":"f44f3385-bd49-41d7-9add-718b415008f2","Type":"ContainerStarted","Data":"1da3ca37de71d501a467fe0530785de9ae1d35ea57150be91997d7199a83df0c"}
Oct 30 13:29:43.272949 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:43.272830    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf" podStartSLOduration=2.27281494 podStartE2EDuration="2.27281494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:43.269252204 +0000 UTC m=+2678.316386153" watchObservedRunningTime="2026-10-30 13:29:43.27281494 +0000 UTC m=+2678.319948894"
Oct 30 13:29:43.705791 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:43.705743    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:29:43.707527 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:43.707503    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Oct 30 13:29:43.707607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:43.707566    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf" podUID="f44f3385-bd49-41d7-9add-718b415008f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Oct 30 13:29:43.984612 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:43.984522    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 30 13:29:43.984612 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:43.984593    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd" podUID="f559800f-587f-4681-8e8c-5694a16cff06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 30 13:29:44.163728 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:44.163676    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:44.705731 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:44.705694    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Oct 30 13:29:44.706145 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:44.705760    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf" podUID="f44f3385-bd49-41d7-9add-718b415008f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Oct 30 13:29:44.983557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:44.983467    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 30 13:29:44.983557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:44.983516    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd" podUID="f559800f-587f-4681-8e8c-5694a16cff06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 30 13:29:45.706507 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:45.706468    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Oct 30 13:29:45.706939 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:45.706541    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf" podUID="f44f3385-bd49-41d7-9add-718b415008f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Oct 30 13:29:45.843979 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:45.843935    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:45.998301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:45.997363    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 30 13:29:45.998301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:45.997432    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd" podUID="f559800f-587f-4681-8e8c-5694a16cff06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 30 13:29:46.189557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:46.189520    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:46.706271 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:46.706214    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Oct 30 13:29:46.706480 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:46.706285    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf" podUID="f44f3385-bd49-41d7-9add-718b415008f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Oct 30 13:29:46.988946 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:46.988857    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:29:46.989442 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:46.989358    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:29:46.990014 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:46.989996    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:29:47.705918 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:47.705876    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Oct 30 13:29:47.706110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:47.705937    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf" podUID="f44f3385-bd49-41d7-9add-718b415008f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Oct 30 13:29:48.134359 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:48.134319    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:48.706409 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:48.706361    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Oct 30 13:29:48.706639 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:48.706438    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf" podUID="f44f3385-bd49-41d7-9add-718b415008f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Oct 30 13:29:49.710992 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:49.710958    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:29:49.711505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:49.711314    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:29:49.712004 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:49.711985    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:29:49.887109 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:49.887074    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:51.665077 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:51.665041    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:52.029785 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:52.029700    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2366f66d-znpvm"]
Oct 30 13:29:52.041510 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:52.041480    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2366f66d-znpvm"]
Oct 30 13:29:53.088907 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:53.088871    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f" path="/var/lib/kubelet/pods/2b26e4dc-4156-49e8-aa8b-dbfab9b4ec6f/volumes"
Oct 30 13:29:53.589830 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:53.589796    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:54.857161 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:54.857121    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s" podUID="3f87a8dd-dd6e-4e00-942d-cfe614d28e6f" containerName="main" containerID="cri-o://2359691669bd197d08cc073f232481502857cd8c20251d498caadf3e2a98edae" gracePeriod=45
Oct 30 13:29:55.149148 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:55.149119    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s"
Oct 30 13:29:55.265667 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:55.265630    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-tmp-dir\") pod \"3f87a8dd-dd6e-4e00-942d-cfe614d28e6f\" (UID: \"3f87a8dd-dd6e-4e00-942d-cfe614d28e6f\") "
Oct 30 13:29:55.265856 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:55.265729    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-tls-certs\") pod \"3f87a8dd-dd6e-4e00-942d-cfe614d28e6f\" (UID: \"3f87a8dd-dd6e-4e00-942d-cfe614d28e6f\") "
Oct 30 13:29:55.265856 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:55.265828    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-model-cache\") pod \"3f87a8dd-dd6e-4e00-942d-cfe614d28e6f\" (UID: \"3f87a8dd-dd6e-4e00-942d-cfe614d28e6f\") "
Oct 30 13:29:55.265969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:55.265868    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-home\" (UniqueName: \"kubernetes.io/empty-dir/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-home\") pod \"3f87a8dd-dd6e-4e00-942d-cfe614d28e6f\" (UID: \"3f87a8dd-dd6e-4e00-942d-cfe614d28e6f\") "
Oct 30 13:29:55.265969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:55.265893    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-dshm\") pod \"3f87a8dd-dd6e-4e00-942d-cfe614d28e6f\" (UID: \"3f87a8dd-dd6e-4e00-942d-cfe614d28e6f\") "
Oct 30 13:29:55.265969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:55.265913    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-tmp-dir" pod "3f87a8dd-dd6e-4e00-942d-cfe614d28e6f" (UID: "3f87a8dd-dd6e-4e00-942d-cfe614d28e6f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:55.265969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:55.265928    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-kube-api-access-6pzfq\" (UniqueName: \"kubernetes.io/projected/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-kube-api-access-6pzfq\") pod \"3f87a8dd-dd6e-4e00-942d-cfe614d28e6f\" (UID: \"3f87a8dd-dd6e-4e00-942d-cfe614d28e6f\") "
Oct 30 13:29:55.266195 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:55.266114    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-model-cache" pod "3f87a8dd-dd6e-4e00-942d-cfe614d28e6f" (UID: "3f87a8dd-dd6e-4e00-942d-cfe614d28e6f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:55.266195 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:55.266149    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-home" pod "3f87a8dd-dd6e-4e00-942d-cfe614d28e6f" (UID: "3f87a8dd-dd6e-4e00-942d-cfe614d28e6f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:55.266620 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:55.266599    4544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-model-cache\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:29:55.266709 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:55.266625    4544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-home\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:29:55.266709 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:55.266638    4544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-tmp-dir\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:29:55.268141 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:55.268118    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-tls-certs" pod "3f87a8dd-dd6e-4e00-942d-cfe614d28e6f" (UID: "3f87a8dd-dd6e-4e00-942d-cfe614d28e6f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:29:55.268244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:55.268138    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-dshm" pod "3f87a8dd-dd6e-4e00-942d-cfe614d28e6f" (UID: "3f87a8dd-dd6e-4e00-942d-cfe614d28e6f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:55.268307 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:55.268265    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-kube-api-access-6pzfq" pod "3f87a8dd-dd6e-4e00-942d-cfe614d28e6f" (UID: "3f87a8dd-dd6e-4e00-942d-cfe614d28e6f"). InnerVolumeSpecName "kube-api-access-6pzfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:55.367944 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:55.367849    4544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-tls-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:29:55.367944 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:55.367886    4544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-dshm\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:29:55.367944 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:55.367900    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pzfq\" (UniqueName: \"kubernetes.io/projected/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f-kube-api-access-6pzfq\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:29:55.374776 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:55.374736    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:55.451019 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:55.450973    4544 generic.go:356] "Generic (PLEG): container finished" podID="3f87a8dd-dd6e-4e00-942d-cfe614d28e6f" containerID="2359691669bd197d08cc073f232481502857cd8c20251d498caadf3e2a98edae" exitCode=0
Oct 30 13:29:55.451273 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:55.451033    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s"
Oct 30 13:29:55.451273 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:55.451075    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s" event={"ID":"3f87a8dd-dd6e-4e00-942d-cfe614d28e6f","Type":"ContainerDied","Data":"2359691669bd197d08cc073f232481502857cd8c20251d498caadf3e2a98edae"}
Oct 30 13:29:55.451273 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:55.451113    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s" event={"ID":"3f87a8dd-dd6e-4e00-942d-cfe614d28e6f","Type":"ContainerDied","Data":"0aa64d90973d76bf10e3dbcc3eb3201e62f4f78ebad6073779b45bcccdceed25"}
Oct 30 13:29:55.451273 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:55.451136    4544 scope.go:122] "RemoveContainer" containerID="2359691669bd197d08cc073f232481502857cd8c20251d498caadf3e2a98edae"
Oct 30 13:29:55.466405 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:55.466385    4544 scope.go:122] "RemoveContainer" containerID="2359691669bd197d08cc073f232481502857cd8c20251d498caadf3e2a98edae"
Oct 30 13:29:55.466779 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:29:55.466752    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2359691669bd197d08cc073f232481502857cd8c20251d498caadf3e2a98edae\": container with ID starting with 2359691669bd197d08cc073f232481502857cd8c20251d498caadf3e2a98edae not found: ID does not exist" containerID="2359691669bd197d08cc073f232481502857cd8c20251d498caadf3e2a98edae"
Oct 30 13:29:55.466862 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:55.466791    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2359691669bd197d08cc073f232481502857cd8c20251d498caadf3e2a98edae"} err="failed to get container status \"2359691669bd197d08cc073f232481502857cd8c20251d498caadf3e2a98edae\": rpc error: code = NotFound desc = could not find container \"2359691669bd197d08cc073f232481502857cd8c20251d498caadf3e2a98edae\": container with ID starting with 2359691669bd197d08cc073f232481502857cd8c20251d498caadf3e2a98edae not found: ID does not exist"
Oct 30 13:29:55.505606 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:55.505581    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s"]
Oct 30 13:29:55.509751 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:55.509714    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-0b2c92/auth-test-model-0b2c92-kserve-9855b8f97-lqt6s"]
Oct 30 13:29:56.965490 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:56.965450    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd"]
Oct 30 13:29:57.019660 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:57.019624    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:57.128151 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:57.128107    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f87a8dd-dd6e-4e00-942d-cfe614d28e6f" path="/var/lib/kubelet/pods/3f87a8dd-dd6e-4e00-942d-cfe614d28e6f/volumes"
Oct 30 13:29:57.160177 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:57.158094    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:57.384699 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:57.384662    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:57.687927 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:57.687842    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:58.372885 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.372854    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"]
Oct 30 13:29:58.373788 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.373766    4544 cpu_manager.go:414] "containerMap: removing container" podUID="3f87a8dd-dd6e-4e00-942d-cfe614d28e6f" containerName="main"
Oct 30 13:29:58.373788 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.373787    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f87a8dd-dd6e-4e00-942d-cfe614d28e6f" containerName="main"
Oct 30 13:29:58.373931 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.373922    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f87a8dd-dd6e-4e00-942d-cfe614d28e6f" containerName="main"
Oct 30 13:29:58.379967 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.379888    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:29:58.395812 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.395789    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-a4f385-openshift-default-dockercfg-fsnt7\""
Oct 30 13:29:58.416661 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.416632    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"]
Oct 30 13:29:58.440701 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.440573    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:58.568256 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.562498    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-workload-certs\") pod \"e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") " pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:29:58.572426 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.572387    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68fa1f90-25f8-442f-96ff-12e6a716db84-istiod-ca-cert\") pod \"e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") " pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:29:58.572769 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.572723    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68fa1f90-25f8-442f-96ff-12e6a716db84-istio-token\") pod \"e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") " pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:29:58.572923 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.572835    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nnlx\" (UniqueName: \"kubernetes.io/projected/68fa1f90-25f8-442f-96ff-12e6a716db84-kube-api-access-5nnlx\") pod \"e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") " pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:29:58.573002 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.572927    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-workload-socket\") pod \"e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") " pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:29:58.573236 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.573081    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68fa1f90-25f8-442f-96ff-12e6a716db84-istio-podinfo\") pod \"e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") " pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:29:58.573236 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.573113    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-istio-data\") pod \"e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") " pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:29:58.573236 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.573159    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-istio-envoy\") pod \"e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") " pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:29:58.573236 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.573210    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-credential-socket\") pod \"e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") " pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:29:58.602718 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.602603    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:58.676085 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.676049    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68fa1f90-25f8-442f-96ff-12e6a716db84-istio-podinfo\") pod \"e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") " pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:29:58.676085 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.676087    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-istio-data\") pod \"e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") " pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:29:58.676452 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.676213    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-istio-envoy\") pod \"e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") " pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:29:58.676452 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.676310    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-credential-socket\") pod \"e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") " pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:29:58.676452 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.676414    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-workload-certs\") pod \"e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") " pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:29:58.676452 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.676436    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68fa1f90-25f8-442f-96ff-12e6a716db84-istiod-ca-cert\") pod \"e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") " pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:29:58.676694 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.676478    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68fa1f90-25f8-442f-96ff-12e6a716db84-istio-token\") pod \"e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") " pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:29:58.676694 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.676579    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nnlx\" (UniqueName: \"kubernetes.io/projected/68fa1f90-25f8-442f-96ff-12e6a716db84-kube-api-access-5nnlx\") pod \"e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") " pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:29:58.676694 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.676669    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-workload-socket\") pod \"e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") " pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:29:58.676860 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.676699    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-istio-data\") pod \"e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") " pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:29:58.676860 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.676760    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-credential-socket\") pod \"e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") " pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:29:58.677132 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.677101    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-workload-certs\") pod \"e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") " pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:29:58.677274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.677183    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-workload-socket\") pod \"e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") " pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:29:58.677360 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.677300    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68fa1f90-25f8-442f-96ff-12e6a716db84-istiod-ca-cert\") pod \"e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") " pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:29:58.678609 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.678583    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-istio-envoy\") pod \"e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") " pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:29:58.678854 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.678838    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68fa1f90-25f8-442f-96ff-12e6a716db84-istio-podinfo\") pod \"e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") " pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:29:58.699784 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.699728    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68fa1f90-25f8-442f-96ff-12e6a716db84-istio-token\") pod \"e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") " pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:29:58.705165 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:58.705140    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nnlx\" (UniqueName: \"kubernetes.io/projected/68fa1f90-25f8-442f-96ff-12e6a716db84-kube-api-access-5nnlx\") pod \"e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") " pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:29:59.002108 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:59.002019    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:29:59.008679 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:29:59.008648    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:29:59.321445 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:59.321347    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:29:59.462729 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:59.462697    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"]
Oct 30 13:29:59.464869 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:29:59.464838    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68fa1f90_25f8_442f_96ff_12e6a716db84.slice/crio-39463f8324d3e8f456aa4f1046b0a7619495d383011e3148539c6826e54458cb WatchSource:0}: Error finding container 39463f8324d3e8f456aa4f1046b0a7619495d383011e3148539c6826e54458cb: Status 404 returned error can't find the container with id 39463f8324d3e8f456aa4f1046b0a7619495d383011e3148539c6826e54458cb
Oct 30 13:29:59.467813 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:59.467778    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:29:59.467897 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:59.467861    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:29:59.467897 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:59.467890    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:29:59.491571 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:29:59.491543    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt" event={"ID":"68fa1f90-25f8-442f-96ff-12e6a716db84","Type":"ContainerStarted","Data":"39463f8324d3e8f456aa4f1046b0a7619495d383011e3148539c6826e54458cb"}
Oct 30 13:30:00.174252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.173624    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk"]
Oct 30 13:30:00.181246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.180410    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk"
Oct 30 13:30:00.192939 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.192902    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427"]
Oct 30 13:30:00.198446 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.198415    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427"
Oct 30 13:30:00.213516 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.213485    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t"]
Oct 30 13:30:00.219473 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.219446    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t"
Oct 30 13:30:00.228433 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.228412    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:30:00.401799 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.401473    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00b65da6-a93a-4b46-8a6f-912049acda4e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk\" (UID: \"00b65da6-a93a-4b46-8a6f-912049acda4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk"
Oct 30 13:30:00.401799 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.401529    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds7rl\" (UniqueName: \"kubernetes.io/projected/00b65da6-a93a-4b46-8a6f-912049acda4e-kube-api-access-ds7rl\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk\" (UID: \"00b65da6-a93a-4b46-8a6f-912049acda4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk"
Oct 30 13:30:00.401799 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.401561    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c9fb4e3-399f-4870-943c-8744296adab5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427\" (UID: \"7c9fb4e3-399f-4870-943c-8744296adab5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427"
Oct 30 13:30:00.401799 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.401623    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0625ba18-6017-481b-9bf2-983358ae2123-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t\" (UID: \"0625ba18-6017-481b-9bf2-983358ae2123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t"
Oct 30 13:30:00.401799 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.401677    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75xwr\" (UniqueName: \"kubernetes.io/projected/0625ba18-6017-481b-9bf2-983358ae2123-kube-api-access-75xwr\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t\" (UID: \"0625ba18-6017-481b-9bf2-983358ae2123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t"
Oct 30 13:30:00.401799 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.401738    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ngb8\" (UniqueName: \"kubernetes.io/projected/7c9fb4e3-399f-4870-943c-8744296adab5-kube-api-access-6ngb8\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427\" (UID: \"7c9fb4e3-399f-4870-943c-8744296adab5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427"
Oct 30 13:30:00.406353 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.406309    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk"]
Oct 30 13:30:00.416369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.416339    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k"]
Oct 30 13:30:00.475354 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.455428    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427"]
Oct 30 13:30:00.477015 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.476985    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t"]
Oct 30 13:30:00.477206 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.477190    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf"]
Oct 30 13:30:00.483894 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.463341    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k"
Oct 30 13:30:00.508783 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.505568    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00b65da6-a93a-4b46-8a6f-912049acda4e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk\" (UID: \"00b65da6-a93a-4b46-8a6f-912049acda4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk"
Oct 30 13:30:00.508783 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.505633    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds7rl\" (UniqueName: \"kubernetes.io/projected/00b65da6-a93a-4b46-8a6f-912049acda4e-kube-api-access-ds7rl\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk\" (UID: \"00b65da6-a93a-4b46-8a6f-912049acda4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk"
Oct 30 13:30:00.508783 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.505668    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c9fb4e3-399f-4870-943c-8744296adab5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427\" (UID: \"7c9fb4e3-399f-4870-943c-8744296adab5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427"
Oct 30 13:30:00.508783 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.505739    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0625ba18-6017-481b-9bf2-983358ae2123-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t\" (UID: \"0625ba18-6017-481b-9bf2-983358ae2123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t"
Oct 30 13:30:00.508783 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.505798    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75xwr\" (UniqueName: \"kubernetes.io/projected/0625ba18-6017-481b-9bf2-983358ae2123-kube-api-access-75xwr\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t\" (UID: \"0625ba18-6017-481b-9bf2-983358ae2123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t"
Oct 30 13:30:00.508783 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.505857    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ngb8\" (UniqueName: \"kubernetes.io/projected/7c9fb4e3-399f-4870-943c-8744296adab5-kube-api-access-6ngb8\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427\" (UID: \"7c9fb4e3-399f-4870-943c-8744296adab5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427"
Oct 30 13:30:00.508783 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.507638    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00b65da6-a93a-4b46-8a6f-912049acda4e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk\" (UID: \"00b65da6-a93a-4b46-8a6f-912049acda4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk"
Oct 30 13:30:00.566078 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.522571    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0625ba18-6017-481b-9bf2-983358ae2123-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t\" (UID: \"0625ba18-6017-481b-9bf2-983358ae2123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t"
Oct 30 13:30:00.592245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.538972    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ngb8\" (UniqueName: \"kubernetes.io/projected/7c9fb4e3-399f-4870-943c-8744296adab5-kube-api-access-6ngb8\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427\" (UID: \"7c9fb4e3-399f-4870-943c-8744296adab5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427"
Oct 30 13:30:00.592513 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.548348    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c9fb4e3-399f-4870-943c-8744296adab5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427\" (UID: \"7c9fb4e3-399f-4870-943c-8744296adab5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427"
Oct 30 13:30:00.624247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.607514    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d84pc\" (UniqueName: \"kubernetes.io/projected/dcbe2566-fac2-4912-9f80-9e3dbb313226-kube-api-access-d84pc\") pod \"maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k\" (UID: \"dcbe2566-fac2-4912-9f80-9e3dbb313226\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k"
Oct 30 13:30:00.624247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.607615    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbe2566-fac2-4912-9f80-9e3dbb313226-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k\" (UID: \"dcbe2566-fac2-4912-9f80-9e3dbb313226\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k"
Oct 30 13:30:00.658678 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.658648    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k"]
Oct 30 13:30:00.658897 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.658883    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c"]
Oct 30 13:30:00.662426 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.662401    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf"
Oct 30 13:30:00.728662 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.728571    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d84pc\" (UniqueName: \"kubernetes.io/projected/dcbe2566-fac2-4912-9f80-9e3dbb313226-kube-api-access-d84pc\") pod \"maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k\" (UID: \"dcbe2566-fac2-4912-9f80-9e3dbb313226\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k"
Oct 30 13:30:00.728969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.728943    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbe2566-fac2-4912-9f80-9e3dbb313226-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k\" (UID: \"dcbe2566-fac2-4912-9f80-9e3dbb313226\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k"
Oct 30 13:30:00.811260 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.811207    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbe2566-fac2-4912-9f80-9e3dbb313226-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k\" (UID: \"dcbe2566-fac2-4912-9f80-9e3dbb313226\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k"
Oct 30 13:30:00.861265 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.813336    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75xwr\" (UniqueName: \"kubernetes.io/projected/0625ba18-6017-481b-9bf2-983358ae2123-kube-api-access-75xwr\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t\" (UID: \"0625ba18-6017-481b-9bf2-983358ae2123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t"
Oct 30 13:30:00.903928 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.862890    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1a04d8b-24e3-4321-b21c-4e2d698ea5d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf\" (UID: \"e1a04d8b-24e3-4321-b21c-4e2d698ea5d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf"
Oct 30 13:30:00.903928 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.862987    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ms76\" (UniqueName: \"kubernetes.io/projected/e1a04d8b-24e3-4321-b21c-4e2d698ea5d2-kube-api-access-7ms76\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf\" (UID: \"e1a04d8b-24e3-4321-b21c-4e2d698ea5d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf"
Oct 30 13:30:00.903928 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.900181    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t"
Oct 30 13:30:00.905825 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.905695    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427"
Oct 30 13:30:00.906769 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.906739    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d84pc\" (UniqueName: \"kubernetes.io/projected/dcbe2566-fac2-4912-9f80-9e3dbb313226-kube-api-access-d84pc\") pod \"maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k\" (UID: \"dcbe2566-fac2-4912-9f80-9e3dbb313226\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k"
Oct 30 13:30:00.964525 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.964482    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k"
Oct 30 13:30:00.967917 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.967883    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5"]
Oct 30 13:30:00.971004 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.970262    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c"
Oct 30 13:30:00.972344 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.972307    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1a04d8b-24e3-4321-b21c-4e2d698ea5d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf\" (UID: \"e1a04d8b-24e3-4321-b21c-4e2d698ea5d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf"
Oct 30 13:30:00.972463 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.972413    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ms76\" (UniqueName: \"kubernetes.io/projected/e1a04d8b-24e3-4321-b21c-4e2d698ea5d2-kube-api-access-7ms76\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf\" (UID: \"e1a04d8b-24e3-4321-b21c-4e2d698ea5d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf"
Oct 30 13:30:00.975824 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:00.975800    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1a04d8b-24e3-4321-b21c-4e2d698ea5d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf\" (UID: \"e1a04d8b-24e3-4321-b21c-4e2d698ea5d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf"
Oct 30 13:30:01.005964 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:01.005923    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt" event={"ID":"68fa1f90-25f8-442f-96ff-12e6a716db84","Type":"ContainerStarted","Data":"1b5b1864e485d3ba2317e500e7781d058ddcd9712fb6a5eaaf154303bec16dd3"}
Oct 30 13:30:01.006246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:01.006205    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c"]
Oct 30 13:30:01.006589 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:01.006568    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5"
Oct 30 13:30:01.011281 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:01.010602    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:30:01.051345 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:01.046630    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf"]
Oct 30 13:30:01.051345 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:01.046712    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5"]
Oct 30 13:30:01.051345 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:01.047211    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:30:01.058255 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:01.051882    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ms76\" (UniqueName: \"kubernetes.io/projected/e1a04d8b-24e3-4321-b21c-4e2d698ea5d2-kube-api-access-7ms76\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf\" (UID: \"e1a04d8b-24e3-4321-b21c-4e2d698ea5d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf"
Oct 30 13:30:01.128244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:01.122725    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdm9c\" (UniqueName: \"kubernetes.io/projected/ebce006c-8200-434b-99e8-5934eadad9a3-kube-api-access-gdm9c\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5\" (UID: \"ebce006c-8200-434b-99e8-5934eadad9a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5"
Oct 30 13:30:01.128244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:01.122843    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6246b4d-a9f8-495d-8f1d-8f8f57fee223-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c\" (UID: \"d6246b4d-a9f8-495d-8f1d-8f8f57fee223\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c"
Oct 30 13:30:01.128244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:01.122878    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebce006c-8200-434b-99e8-5934eadad9a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5\" (UID: \"ebce006c-8200-434b-99e8-5934eadad9a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5"
Oct 30 13:30:01.128244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:01.122920    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn7xt\" (UniqueName: \"kubernetes.io/projected/d6246b4d-a9f8-495d-8f1d-8f8f57fee223-kube-api-access-qn7xt\") pod \"maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c\" (UID: \"d6246b4d-a9f8-495d-8f1d-8f8f57fee223\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c"
Oct 30 13:30:01.214100 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:01.214030    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds7rl\" (UniqueName: \"kubernetes.io/projected/00b65da6-a93a-4b46-8a6f-912049acda4e-kube-api-access-ds7rl\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk\" (UID: \"00b65da6-a93a-4b46-8a6f-912049acda4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk"
Oct 30 13:30:01.230280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:01.229305    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn7xt\" (UniqueName: \"kubernetes.io/projected/d6246b4d-a9f8-495d-8f1d-8f8f57fee223-kube-api-access-qn7xt\") pod \"maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c\" (UID: \"d6246b4d-a9f8-495d-8f1d-8f8f57fee223\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c"
Oct 30 13:30:01.230280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:01.229664    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdm9c\" (UniqueName: \"kubernetes.io/projected/ebce006c-8200-434b-99e8-5934eadad9a3-kube-api-access-gdm9c\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5\" (UID: \"ebce006c-8200-434b-99e8-5934eadad9a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5"
Oct 30 13:30:01.230280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:01.229764    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6246b4d-a9f8-495d-8f1d-8f8f57fee223-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c\" (UID: \"d6246b4d-a9f8-495d-8f1d-8f8f57fee223\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c"
Oct 30 13:30:01.230280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:01.229796    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebce006c-8200-434b-99e8-5934eadad9a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5\" (UID: \"ebce006c-8200-434b-99e8-5934eadad9a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5"
Oct 30 13:30:01.231346 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:01.230775    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebce006c-8200-434b-99e8-5934eadad9a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5\" (UID: \"ebce006c-8200-434b-99e8-5934eadad9a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5"
Oct 30 13:30:01.273320 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:01.265125    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6246b4d-a9f8-495d-8f1d-8f8f57fee223-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c\" (UID: \"d6246b4d-a9f8-495d-8f1d-8f8f57fee223\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c"
Oct 30 13:30:01.282698 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:01.282564    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf"
Oct 30 13:30:01.319623 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:01.319583    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn7xt\" (UniqueName: \"kubernetes.io/projected/d6246b4d-a9f8-495d-8f1d-8f8f57fee223-kube-api-access-qn7xt\") pod \"maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c\" (UID: \"d6246b4d-a9f8-495d-8f1d-8f8f57fee223\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c"
Oct 30 13:30:01.338317 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:01.337003    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdm9c\" (UniqueName: \"kubernetes.io/projected/ebce006c-8200-434b-99e8-5934eadad9a3-kube-api-access-gdm9c\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5\" (UID: \"ebce006c-8200-434b-99e8-5934eadad9a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5"
Oct 30 13:30:01.359733 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:01.359656    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt" podStartSLOduration=3.359627552 podStartE2EDuration="3.359627552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:00.727314747 +0000 UTC m=+2695.774448692" watchObservedRunningTime="2026-10-30 13:30:01.359627552 +0000 UTC m=+2696.406761506"
Oct 30 13:30:01.424525 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:01.423552    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk"
Oct 30 13:30:01.856301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:01.854501    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5"
Oct 30 13:30:01.873088 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:01.860015    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:01.891256 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:01.886472    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c"
Oct 30 13:30:01.902008 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:01.900483    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:30:01.902008 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:01.901917    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:30:02.269590 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:02.269554    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t"]
Oct 30 13:30:02.438135 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:02.438092    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427"]
Oct 30 13:30:02.442891 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:02.442857    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k"]
Oct 30 13:30:02.472770 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:02.472737    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf"]
Oct 30 13:30:02.490791 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:02.490767    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5"]
Oct 30 13:30:02.516099 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:30:02.516062    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c9fb4e3_399f_4870_943c_8744296adab5.slice/crio-d65602685d9964b6d2e61dd0fa3b67de60118a6db57e18faa4bbe785f5f6d99a WatchSource:0}: Error finding container d65602685d9964b6d2e61dd0fa3b67de60118a6db57e18faa4bbe785f5f6d99a: Status 404 returned error can't find the container with id d65602685d9964b6d2e61dd0fa3b67de60118a6db57e18faa4bbe785f5f6d99a
Oct 30 13:30:02.520897 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:30:02.519597    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcbe2566_fac2_4912_9f80_9e3dbb313226.slice/crio-34289c39f411fb402aa60e662537df45905a721f2ccb92ae73ef064bcab5cd00 WatchSource:0}: Error finding container 34289c39f411fb402aa60e662537df45905a721f2ccb92ae73ef064bcab5cd00: Status 404 returned error can't find the container with id 34289c39f411fb402aa60e662537df45905a721f2ccb92ae73ef064bcab5cd00
Oct 30 13:30:02.522506 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:02.522377    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c"]
Oct 30 13:30:02.526689 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:30:02.526328    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6246b4d_a9f8_495d_8f1d_8f8f57fee223.slice/crio-7496738145ef41725ade3909d2b2aed0acbc35d53abdacd248a6a25b09fed29b WatchSource:0}: Error finding container 7496738145ef41725ade3909d2b2aed0acbc35d53abdacd248a6a25b09fed29b: Status 404 returned error can't find the container with id 7496738145ef41725ade3909d2b2aed0acbc35d53abdacd248a6a25b09fed29b
Oct 30 13:30:02.598444 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:02.598413    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk"]
Oct 30 13:30:02.600269 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:30:02.600238    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b65da6_a93a_4b46_8a6f_912049acda4e.slice/crio-3c6819c7689dbc3dfce2583afb1cbfee5c1933ec0a7aaa2e3fbba49ce7448fca WatchSource:0}: Error finding container 3c6819c7689dbc3dfce2583afb1cbfee5c1933ec0a7aaa2e3fbba49ce7448fca: Status 404 returned error can't find the container with id 3c6819c7689dbc3dfce2583afb1cbfee5c1933ec0a7aaa2e3fbba49ce7448fca
Oct 30 13:30:02.894168 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:02.894122    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t" event={"ID":"0625ba18-6017-481b-9bf2-983358ae2123","Type":"ContainerStarted","Data":"eff60e5cde7437a1241093498a42730a2cf6a848d1d43b0117d9a215586d40ec"}
Oct 30 13:30:02.895505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:02.895478    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf" event={"ID":"e1a04d8b-24e3-4321-b21c-4e2d698ea5d2","Type":"ContainerStarted","Data":"06974cba6dec89ff790439f423a2c016040b192d62bee2008bfffe32a5849195"}
Oct 30 13:30:02.896733 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:02.896704    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k" event={"ID":"dcbe2566-fac2-4912-9f80-9e3dbb313226","Type":"ContainerStarted","Data":"34289c39f411fb402aa60e662537df45905a721f2ccb92ae73ef064bcab5cd00"}
Oct 30 13:30:02.897952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:02.897922    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c" event={"ID":"d6246b4d-a9f8-495d-8f1d-8f8f57fee223","Type":"ContainerStarted","Data":"7496738145ef41725ade3909d2b2aed0acbc35d53abdacd248a6a25b09fed29b"}
Oct 30 13:30:02.899336 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:02.899301    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5" event={"ID":"ebce006c-8200-434b-99e8-5934eadad9a3","Type":"ContainerStarted","Data":"a809fe36ef98151047b4ecd35469e08488f13369e7f405ae1e6948da5032414a"}
Oct 30 13:30:02.900706 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:02.900683    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk" event={"ID":"00b65da6-a93a-4b46-8a6f-912049acda4e","Type":"ContainerStarted","Data":"3c6819c7689dbc3dfce2583afb1cbfee5c1933ec0a7aaa2e3fbba49ce7448fca"}
Oct 30 13:30:02.901977 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:02.901955    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427" event={"ID":"7c9fb4e3-399f-4870-943c-8744296adab5","Type":"ContainerStarted","Data":"d65602685d9964b6d2e61dd0fa3b67de60118a6db57e18faa4bbe785f5f6d99a"}
Oct 30 13:30:03.035437 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:03.032922    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0557492-99fxd"]
Oct 30 13:30:03.044354 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:03.044322    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0557492-99fxd"]
Oct 30 13:30:03.091287 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:03.091208    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2336220-cd50-48ec-8dfa-18fec98bcd39" path="/var/lib/kubelet/pods/e2336220-cd50-48ec-8dfa-18fec98bcd39/volumes"
Oct 30 13:30:03.349859 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:03.349822    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:03.502175 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:03.502140    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:03.914903 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:03.914875    4544 generic.go:356] "Generic (PLEG): container finished" podID="0625ba18-6017-481b-9bf2-983358ae2123" containerID="9306b0cf18bf67e709b29c3d18885446cc179b3c49f76a31e4db9900692ef765" exitCode=60
Oct 30 13:30:03.915302 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:03.914955    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t" event={"ID":"0625ba18-6017-481b-9bf2-983358ae2123","Type":"ContainerDied","Data":"9306b0cf18bf67e709b29c3d18885446cc179b3c49f76a31e4db9900692ef765"}
Oct 30 13:30:03.915360 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:03.915333    4544 scope.go:122] "RemoveContainer" containerID="9306b0cf18bf67e709b29c3d18885446cc179b3c49f76a31e4db9900692ef765"
Oct 30 13:30:03.916905 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:03.916879    4544 generic.go:356] "Generic (PLEG): container finished" podID="e1a04d8b-24e3-4321-b21c-4e2d698ea5d2" containerID="161df8f94b6644c222eb43f07d9a03c860850e10e0df841f706e0e1c808fbf86" exitCode=60
Oct 30 13:30:03.917034 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:03.916966    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf" event={"ID":"e1a04d8b-24e3-4321-b21c-4e2d698ea5d2","Type":"ContainerDied","Data":"161df8f94b6644c222eb43f07d9a03c860850e10e0df841f706e0e1c808fbf86"}
Oct 30 13:30:03.917478 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:03.917453    4544 scope.go:122] "RemoveContainer" containerID="161df8f94b6644c222eb43f07d9a03c860850e10e0df841f706e0e1c808fbf86"
Oct 30 13:30:03.919325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:03.919302    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k" event={"ID":"dcbe2566-fac2-4912-9f80-9e3dbb313226","Type":"ContainerStarted","Data":"eab47a090845ed9e7950f0d53887c059da89bffa73a7c218aab99aab7819df46"}
Oct 30 13:30:03.922165 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:03.922144    4544 generic.go:356] "Generic (PLEG): container finished" podID="d6246b4d-a9f8-495d-8f1d-8f8f57fee223" containerID="28ac659d6b304ed138b7249966165ac1f75ac50aae20572a2212446629d135a7" exitCode=60
Oct 30 13:30:03.922291 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:03.922233    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c" event={"ID":"d6246b4d-a9f8-495d-8f1d-8f8f57fee223","Type":"ContainerDied","Data":"28ac659d6b304ed138b7249966165ac1f75ac50aae20572a2212446629d135a7"}
Oct 30 13:30:03.922500 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:03.922482    4544 scope.go:122] "RemoveContainer" containerID="28ac659d6b304ed138b7249966165ac1f75ac50aae20572a2212446629d135a7"
Oct 30 13:30:03.924394 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:03.924372    4544 generic.go:356] "Generic (PLEG): container finished" podID="ebce006c-8200-434b-99e8-5934eadad9a3" containerID="ff1671e842fc90bcc1629c62e1bc4aecdb97e78325dd61583a16520b5c26b099" exitCode=60
Oct 30 13:30:03.924545 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:03.924530    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5" event={"ID":"ebce006c-8200-434b-99e8-5934eadad9a3","Type":"ContainerDied","Data":"ff1671e842fc90bcc1629c62e1bc4aecdb97e78325dd61583a16520b5c26b099"}
Oct 30 13:30:03.924826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:03.924806    4544 scope.go:122] "RemoveContainer" containerID="ff1671e842fc90bcc1629c62e1bc4aecdb97e78325dd61583a16520b5c26b099"
Oct 30 13:30:03.927173 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:03.927143    4544 generic.go:356] "Generic (PLEG): container finished" podID="00b65da6-a93a-4b46-8a6f-912049acda4e" containerID="0c3e46318826df61824a62b20fa5b6aa88cc23acc826ccc9d83f64de668c400d" exitCode=60
Oct 30 13:30:03.927309 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:03.927210    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk" event={"ID":"00b65da6-a93a-4b46-8a6f-912049acda4e","Type":"ContainerDied","Data":"0c3e46318826df61824a62b20fa5b6aa88cc23acc826ccc9d83f64de668c400d"}
Oct 30 13:30:03.927563 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:03.927544    4544 scope.go:122] "RemoveContainer" containerID="0c3e46318826df61824a62b20fa5b6aa88cc23acc826ccc9d83f64de668c400d"
Oct 30 13:30:03.931072 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:03.931050    4544 generic.go:356] "Generic (PLEG): container finished" podID="7c9fb4e3-399f-4870-943c-8744296adab5" containerID="6112509616f237696542c652a105d36692bd4776d249fb982b38023dd2388d08" exitCode=60
Oct 30 13:30:03.931204 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:03.931146    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427" event={"ID":"7c9fb4e3-399f-4870-943c-8744296adab5","Type":"ContainerDied","Data":"6112509616f237696542c652a105d36692bd4776d249fb982b38023dd2388d08"}
Oct 30 13:30:03.931431 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:03.931415    4544 scope.go:122] "RemoveContainer" containerID="6112509616f237696542c652a105d36692bd4776d249fb982b38023dd2388d08"
Oct 30 13:30:04.247340 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:04.247303    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:04.458827 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:04.458789    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:04.944154 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:04.944113    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk" event={"ID":"00b65da6-a93a-4b46-8a6f-912049acda4e","Type":"ContainerStarted","Data":"245c44f2f59179e39c035d831cc87762fd3a9d7df2f6b11b3889d8ec3e9f19a8"}
Oct 30 13:30:04.946610 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:04.946568    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427" event={"ID":"7c9fb4e3-399f-4870-943c-8744296adab5","Type":"ContainerStarted","Data":"36158f0bb6b48dd4c92295bdebd85ea90a8caad1b6d4597707f493149b5e8249"}
Oct 30 13:30:04.948751 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:04.948726    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t" event={"ID":"0625ba18-6017-481b-9bf2-983358ae2123","Type":"ContainerStarted","Data":"89653efeca3bf1eadae508666013a754b714795695a21fc7b165ef619462102f"}
Oct 30 13:30:04.950960 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:04.950937    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf" event={"ID":"e1a04d8b-24e3-4321-b21c-4e2d698ea5d2","Type":"ContainerStarted","Data":"51a750784688719f19c1230e0088d86d3b1a96f188c604471780b0617ed6b695"}
Oct 30 13:30:04.952688 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:04.952661    4544 generic.go:356] "Generic (PLEG): container finished" podID="dcbe2566-fac2-4912-9f80-9e3dbb313226" containerID="eab47a090845ed9e7950f0d53887c059da89bffa73a7c218aab99aab7819df46" exitCode=60
Oct 30 13:30:04.952775 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:04.952731    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k" event={"ID":"dcbe2566-fac2-4912-9f80-9e3dbb313226","Type":"ContainerDied","Data":"eab47a090845ed9e7950f0d53887c059da89bffa73a7c218aab99aab7819df46"}
Oct 30 13:30:04.952994 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:04.952974    4544 scope.go:122] "RemoveContainer" containerID="eab47a090845ed9e7950f0d53887c059da89bffa73a7c218aab99aab7819df46"
Oct 30 13:30:04.954914 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:04.954889    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c" event={"ID":"d6246b4d-a9f8-495d-8f1d-8f8f57fee223","Type":"ContainerStarted","Data":"18e0559709bcbb20645349e6c19d6589edfe26586b2c459d70caa551a2d56353"}
Oct 30 13:30:04.967696 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:04.967652    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5" event={"ID":"ebce006c-8200-434b-99e8-5934eadad9a3","Type":"ContainerStarted","Data":"d0648c55aa30dc913c771498e1c4b8cf39811fc95fbf690a8fb006cd0f9964d8"}
Oct 30 13:30:04.998682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:04.998625    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk" podStartSLOduration=4.9986082530000004 podStartE2EDuration="4.998608253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:04.995877472 +0000 UTC m=+2700.043011628" watchObservedRunningTime="2026-10-30 13:30:04.998608253 +0000 UTC m=+2700.045742232"
Oct 30 13:30:04.999152 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:04.999127    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k" podStartSLOduration=4.999118202 podStartE2EDuration="4.999118202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:04.386002094 +0000 UTC m=+2699.433136043" watchObservedRunningTime="2026-10-30 13:30:04.999118202 +0000 UTC m=+2700.046252203"
Oct 30 13:30:05.042001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:05.041948    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c" podStartSLOduration=5.041930486 podStartE2EDuration="5.041930486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:05.036882062 +0000 UTC m=+2700.084016027" watchObservedRunningTime="2026-10-30 13:30:05.041930486 +0000 UTC m=+2700.089064440"
Oct 30 13:30:05.089207 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:05.089149    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5" podStartSLOduration=5.089134641 podStartE2EDuration="5.089134641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:05.085397983 +0000 UTC m=+2700.132531937" watchObservedRunningTime="2026-10-30 13:30:05.089134641 +0000 UTC m=+2700.136268594"
Oct 30 13:30:05.172001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:05.171944    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t" podStartSLOduration=5.171927427 podStartE2EDuration="5.171927427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:05.167599066 +0000 UTC m=+2700.214733020" watchObservedRunningTime="2026-10-30 13:30:05.171927427 +0000 UTC m=+2700.219061380"
Oct 30 13:30:05.216778 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:05.216648    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf" podStartSLOduration=5.216628172 podStartE2EDuration="5.216628172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:05.214010815 +0000 UTC m=+2700.261144796" watchObservedRunningTime="2026-10-30 13:30:05.216628172 +0000 UTC m=+2700.263762125"
Oct 30 13:30:05.218341 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:05.218311    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:05.267254 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:05.267188    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427" podStartSLOduration=5.267173116 podStartE2EDuration="5.267173116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:05.263574517 +0000 UTC m=+2700.310708469" watchObservedRunningTime="2026-10-30 13:30:05.267173116 +0000 UTC m=+2700.314307069"
Oct 30 13:30:05.672382 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:05.672348    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:05.980818 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:05.980784    4544 generic.go:356] "Generic (PLEG): container finished" podID="0625ba18-6017-481b-9bf2-983358ae2123" containerID="89653efeca3bf1eadae508666013a754b714795695a21fc7b165ef619462102f" exitCode=60
Oct 30 13:30:05.981334 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:05.980834    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t" event={"ID":"0625ba18-6017-481b-9bf2-983358ae2123","Type":"ContainerDied","Data":"89653efeca3bf1eadae508666013a754b714795695a21fc7b165ef619462102f"}
Oct 30 13:30:05.981334 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:05.980895    4544 scope.go:122] "RemoveContainer" containerID="9306b0cf18bf67e709b29c3d18885446cc179b3c49f76a31e4db9900692ef765"
Oct 30 13:30:05.981334 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:05.981214    4544 scope.go:122] "RemoveContainer" containerID="89653efeca3bf1eadae508666013a754b714795695a21fc7b165ef619462102f"
Oct 30 13:30:05.981700 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:30:05.981661    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t_odh-ai-gateway-infra(0625ba18-6017-481b-9bf2-983358ae2123)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t" podUID="0625ba18-6017-481b-9bf2-983358ae2123"
Oct 30 13:30:05.982962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:05.982942    4544 generic.go:356] "Generic (PLEG): container finished" podID="e1a04d8b-24e3-4321-b21c-4e2d698ea5d2" containerID="51a750784688719f19c1230e0088d86d3b1a96f188c604471780b0617ed6b695" exitCode=60
Oct 30 13:30:05.983050 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:05.983012    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf" event={"ID":"e1a04d8b-24e3-4321-b21c-4e2d698ea5d2","Type":"ContainerDied","Data":"51a750784688719f19c1230e0088d86d3b1a96f188c604471780b0617ed6b695"}
Oct 30 13:30:05.983334 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:05.983311    4544 scope.go:122] "RemoveContainer" containerID="51a750784688719f19c1230e0088d86d3b1a96f188c604471780b0617ed6b695"
Oct 30 13:30:05.983673 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:30:05.983649    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf_odh-ai-gateway-infra(e1a04d8b-24e3-4321-b21c-4e2d698ea5d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf" podUID="e1a04d8b-24e3-4321-b21c-4e2d698ea5d2"
Oct 30 13:30:05.985064 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:05.985041    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k" event={"ID":"dcbe2566-fac2-4912-9f80-9e3dbb313226","Type":"ContainerStarted","Data":"5c9da85d34150b7e35bc36fd3864ad8b6b5e7e4e9692c8a3094644192a941b4a"}
Oct 30 13:30:05.987361 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:05.987339    4544 generic.go:356] "Generic (PLEG): container finished" podID="d6246b4d-a9f8-495d-8f1d-8f8f57fee223" containerID="18e0559709bcbb20645349e6c19d6589edfe26586b2c459d70caa551a2d56353" exitCode=60
Oct 30 13:30:05.987474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:05.987411    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c" event={"ID":"d6246b4d-a9f8-495d-8f1d-8f8f57fee223","Type":"ContainerDied","Data":"18e0559709bcbb20645349e6c19d6589edfe26586b2c459d70caa551a2d56353"}
Oct 30 13:30:05.987627 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:05.987613    4544 scope.go:122] "RemoveContainer" containerID="18e0559709bcbb20645349e6c19d6589edfe26586b2c459d70caa551a2d56353"
Oct 30 13:30:05.987885 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:30:05.987860    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c_odh-ai-gateway-infra(d6246b4d-a9f8-495d-8f1d-8f8f57fee223)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c" podUID="d6246b4d-a9f8-495d-8f1d-8f8f57fee223"
Oct 30 13:30:05.989573 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:05.989542    4544 generic.go:356] "Generic (PLEG): container finished" podID="ebce006c-8200-434b-99e8-5934eadad9a3" containerID="d0648c55aa30dc913c771498e1c4b8cf39811fc95fbf690a8fb006cd0f9964d8" exitCode=60
Oct 30 13:30:05.989750 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:05.989731    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5" event={"ID":"ebce006c-8200-434b-99e8-5934eadad9a3","Type":"ContainerDied","Data":"d0648c55aa30dc913c771498e1c4b8cf39811fc95fbf690a8fb006cd0f9964d8"}
Oct 30 13:30:05.990002 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:05.989986    4544 scope.go:122] "RemoveContainer" containerID="d0648c55aa30dc913c771498e1c4b8cf39811fc95fbf690a8fb006cd0f9964d8"
Oct 30 13:30:05.990275 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:30:05.990254    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5_odh-ai-gateway-infra(ebce006c-8200-434b-99e8-5934eadad9a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5" podUID="ebce006c-8200-434b-99e8-5934eadad9a3"
Oct 30 13:30:05.991680 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:05.991655    4544 generic.go:356] "Generic (PLEG): container finished" podID="00b65da6-a93a-4b46-8a6f-912049acda4e" containerID="245c44f2f59179e39c035d831cc87762fd3a9d7df2f6b11b3889d8ec3e9f19a8" exitCode=60
Oct 30 13:30:05.991780 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:05.991735    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk" event={"ID":"00b65da6-a93a-4b46-8a6f-912049acda4e","Type":"ContainerDied","Data":"245c44f2f59179e39c035d831cc87762fd3a9d7df2f6b11b3889d8ec3e9f19a8"}
Oct 30 13:30:05.991935 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:05.991918    4544 scope.go:122] "RemoveContainer" containerID="245c44f2f59179e39c035d831cc87762fd3a9d7df2f6b11b3889d8ec3e9f19a8"
Oct 30 13:30:05.992172 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:30:05.992151    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk_odh-ai-gateway-infra(00b65da6-a93a-4b46-8a6f-912049acda4e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk" podUID="00b65da6-a93a-4b46-8a6f-912049acda4e"
Oct 30 13:30:05.993668 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:05.993646    4544 generic.go:356] "Generic (PLEG): container finished" podID="7c9fb4e3-399f-4870-943c-8744296adab5" containerID="36158f0bb6b48dd4c92295bdebd85ea90a8caad1b6d4597707f493149b5e8249" exitCode=60
Oct 30 13:30:05.993788 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:05.993732    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427" event={"ID":"7c9fb4e3-399f-4870-943c-8744296adab5","Type":"ContainerDied","Data":"36158f0bb6b48dd4c92295bdebd85ea90a8caad1b6d4597707f493149b5e8249"}
Oct 30 13:30:05.993912 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:05.993896    4544 scope.go:122] "RemoveContainer" containerID="36158f0bb6b48dd4c92295bdebd85ea90a8caad1b6d4597707f493149b5e8249"
Oct 30 13:30:05.994124 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:30:05.994106    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427_odh-ai-gateway-infra(7c9fb4e3-399f-4870-943c-8744296adab5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427" podUID="7c9fb4e3-399f-4870-943c-8744296adab5"
Oct 30 13:30:05.996434 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:05.996418    4544 scope.go:122] "RemoveContainer" containerID="161df8f94b6644c222eb43f07d9a03c860850e10e0df841f706e0e1c808fbf86"
Oct 30 13:30:06.011688 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:06.011656    4544 scope.go:122] "RemoveContainer" containerID="28ac659d6b304ed138b7249966165ac1f75ac50aae20572a2212446629d135a7"
Oct 30 13:30:06.107321 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:06.107299    4544 scope.go:122] "RemoveContainer" containerID="ff1671e842fc90bcc1629c62e1bc4aecdb97e78325dd61583a16520b5c26b099"
Oct 30 13:30:06.138447 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:06.138409    4544 scope.go:122] "RemoveContainer" containerID="0c3e46318826df61824a62b20fa5b6aa88cc23acc826ccc9d83f64de668c400d"
Oct 30 13:30:06.156042 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:06.155993    4544 scope.go:122] "RemoveContainer" containerID="6112509616f237696542c652a105d36692bd4776d249fb982b38023dd2388d08"
Oct 30 13:30:06.245185 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:06.235939    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:06.588348 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:06.588316    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:06.744517 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:06.744477    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:07.004618 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:07.004514    4544 generic.go:356] "Generic (PLEG): container finished" podID="dcbe2566-fac2-4912-9f80-9e3dbb313226" containerID="5c9da85d34150b7e35bc36fd3864ad8b6b5e7e4e9692c8a3094644192a941b4a" exitCode=60
Oct 30 13:30:07.005203 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:07.004616    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k" event={"ID":"dcbe2566-fac2-4912-9f80-9e3dbb313226","Type":"ContainerDied","Data":"5c9da85d34150b7e35bc36fd3864ad8b6b5e7e4e9692c8a3094644192a941b4a"}
Oct 30 13:30:07.005203 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:07.004727    4544 scope.go:122] "RemoveContainer" containerID="eab47a090845ed9e7950f0d53887c059da89bffa73a7c218aab99aab7819df46"
Oct 30 13:30:07.005461 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:07.005442    4544 scope.go:122] "RemoveContainer" containerID="5c9da85d34150b7e35bc36fd3864ad8b6b5e7e4e9692c8a3094644192a941b4a"
Oct 30 13:30:07.006045 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:30:07.005898    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k_odh-ai-gateway-infra(dcbe2566-fac2-4912-9f80-9e3dbb313226)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k" podUID="dcbe2566-fac2-4912-9f80-9e3dbb313226"
Oct 30 13:30:07.007088 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:07.007068    4544 scope.go:122] "RemoveContainer" containerID="18e0559709bcbb20645349e6c19d6589edfe26586b2c459d70caa551a2d56353"
Oct 30 13:30:07.007404 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:30:07.007382    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c_odh-ai-gateway-infra(d6246b4d-a9f8-495d-8f1d-8f8f57fee223)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c" podUID="d6246b4d-a9f8-495d-8f1d-8f8f57fee223"
Oct 30 13:30:07.008759 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:07.008740    4544 scope.go:122] "RemoveContainer" containerID="d0648c55aa30dc913c771498e1c4b8cf39811fc95fbf690a8fb006cd0f9964d8"
Oct 30 13:30:07.008991 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:30:07.008969    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5_odh-ai-gateway-infra(ebce006c-8200-434b-99e8-5934eadad9a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5" podUID="ebce006c-8200-434b-99e8-5934eadad9a3"
Oct 30 13:30:07.010576 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:07.010558    4544 scope.go:122] "RemoveContainer" containerID="245c44f2f59179e39c035d831cc87762fd3a9d7df2f6b11b3889d8ec3e9f19a8"
Oct 30 13:30:07.010838 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:30:07.010813    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk_odh-ai-gateway-infra(00b65da6-a93a-4b46-8a6f-912049acda4e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk" podUID="00b65da6-a93a-4b46-8a6f-912049acda4e"
Oct 30 13:30:07.012364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:07.012349    4544 scope.go:122] "RemoveContainer" containerID="36158f0bb6b48dd4c92295bdebd85ea90a8caad1b6d4597707f493149b5e8249"
Oct 30 13:30:07.012597 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:30:07.012574    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427_odh-ai-gateway-infra(7c9fb4e3-399f-4870-943c-8744296adab5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427" podUID="7c9fb4e3-399f-4870-943c-8744296adab5"
Oct 30 13:30:07.014411 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:07.014395    4544 scope.go:122] "RemoveContainer" containerID="89653efeca3bf1eadae508666013a754b714795695a21fc7b165ef619462102f"
Oct 30 13:30:07.014642 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:30:07.014618    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t_odh-ai-gateway-infra(0625ba18-6017-481b-9bf2-983358ae2123)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t" podUID="0625ba18-6017-481b-9bf2-983358ae2123"
Oct 30 13:30:07.016420 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:07.016390    4544 scope.go:122] "RemoveContainer" containerID="51a750784688719f19c1230e0088d86d3b1a96f188c604471780b0617ed6b695"
Oct 30 13:30:07.016675 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:30:07.016649    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf_odh-ai-gateway-infra(e1a04d8b-24e3-4321-b21c-4e2d698ea5d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf" podUID="e1a04d8b-24e3-4321-b21c-4e2d698ea5d2"
Oct 30 13:30:07.422485 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:07.422422    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:07.541701 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:07.541663    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w"]
Oct 30 13:30:07.548170 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:07.548130    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w"
Oct 30 13:30:07.569068 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:07.569032    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a4f385\""
Oct 30 13:30:07.569485 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:07.569464    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a4f385-dockercfg-l6fv4\""
Oct 30 13:30:07.571263 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:07.571239    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w"]
Oct 30 13:30:07.591535 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:07.591496    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn"]
Oct 30 13:30:07.598233 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:07.598190    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn"
Oct 30 13:30:07.604783 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:07.604736    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:07.616359 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:07.616139    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swxzr\" (UniqueName: \"kubernetes.io/projected/39386eb4-a783-44ca-b8f0-ecf162582aee-kube-api-access-swxzr\") pod \"payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn\" (UID: \"39386eb4-a783-44ca-b8f0-ecf162582aee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn"
Oct 30 13:30:07.616549 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:07.616510    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c89b7d-ea4e-425b-b35f-9bc061ecffdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w\" (UID: \"25c89b7d-ea4e-425b-b35f-9bc061ecffdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w"
Oct 30 13:30:07.616632 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:07.616574    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39386eb4-a783-44ca-b8f0-ecf162582aee-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn\" (UID: \"39386eb4-a783-44ca-b8f0-ecf162582aee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn"
Oct 30 13:30:07.616702 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:07.616613    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgxf8\" (UniqueName: \"kubernetes.io/projected/25c89b7d-ea4e-425b-b35f-9bc061ecffdb-kube-api-access-vgxf8\") pod \"payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w\" (UID: \"25c89b7d-ea4e-425b-b35f-9bc061ecffdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w"
Oct 30 13:30:07.623403 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:07.623363    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn"]
Oct 30 13:30:07.725296 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:07.722191    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgxf8\" (UniqueName: \"kubernetes.io/projected/25c89b7d-ea4e-425b-b35f-9bc061ecffdb-kube-api-access-vgxf8\") pod \"payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w\" (UID: \"25c89b7d-ea4e-425b-b35f-9bc061ecffdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w"
Oct 30 13:30:07.725296 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:07.722321    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swxzr\" (UniqueName: \"kubernetes.io/projected/39386eb4-a783-44ca-b8f0-ecf162582aee-kube-api-access-swxzr\") pod \"payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn\" (UID: \"39386eb4-a783-44ca-b8f0-ecf162582aee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn"
Oct 30 13:30:07.725296 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:07.722544    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c89b7d-ea4e-425b-b35f-9bc061ecffdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w\" (UID: \"25c89b7d-ea4e-425b-b35f-9bc061ecffdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w"
Oct 30 13:30:07.725296 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:07.722588    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39386eb4-a783-44ca-b8f0-ecf162582aee-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn\" (UID: \"39386eb4-a783-44ca-b8f0-ecf162582aee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn"
Oct 30 13:30:07.725296 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:07.723593    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39386eb4-a783-44ca-b8f0-ecf162582aee-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn\" (UID: \"39386eb4-a783-44ca-b8f0-ecf162582aee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn"
Oct 30 13:30:07.770383 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:07.770312    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgxf8\" (UniqueName: \"kubernetes.io/projected/25c89b7d-ea4e-425b-b35f-9bc061ecffdb-kube-api-access-vgxf8\") pod \"payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w\" (UID: \"25c89b7d-ea4e-425b-b35f-9bc061ecffdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w"
Oct 30 13:30:07.774311 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:07.773009    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c89b7d-ea4e-425b-b35f-9bc061ecffdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w\" (UID: \"25c89b7d-ea4e-425b-b35f-9bc061ecffdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w"
Oct 30 13:30:07.813624 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:07.813591    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swxzr\" (UniqueName: \"kubernetes.io/projected/39386eb4-a783-44ca-b8f0-ecf162582aee-kube-api-access-swxzr\") pod \"payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn\" (UID: \"39386eb4-a783-44ca-b8f0-ecf162582aee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn"
Oct 30 13:30:07.864372 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:07.864328    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w"
Oct 30 13:30:07.921896 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:07.921859    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn"
Oct 30 13:30:08.037668 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:08.037634    4544 scope.go:122] "RemoveContainer" containerID="5c9da85d34150b7e35bc36fd3864ad8b6b5e7e4e9692c8a3094644192a941b4a"
Oct 30 13:30:08.038604 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:30:08.038563    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k_odh-ai-gateway-infra(dcbe2566-fac2-4912-9f80-9e3dbb313226)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k" podUID="dcbe2566-fac2-4912-9f80-9e3dbb313226"
Oct 30 13:30:08.198325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:08.198295    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w"]
Oct 30 13:30:08.200037 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:30:08.200012    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25c89b7d_ea4e_425b_b35f_9bc061ecffdb.slice/crio-b4c0e72f61379e71e2cfa592079d9f781dbb818658014fd9d8b64566e542a638 WatchSource:0}: Error finding container b4c0e72f61379e71e2cfa592079d9f781dbb818658014fd9d8b64566e542a638: Status 404 returned error can't find the container with id b4c0e72f61379e71e2cfa592079d9f781dbb818658014fd9d8b64566e542a638
Oct 30 13:30:08.262550 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:08.262522    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn"]
Oct 30 13:30:08.268940 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:30:08.268905    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39386eb4_a783_44ca_b8f0_ecf162582aee.slice/crio-f6467cea6a07e6718b94026761560099f38c4121eb0caf745afb1d561708ec58 WatchSource:0}: Error finding container f6467cea6a07e6718b94026761560099f38c4121eb0caf745afb1d561708ec58: Status 404 returned error can't find the container with id f6467cea6a07e6718b94026761560099f38c4121eb0caf745afb1d561708ec58
Oct 30 13:30:09.019813 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:09.019777    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:09.047023 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:09.046985    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn" event={"ID":"39386eb4-a783-44ca-b8f0-ecf162582aee","Type":"ContainerStarted","Data":"3f1f6210850e23bc3f4e9c6a26c8db6c40fb6dc0daf0b87fd65c5f0721b2bd26"}
Oct 30 13:30:09.047525 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:09.047057    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn"
Oct 30 13:30:09.047525 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:09.047084    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn" event={"ID":"39386eb4-a783-44ca-b8f0-ecf162582aee","Type":"ContainerStarted","Data":"f6467cea6a07e6718b94026761560099f38c4121eb0caf745afb1d561708ec58"}
Oct 30 13:30:09.048741 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:09.048710    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w" event={"ID":"25c89b7d-ea4e-425b-b35f-9bc061ecffdb","Type":"ContainerStarted","Data":"ba362cd24f4fb79cbbf3d03979d7cfe699d54e7c893155241f5bfa41b4523a2b"}
Oct 30 13:30:09.048866 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:09.048762    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w" event={"ID":"25c89b7d-ea4e-425b-b35f-9bc061ecffdb","Type":"ContainerStarted","Data":"b4c0e72f61379e71e2cfa592079d9f781dbb818658014fd9d8b64566e542a638"}
Oct 30 13:30:09.048866 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:09.048792    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w"
Oct 30 13:30:09.111841 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:09.111790    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn" podStartSLOduration=2.111775316 podStartE2EDuration="2.111775316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:09.104805487 +0000 UTC m=+2704.151939441" watchObservedRunningTime="2026-10-30 13:30:09.111775316 +0000 UTC m=+2704.158909269"
Oct 30 13:30:09.147499 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:09.147420    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w" podStartSLOduration=2.147402703 podStartE2EDuration="2.147402703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:09.145523236 +0000 UTC m=+2704.192657189" watchObservedRunningTime="2026-10-30 13:30:09.147402703 +0000 UTC m=+2704.194536657"
Oct 30 13:30:10.118376 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.118339    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:10.217938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.217900    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"]
Oct 30 13:30:10.223254 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.223196    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:30:10.242444 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.242415    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-1ba901-openshift-default-dockercfg-fl7pd\""
Oct 30 13:30:10.262276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.259703    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-credential-socket\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:30:10.262276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.259758    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istio-podinfo\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:30:10.262276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.259808    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istio-token\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:30:10.262276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.259942    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-workload-socket\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:30:10.262276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.260001    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smh6p\" (UniqueName: \"kubernetes.io/projected/d85faa5b-0e4f-477e-a7ac-784f37404bb2-kube-api-access-smh6p\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:30:10.262276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.260047    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istio-envoy\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:30:10.262276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.260129    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-workload-certs\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:30:10.262276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.260333    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istio-data\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:30:10.262276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.260365    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istiod-ca-cert\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:30:10.264318 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.263129    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"]
Oct 30 13:30:10.295121 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.295072    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:10.362347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.362291    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istio-data\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:30:10.362347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.362329    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istiod-ca-cert\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:30:10.362620 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.362410    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-credential-socket\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:30:10.362620 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.362444    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istio-podinfo\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:30:10.362620 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.362486    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istio-token\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:30:10.362808 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.362640    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-workload-socket\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:30:10.362808 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.362747    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istio-data\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:30:10.362808 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.362756    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smh6p\" (UniqueName: \"kubernetes.io/projected/d85faa5b-0e4f-477e-a7ac-784f37404bb2-kube-api-access-smh6p\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:30:10.362984 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.362815    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istio-envoy\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:30:10.362984 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.362897    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-workload-certs\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:30:10.363096 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.363007    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-workload-socket\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:30:10.363155 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.363134    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istiod-ca-cert\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:30:10.363208 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.363153    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-credential-socket\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:30:10.363384 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.363326    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-workload-certs\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:30:10.365018 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.364990    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istio-envoy\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:30:10.365274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.365255    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istio-podinfo\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:30:10.390046 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.389961    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istio-token\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:30:10.390239 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.390062    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smh6p\" (UniqueName: \"kubernetes.io/projected/d85faa5b-0e4f-477e-a7ac-784f37404bb2-kube-api-access-smh6p\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:30:10.540593 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.540555    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:30:10.874408 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.874373    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"]
Oct 30 13:30:10.875007 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:30:10.874973    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd85faa5b_0e4f_477e_a7ac_784f37404bb2.slice/crio-83cb52e05bed32b14f2e41f250ea5366ca2c9c7e1151e5294143fcb867614914 WatchSource:0}: Error finding container 83cb52e05bed32b14f2e41f250ea5366ca2c9c7e1151e5294143fcb867614914: Status 404 returned error can't find the container with id 83cb52e05bed32b14f2e41f250ea5366ca2c9c7e1151e5294143fcb867614914
Oct 30 13:30:10.878032 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.877992    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:30:10.878139 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.878073    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:30:10.878139 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:10.878104    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:30:11.068816 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:11.068779    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc" event={"ID":"d85faa5b-0e4f-477e-a7ac-784f37404bb2","Type":"ContainerStarted","Data":"8cf8fadf6c3b8d0bdc27229bcdc77cb1d7e03a6e6972d7419ff800702a686a0c"}
Oct 30 13:30:11.069045 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:11.068845    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc" event={"ID":"d85faa5b-0e4f-477e-a7ac-784f37404bb2","Type":"ContainerStarted","Data":"83cb52e05bed32b14f2e41f250ea5366ca2c9c7e1151e5294143fcb867614914"}
Oct 30 13:30:11.122920 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:11.122806    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc" podStartSLOduration=1.122787441 podStartE2EDuration="1.122787441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:11.114987244 +0000 UTC m=+2706.162121197" watchObservedRunningTime="2026-10-30 13:30:11.122787441 +0000 UTC m=+2706.169921393"
Oct 30 13:30:11.215478 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:11.215443    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:11.541361 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:11.541285    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:30:11.543489 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:11.543453    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 30 13:30:11.543648 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:11.543530    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc" podUID="d85faa5b-0e4f-477e-a7ac-784f37404bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 30 13:30:11.988964 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:11.988917    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd" podUID="84804195-56b8-4aae-99a4-73ee1e589fed" containerName="main" containerID="cri-o://ef9463b3088302cd0c552038a5ce5ad70eddd2c1f059aa2f27e20b11040de16d" gracePeriod=45
Oct 30 13:30:12.542491 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:12.542447    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 30 13:30:12.543012 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:12.542533    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc" podUID="d85faa5b-0e4f-477e-a7ac-784f37404bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 30 13:30:12.580970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:12.580933    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd"
Oct 30 13:30:12.645170 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:12.645079    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84804195-56b8-4aae-99a4-73ee1e589fed-home\" (UniqueName: \"kubernetes.io/empty-dir/84804195-56b8-4aae-99a4-73ee1e589fed-home\") pod \"84804195-56b8-4aae-99a4-73ee1e589fed\" (UID: \"84804195-56b8-4aae-99a4-73ee1e589fed\") "
Oct 30 13:30:12.645170 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:12.645136    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84804195-56b8-4aae-99a4-73ee1e589fed-kube-api-access-lf7d6\" (UniqueName: \"kubernetes.io/projected/84804195-56b8-4aae-99a4-73ee1e589fed-kube-api-access-lf7d6\") pod \"84804195-56b8-4aae-99a4-73ee1e589fed\" (UID: \"84804195-56b8-4aae-99a4-73ee1e589fed\") "
Oct 30 13:30:12.645170 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:12.645167    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84804195-56b8-4aae-99a4-73ee1e589fed-tls-certs\" (UniqueName: \"kubernetes.io/secret/84804195-56b8-4aae-99a4-73ee1e589fed-tls-certs\") pod \"84804195-56b8-4aae-99a4-73ee1e589fed\" (UID: \"84804195-56b8-4aae-99a4-73ee1e589fed\") "
Oct 30 13:30:12.645537 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:12.645210    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84804195-56b8-4aae-99a4-73ee1e589fed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84804195-56b8-4aae-99a4-73ee1e589fed-model-cache\") pod \"84804195-56b8-4aae-99a4-73ee1e589fed\" (UID: \"84804195-56b8-4aae-99a4-73ee1e589fed\") "
Oct 30 13:30:12.645537 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:12.645311    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84804195-56b8-4aae-99a4-73ee1e589fed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84804195-56b8-4aae-99a4-73ee1e589fed-tmp-dir\") pod \"84804195-56b8-4aae-99a4-73ee1e589fed\" (UID: \"84804195-56b8-4aae-99a4-73ee1e589fed\") "
Oct 30 13:30:12.645537 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:12.645364    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84804195-56b8-4aae-99a4-73ee1e589fed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/84804195-56b8-4aae-99a4-73ee1e589fed-dshm\") pod \"84804195-56b8-4aae-99a4-73ee1e589fed\" (UID: \"84804195-56b8-4aae-99a4-73ee1e589fed\") "
Oct 30 13:30:12.645537 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:12.645432    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84804195-56b8-4aae-99a4-73ee1e589fed-home" pod "84804195-56b8-4aae-99a4-73ee1e589fed" (UID: "84804195-56b8-4aae-99a4-73ee1e589fed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:30:12.645720 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:12.645584    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84804195-56b8-4aae-99a4-73ee1e589fed-tmp-dir" pod "84804195-56b8-4aae-99a4-73ee1e589fed" (UID: "84804195-56b8-4aae-99a4-73ee1e589fed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:30:12.645836 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:12.645805    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84804195-56b8-4aae-99a4-73ee1e589fed-model-cache" pod "84804195-56b8-4aae-99a4-73ee1e589fed" (UID: "84804195-56b8-4aae-99a4-73ee1e589fed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:30:12.646189 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:12.646166    4544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84804195-56b8-4aae-99a4-73ee1e589fed-home\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:30:12.646299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:12.646195    4544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84804195-56b8-4aae-99a4-73ee1e589fed-model-cache\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:30:12.646299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:12.646210    4544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84804195-56b8-4aae-99a4-73ee1e589fed-tmp-dir\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:30:12.647607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:12.647582    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84804195-56b8-4aae-99a4-73ee1e589fed-dshm" pod "84804195-56b8-4aae-99a4-73ee1e589fed" (UID: "84804195-56b8-4aae-99a4-73ee1e589fed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:30:12.647772 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:12.647751    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84804195-56b8-4aae-99a4-73ee1e589fed-tls-certs" pod "84804195-56b8-4aae-99a4-73ee1e589fed" (UID: "84804195-56b8-4aae-99a4-73ee1e589fed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:30:12.647839 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:12.647771    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84804195-56b8-4aae-99a4-73ee1e589fed-kube-api-access-lf7d6" pod "84804195-56b8-4aae-99a4-73ee1e589fed" (UID: "84804195-56b8-4aae-99a4-73ee1e589fed"). InnerVolumeSpecName "kube-api-access-lf7d6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:12.747259 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:12.747207    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf7d6\" (UniqueName: \"kubernetes.io/projected/84804195-56b8-4aae-99a4-73ee1e589fed-kube-api-access-lf7d6\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:30:12.747259 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:12.747263    4544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84804195-56b8-4aae-99a4-73ee1e589fed-tls-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:30:12.747495 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:12.747276    4544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84804195-56b8-4aae-99a4-73ee1e589fed-dshm\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:30:12.775632 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:12.775584    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:13.172697 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:13.172665    4544 generic.go:356] "Generic (PLEG): container finished" podID="84804195-56b8-4aae-99a4-73ee1e589fed" containerID="ef9463b3088302cd0c552038a5ce5ad70eddd2c1f059aa2f27e20b11040de16d" exitCode=0
Oct 30 13:30:13.172874 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:13.172711    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd"
Oct 30 13:30:13.172874 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:13.172750    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd" event={"ID":"84804195-56b8-4aae-99a4-73ee1e589fed","Type":"ContainerDied","Data":"ef9463b3088302cd0c552038a5ce5ad70eddd2c1f059aa2f27e20b11040de16d"}
Oct 30 13:30:13.172874 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:13.172801    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd" event={"ID":"84804195-56b8-4aae-99a4-73ee1e589fed","Type":"ContainerDied","Data":"e8336e870279968a99bbabe0de142a7b40e1e63c80b37a529801ea2cac49423b"}
Oct 30 13:30:13.172874 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:13.172828    4544 scope.go:122] "RemoveContainer" containerID="ef9463b3088302cd0c552038a5ce5ad70eddd2c1f059aa2f27e20b11040de16d"
Oct 30 13:30:13.183902 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:13.183886    4544 scope.go:122] "RemoveContainer" containerID="ef9463b3088302cd0c552038a5ce5ad70eddd2c1f059aa2f27e20b11040de16d"
Oct 30 13:30:13.184168 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:30:13.184148    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef9463b3088302cd0c552038a5ce5ad70eddd2c1f059aa2f27e20b11040de16d\": container with ID starting with ef9463b3088302cd0c552038a5ce5ad70eddd2c1f059aa2f27e20b11040de16d not found: ID does not exist" containerID="ef9463b3088302cd0c552038a5ce5ad70eddd2c1f059aa2f27e20b11040de16d"
Oct 30 13:30:13.184253 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:13.184174    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef9463b3088302cd0c552038a5ce5ad70eddd2c1f059aa2f27e20b11040de16d"} err="failed to get container status \"ef9463b3088302cd0c552038a5ce5ad70eddd2c1f059aa2f27e20b11040de16d\": rpc error: code = NotFound desc = could not find container \"ef9463b3088302cd0c552038a5ce5ad70eddd2c1f059aa2f27e20b11040de16d\": container with ID starting with ef9463b3088302cd0c552038a5ce5ad70eddd2c1f059aa2f27e20b11040de16d not found: ID does not exist"
Oct 30 13:30:13.245471 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:13.245434    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd"]
Oct 30 13:30:13.250495 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:13.250466    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd"]
Oct 30 13:30:13.468283 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:13.468181    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:13.526245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:13.525423    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-3e8ee4/auth-test-model-3e8ee4-kserve-87b4879cc-c2qvd" podUID="84804195-56b8-4aae-99a4-73ee1e589fed" containerName="main" probeResult="failure" output="Get \"https://10.133.1.6:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:30:13.545258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:13.545033    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 30 13:30:13.545258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:13.545134    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc" podUID="d85faa5b-0e4f-477e-a7ac-784f37404bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 30 13:30:14.004354 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:14.004316    4544 scope.go:122] "RemoveContainer" containerID="74e100dc43c2c66516f42c81f5df8dde200f49202da0bfb7c93d787c50e70102"
Oct 30 13:30:14.044385 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:14.044330    4544 scope.go:122] "RemoveContainer" containerID="d4f279d2fe1fb18ab4103671744e9b4b7cb1133d4a0c65811e6b4f486fff38d1"
Oct 30 13:30:14.056608 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:14.056585    4544 scope.go:122] "RemoveContainer" containerID="38dc114eef1c13fe6d0baa6f99d04271bf3000ef3169e19e318403c616a46364"
Oct 30 13:30:14.541867 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:14.541823    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 30 13:30:14.542103 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:14.541900    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc" podUID="d85faa5b-0e4f-477e-a7ac-784f37404bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 30 13:30:15.090823 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:15.090791    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84804195-56b8-4aae-99a4-73ee1e589fed" path="/var/lib/kubelet/pods/84804195-56b8-4aae-99a4-73ee1e589fed/volumes"
Oct 30 13:30:15.338800 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:15.338746    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:15.541852 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:15.541767    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 30 13:30:15.541852 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:15.541836    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc" podUID="d85faa5b-0e4f-477e-a7ac-784f37404bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 30 13:30:16.078989 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:16.078958    4544 scope.go:122] "RemoveContainer" containerID="d0648c55aa30dc913c771498e1c4b8cf39811fc95fbf690a8fb006cd0f9964d8"
Oct 30 13:30:16.541919 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:16.541876    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 30 13:30:16.542349 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:16.541987    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc" podUID="d85faa5b-0e4f-477e-a7ac-784f37404bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 30 13:30:17.078577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:17.078550    4544 scope.go:122] "RemoveContainer" containerID="245c44f2f59179e39c035d831cc87762fd3a9d7df2f6b11b3889d8ec3e9f19a8"
Oct 30 13:30:17.176307 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:17.176214    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:17.209547 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:17.209508    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5" event={"ID":"ebce006c-8200-434b-99e8-5934eadad9a3","Type":"ContainerStarted","Data":"aee32f8083ed9e0a65aa44e17e9e3e1b1a6e886c86840e3f69c7449b878d29cf"}
Oct 30 13:30:17.541950 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:17.541910    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 30 13:30:17.542376 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:17.541977    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc" podUID="d85faa5b-0e4f-477e-a7ac-784f37404bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 30 13:30:18.220447 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.220411    4544 generic.go:356] "Generic (PLEG): container finished" podID="ebce006c-8200-434b-99e8-5934eadad9a3" containerID="aee32f8083ed9e0a65aa44e17e9e3e1b1a6e886c86840e3f69c7449b878d29cf" exitCode=60
Oct 30 13:30:18.220650 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.220488    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5" event={"ID":"ebce006c-8200-434b-99e8-5934eadad9a3","Type":"ContainerDied","Data":"aee32f8083ed9e0a65aa44e17e9e3e1b1a6e886c86840e3f69c7449b878d29cf"}
Oct 30 13:30:18.220650 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.220557    4544 scope.go:122] "RemoveContainer" containerID="d0648c55aa30dc913c771498e1c4b8cf39811fc95fbf690a8fb006cd0f9964d8"
Oct 30 13:30:18.221048 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.221029    4544 scope.go:122] "RemoveContainer" containerID="aee32f8083ed9e0a65aa44e17e9e3e1b1a6e886c86840e3f69c7449b878d29cf"
Oct 30 13:30:18.221510 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:30:18.221472    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5_odh-ai-gateway-infra(ebce006c-8200-434b-99e8-5934eadad9a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5" podUID="ebce006c-8200-434b-99e8-5934eadad9a3"
Oct 30 13:30:18.222962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.222940    4544 generic.go:356] "Generic (PLEG): container finished" podID="00b65da6-a93a-4b46-8a6f-912049acda4e" containerID="c03cee82e86aa94c5bbee5081aede34eb534e4f2b09d995ed16be767ebd26493" exitCode=60
Oct 30 13:30:18.223064 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.223021    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk" event={"ID":"00b65da6-a93a-4b46-8a6f-912049acda4e","Type":"ContainerDied","Data":"c03cee82e86aa94c5bbee5081aede34eb534e4f2b09d995ed16be767ebd26493"}
Oct 30 13:30:18.223561 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.223541    4544 scope.go:122] "RemoveContainer" containerID="c03cee82e86aa94c5bbee5081aede34eb534e4f2b09d995ed16be767ebd26493"
Oct 30 13:30:18.223922 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:30:18.223902    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk_odh-ai-gateway-infra(00b65da6-a93a-4b46-8a6f-912049acda4e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk" podUID="00b65da6-a93a-4b46-8a6f-912049acda4e"
Oct 30 13:30:18.237042 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.237021    4544 scope.go:122] "RemoveContainer" containerID="245c44f2f59179e39c035d831cc87762fd3a9d7df2f6b11b3889d8ec3e9f19a8"
Oct 30 13:30:18.293747 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.293706    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5"]
Oct 30 13:30:18.442970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.442878    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf"]
Oct 30 13:30:18.444044 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.444016    4544 cpu_manager.go:414] "containerMap: removing container" podUID="84804195-56b8-4aae-99a4-73ee1e589fed" containerName="main"
Oct 30 13:30:18.444044 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.444041    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84804195-56b8-4aae-99a4-73ee1e589fed" containerName="main"
Oct 30 13:30:18.444240 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.444193    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="84804195-56b8-4aae-99a4-73ee1e589fed" containerName="main"
Oct 30 13:30:18.449542 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.449514    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf"
Oct 30 13:30:18.472476 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.472444    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-8a5d5e\"/\"default-dockercfg-bjxjj\""
Oct 30 13:30:18.472662 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.472573    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-8a5d5e\"/\"auth-test-model-8a5d5e-kserve-self-signed-certs\""
Oct 30 13:30:18.472662 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.472613    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-8a5d5e\"/\"openshift-service-ca.crt\""
Oct 30 13:30:18.472916 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.472903    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-8a5d5e\"/\"kube-root-ca.crt\""
Oct 30 13:30:18.484819 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.484783    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf"]
Oct 30 13:30:18.547570 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.547538    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:30:18.548076 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.548052    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:30:18.548899 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.548879    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:30:18.579176 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.579135    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05286a92-510f-4ddf-acdc-1dab447686f1-tls-certs\") pod \"auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf\" (UID: \"05286a92-510f-4ddf-acdc-1dab447686f1\") " pod="ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf"
Oct 30 13:30:18.579437 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.579261    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp8h5\" (UniqueName: \"kubernetes.io/projected/05286a92-510f-4ddf-acdc-1dab447686f1-kube-api-access-hp8h5\") pod \"auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf\" (UID: \"05286a92-510f-4ddf-acdc-1dab447686f1\") " pod="ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf"
Oct 30 13:30:18.579520 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.579428    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05286a92-510f-4ddf-acdc-1dab447686f1-home\") pod \"auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf\" (UID: \"05286a92-510f-4ddf-acdc-1dab447686f1\") " pod="ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf"
Oct 30 13:30:18.579520 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.579500    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05286a92-510f-4ddf-acdc-1dab447686f1-dshm\") pod \"auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf\" (UID: \"05286a92-510f-4ddf-acdc-1dab447686f1\") " pod="ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf"
Oct 30 13:30:18.579635 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.579565    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05286a92-510f-4ddf-acdc-1dab447686f1-tmp-dir\") pod \"auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf\" (UID: \"05286a92-510f-4ddf-acdc-1dab447686f1\") " pod="ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf"
Oct 30 13:30:18.579741 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.579725    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05286a92-510f-4ddf-acdc-1dab447686f1-model-cache\") pod \"auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf\" (UID: \"05286a92-510f-4ddf-acdc-1dab447686f1\") " pod="ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf"
Oct 30 13:30:18.619579 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.619538    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:18.681681 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.681637    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp8h5\" (UniqueName: \"kubernetes.io/projected/05286a92-510f-4ddf-acdc-1dab447686f1-kube-api-access-hp8h5\") pod \"auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf\" (UID: \"05286a92-510f-4ddf-acdc-1dab447686f1\") " pod="ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf"
Oct 30 13:30:18.681901 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.681773    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05286a92-510f-4ddf-acdc-1dab447686f1-home\") pod \"auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf\" (UID: \"05286a92-510f-4ddf-acdc-1dab447686f1\") " pod="ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf"
Oct 30 13:30:18.681901 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.681840    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05286a92-510f-4ddf-acdc-1dab447686f1-dshm\") pod \"auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf\" (UID: \"05286a92-510f-4ddf-acdc-1dab447686f1\") " pod="ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf"
Oct 30 13:30:18.682038 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.681902    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05286a92-510f-4ddf-acdc-1dab447686f1-tmp-dir\") pod \"auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf\" (UID: \"05286a92-510f-4ddf-acdc-1dab447686f1\") " pod="ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf"
Oct 30 13:30:18.682098 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.682043    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05286a92-510f-4ddf-acdc-1dab447686f1-model-cache\") pod \"auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf\" (UID: \"05286a92-510f-4ddf-acdc-1dab447686f1\") " pod="ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf"
Oct 30 13:30:18.682161 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.682130    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05286a92-510f-4ddf-acdc-1dab447686f1-tls-certs\") pod \"auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf\" (UID: \"05286a92-510f-4ddf-acdc-1dab447686f1\") " pod="ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf"
Oct 30 13:30:18.682282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.682248    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05286a92-510f-4ddf-acdc-1dab447686f1-home\") pod \"auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf\" (UID: \"05286a92-510f-4ddf-acdc-1dab447686f1\") " pod="ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf"
Oct 30 13:30:18.683137 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.682567    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05286a92-510f-4ddf-acdc-1dab447686f1-model-cache\") pod \"auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf\" (UID: \"05286a92-510f-4ddf-acdc-1dab447686f1\") " pod="ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf"
Oct 30 13:30:18.683137 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.682901    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05286a92-510f-4ddf-acdc-1dab447686f1-tmp-dir\") pod \"auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf\" (UID: \"05286a92-510f-4ddf-acdc-1dab447686f1\") " pod="ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf"
Oct 30 13:30:18.685541 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.685206    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05286a92-510f-4ddf-acdc-1dab447686f1-tls-certs\") pod \"auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf\" (UID: \"05286a92-510f-4ddf-acdc-1dab447686f1\") " pod="ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf"
Oct 30 13:30:18.686343 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.686311    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05286a92-510f-4ddf-acdc-1dab447686f1-dshm\") pod \"auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf\" (UID: \"05286a92-510f-4ddf-acdc-1dab447686f1\") " pod="ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf"
Oct 30 13:30:18.709248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.709151    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp8h5\" (UniqueName: \"kubernetes.io/projected/05286a92-510f-4ddf-acdc-1dab447686f1-kube-api-access-hp8h5\") pod \"auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf\" (UID: \"05286a92-510f-4ddf-acdc-1dab447686f1\") " pod="ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf"
Oct 30 13:30:18.765023 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:18.764982    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf"
Oct 30 13:30:19.078817 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:19.078764    4544 scope.go:122] "RemoveContainer" containerID="5c9da85d34150b7e35bc36fd3864ad8b6b5e7e4e9692c8a3094644192a941b4a"
Oct 30 13:30:19.079027 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:19.078958    4544 scope.go:122] "RemoveContainer" containerID="89653efeca3bf1eadae508666013a754b714795695a21fc7b165ef619462102f"
Oct 30 13:30:19.079795 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:19.079772    4544 scope.go:122] "RemoveContainer" containerID="36158f0bb6b48dd4c92295bdebd85ea90a8caad1b6d4597707f493149b5e8249"
Oct 30 13:30:19.079960 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:19.079937    4544 scope.go:122] "RemoveContainer" containerID="18e0559709bcbb20645349e6c19d6589edfe26586b2c459d70caa551a2d56353"
Oct 30 13:30:19.109861 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:19.109819    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf"]
Oct 30 13:30:19.113030 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:30:19.112993    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05286a92_510f_4ddf_acdc_1dab447686f1.slice/crio-9d1b01142f3d96ae9c4543e4d345a96ff688ed597480fb8fc46cfe55155ed912 WatchSource:0}: Error finding container 9d1b01142f3d96ae9c4543e4d345a96ff688ed597480fb8fc46cfe55155ed912: Status 404 returned error can't find the container with id 9d1b01142f3d96ae9c4543e4d345a96ff688ed597480fb8fc46cfe55155ed912
Oct 30 13:30:19.214277 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:19.214201    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:19.237412 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:19.237373    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf" event={"ID":"05286a92-510f-4ddf-acdc-1dab447686f1","Type":"ContainerStarted","Data":"df24c80262d782620e2a635604d43ea6d7e2caaeba877419a49af5f7534cb600"}
Oct 30 13:30:19.237579 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:19.237433    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf" event={"ID":"05286a92-510f-4ddf-acdc-1dab447686f1","Type":"ContainerStarted","Data":"9d1b01142f3d96ae9c4543e4d345a96ff688ed597480fb8fc46cfe55155ed912"}
Oct 30 13:30:19.327614 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:19.326482    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk"]
Oct 30 13:30:19.352962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:19.352862    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf" podStartSLOduration=1.352845118 podStartE2EDuration="1.352845118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:19.347339629 +0000 UTC m=+2714.394473595" watchObservedRunningTime="2026-10-30 13:30:19.352845118 +0000 UTC m=+2714.399979071"
Oct 30 13:30:19.555633 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:19.555584    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk"
Oct 30 13:30:19.619821 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:19.618668    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00b65da6-a93a-4b46-8a6f-912049acda4e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00b65da6-a93a-4b46-8a6f-912049acda4e-maas-api-service-ca\") pod \"00b65da6-a93a-4b46-8a6f-912049acda4e\" (UID: \"00b65da6-a93a-4b46-8a6f-912049acda4e\") "
Oct 30 13:30:19.619821 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:19.618770    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00b65da6-a93a-4b46-8a6f-912049acda4e-kube-api-access-ds7rl\" (UniqueName: \"kubernetes.io/projected/00b65da6-a93a-4b46-8a6f-912049acda4e-kube-api-access-ds7rl\") pod \"00b65da6-a93a-4b46-8a6f-912049acda4e\" (UID: \"00b65da6-a93a-4b46-8a6f-912049acda4e\") "
Oct 30 13:30:19.623448 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:19.623410    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00b65da6-a93a-4b46-8a6f-912049acda4e-kube-api-access-ds7rl" pod "00b65da6-a93a-4b46-8a6f-912049acda4e" (UID: "00b65da6-a93a-4b46-8a6f-912049acda4e"). InnerVolumeSpecName "kube-api-access-ds7rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:19.623808 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:19.623784    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00b65da6-a93a-4b46-8a6f-912049acda4e-maas-api-service-ca" pod "00b65da6-a93a-4b46-8a6f-912049acda4e" (UID: "00b65da6-a93a-4b46-8a6f-912049acda4e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:19.637770 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:19.637737    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5"
Oct 30 13:30:19.720150 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:19.720109    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebce006c-8200-434b-99e8-5934eadad9a3-kube-api-access-gdm9c\" (UniqueName: \"kubernetes.io/projected/ebce006c-8200-434b-99e8-5934eadad9a3-kube-api-access-gdm9c\") pod \"ebce006c-8200-434b-99e8-5934eadad9a3\" (UID: \"ebce006c-8200-434b-99e8-5934eadad9a3\") "
Oct 30 13:30:19.720375 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:19.720334    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebce006c-8200-434b-99e8-5934eadad9a3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebce006c-8200-434b-99e8-5934eadad9a3-maas-api-service-ca\") pod \"ebce006c-8200-434b-99e8-5934eadad9a3\" (UID: \"ebce006c-8200-434b-99e8-5934eadad9a3\") "
Oct 30 13:30:19.720727 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:19.720681    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebce006c-8200-434b-99e8-5934eadad9a3-maas-api-service-ca" pod "ebce006c-8200-434b-99e8-5934eadad9a3" (UID: "ebce006c-8200-434b-99e8-5934eadad9a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:19.720961 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:19.720942    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00b65da6-a93a-4b46-8a6f-912049acda4e-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:30:19.721057 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:19.720962    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebce006c-8200-434b-99e8-5934eadad9a3-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:30:19.721057 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:19.720972    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds7rl\" (UniqueName: \"kubernetes.io/projected/00b65da6-a93a-4b46-8a6f-912049acda4e-kube-api-access-ds7rl\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:30:19.724116 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:19.724079    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebce006c-8200-434b-99e8-5934eadad9a3-kube-api-access-gdm9c" pod "ebce006c-8200-434b-99e8-5934eadad9a3" (UID: "ebce006c-8200-434b-99e8-5934eadad9a3"). InnerVolumeSpecName "kube-api-access-gdm9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:19.765875 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:19.765825    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf"
Oct 30 13:30:19.822819 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:19.822779    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdm9c\" (UniqueName: \"kubernetes.io/projected/ebce006c-8200-434b-99e8-5934eadad9a3-kube-api-access-gdm9c\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:30:20.061483 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:20.061388    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w"
Oct 30 13:30:20.061779 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:20.061751    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn"
Oct 30 13:30:20.078694 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:20.078654    4544 scope.go:122] "RemoveContainer" containerID="51a750784688719f19c1230e0088d86d3b1a96f188c604471780b0617ed6b695"
Oct 30 13:30:20.261888 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:20.261851    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k" event={"ID":"dcbe2566-fac2-4912-9f80-9e3dbb313226","Type":"ContainerStarted","Data":"ad195918dff29215fa59c6da2aa9b5cc5dda7e7d7f5cf98b2cf7e9cc2447b483"}
Oct 30 13:30:20.264394 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:20.264356    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c" event={"ID":"d6246b4d-a9f8-495d-8f1d-8f8f57fee223","Type":"ContainerStarted","Data":"6196b4086c2cabe8a8a8002049dfbd58813d80a326c75bce08cc90f3626f6e45"}
Oct 30 13:30:20.266378 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:20.266354    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5"
Oct 30 13:30:20.266514 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:20.266382    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5" event={"ID":"ebce006c-8200-434b-99e8-5934eadad9a3","Type":"ContainerDied","Data":"a809fe36ef98151047b4ecd35469e08488f13369e7f405ae1e6948da5032414a"}
Oct 30 13:30:20.266514 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:20.266466    4544 scope.go:122] "RemoveContainer" containerID="aee32f8083ed9e0a65aa44e17e9e3e1b1a6e886c86840e3f69c7449b878d29cf"
Oct 30 13:30:20.275474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:20.274734    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk" event={"ID":"00b65da6-a93a-4b46-8a6f-912049acda4e","Type":"ContainerDied","Data":"3c6819c7689dbc3dfce2583afb1cbfee5c1933ec0a7aaa2e3fbba49ce7448fca"}
Oct 30 13:30:20.275474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:20.274989    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk"
Oct 30 13:30:20.281853 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:20.281628    4544 generic.go:356] "Generic (PLEG): container finished" podID="7c9fb4e3-399f-4870-943c-8744296adab5" containerID="bf75bf53c41a76242b88910b4caec1da97833da8caa038d0875209cc59ee73e5" exitCode=60
Oct 30 13:30:20.281853 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:20.281709    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427" event={"ID":"7c9fb4e3-399f-4870-943c-8744296adab5","Type":"ContainerDied","Data":"bf75bf53c41a76242b88910b4caec1da97833da8caa038d0875209cc59ee73e5"}
Oct 30 13:30:20.282634 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:20.282344    4544 scope.go:122] "RemoveContainer" containerID="bf75bf53c41a76242b88910b4caec1da97833da8caa038d0875209cc59ee73e5"
Oct 30 13:30:20.283089 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:30:20.283028    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427_odh-ai-gateway-infra(7c9fb4e3-399f-4870-943c-8744296adab5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427" podUID="7c9fb4e3-399f-4870-943c-8744296adab5"
Oct 30 13:30:20.286552 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:20.286525    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t" event={"ID":"0625ba18-6017-481b-9bf2-983358ae2123","Type":"ContainerStarted","Data":"50e962420e0f35f8fe4af191a31a19a869639ea36fac60aafc6efbc052f6d544"}
Oct 30 13:30:20.446850 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:20.446819    4544 scope.go:122] "RemoveContainer" containerID="c03cee82e86aa94c5bbee5081aede34eb534e4f2b09d995ed16be767ebd26493"
Oct 30 13:30:20.478550 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:20.478522    4544 scope.go:122] "RemoveContainer" containerID="36158f0bb6b48dd4c92295bdebd85ea90a8caad1b6d4597707f493149b5e8249"
Oct 30 13:30:20.521411 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:20.521375    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5"]
Oct 30 13:30:20.528840 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:20.528804    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-8a5d5e-29889450-62xn5"]
Oct 30 13:30:20.569821 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:20.569682    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk"]
Oct 30 13:30:20.573214 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:20.573179    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889450-hlmbk"]
Oct 30 13:30:21.092810 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:21.092774    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00b65da6-a93a-4b46-8a6f-912049acda4e" path="/var/lib/kubelet/pods/00b65da6-a93a-4b46-8a6f-912049acda4e/volumes"
Oct 30 13:30:21.093349 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:21.093331    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebce006c-8200-434b-99e8-5934eadad9a3" path="/var/lib/kubelet/pods/ebce006c-8200-434b-99e8-5934eadad9a3/volumes"
Oct 30 13:30:21.110585 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:21.110552    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:21.299691 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:21.299657    4544 generic.go:356] "Generic (PLEG): container finished" podID="0625ba18-6017-481b-9bf2-983358ae2123" containerID="50e962420e0f35f8fe4af191a31a19a869639ea36fac60aafc6efbc052f6d544" exitCode=60
Oct 30 13:30:21.299880 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:21.299749    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t" event={"ID":"0625ba18-6017-481b-9bf2-983358ae2123","Type":"ContainerDied","Data":"50e962420e0f35f8fe4af191a31a19a869639ea36fac60aafc6efbc052f6d544"}
Oct 30 13:30:21.299880 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:21.299832    4544 scope.go:122] "RemoveContainer" containerID="89653efeca3bf1eadae508666013a754b714795695a21fc7b165ef619462102f"
Oct 30 13:30:21.300313 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:21.300297    4544 scope.go:122] "RemoveContainer" containerID="50e962420e0f35f8fe4af191a31a19a869639ea36fac60aafc6efbc052f6d544"
Oct 30 13:30:21.300722 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:30:21.300698    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t_odh-ai-gateway-infra(0625ba18-6017-481b-9bf2-983358ae2123)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t" podUID="0625ba18-6017-481b-9bf2-983358ae2123"
Oct 30 13:30:21.302849 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:21.302823    4544 generic.go:356] "Generic (PLEG): container finished" podID="e1a04d8b-24e3-4321-b21c-4e2d698ea5d2" containerID="9f72dc706fb697d04c39fee90ba3b154137036683a855a390781e3e4049e40aa" exitCode=60
Oct 30 13:30:21.302934 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:21.302892    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf" event={"ID":"e1a04d8b-24e3-4321-b21c-4e2d698ea5d2","Type":"ContainerDied","Data":"9f72dc706fb697d04c39fee90ba3b154137036683a855a390781e3e4049e40aa"}
Oct 30 13:30:21.303395 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:21.303375    4544 scope.go:122] "RemoveContainer" containerID="9f72dc706fb697d04c39fee90ba3b154137036683a855a390781e3e4049e40aa"
Oct 30 13:30:21.303791 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:30:21.303767    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf_odh-ai-gateway-infra(e1a04d8b-24e3-4321-b21c-4e2d698ea5d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf" podUID="e1a04d8b-24e3-4321-b21c-4e2d698ea5d2"
Oct 30 13:30:21.305328 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:21.305309    4544 generic.go:356] "Generic (PLEG): container finished" podID="dcbe2566-fac2-4912-9f80-9e3dbb313226" containerID="ad195918dff29215fa59c6da2aa9b5cc5dda7e7d7f5cf98b2cf7e9cc2447b483" exitCode=60
Oct 30 13:30:21.305447 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:21.305391    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k" event={"ID":"dcbe2566-fac2-4912-9f80-9e3dbb313226","Type":"ContainerDied","Data":"ad195918dff29215fa59c6da2aa9b5cc5dda7e7d7f5cf98b2cf7e9cc2447b483"}
Oct 30 13:30:21.305879 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:21.305855    4544 scope.go:122] "RemoveContainer" containerID="ad195918dff29215fa59c6da2aa9b5cc5dda7e7d7f5cf98b2cf7e9cc2447b483"
Oct 30 13:30:21.306510 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:30:21.306353    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k_odh-ai-gateway-infra(dcbe2566-fac2-4912-9f80-9e3dbb313226)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k" podUID="dcbe2566-fac2-4912-9f80-9e3dbb313226"
Oct 30 13:30:21.308580 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:21.308557    4544 generic.go:356] "Generic (PLEG): container finished" podID="d6246b4d-a9f8-495d-8f1d-8f8f57fee223" containerID="6196b4086c2cabe8a8a8002049dfbd58813d80a326c75bce08cc90f3626f6e45" exitCode=60
Oct 30 13:30:21.308697 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:21.308631    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c" event={"ID":"d6246b4d-a9f8-495d-8f1d-8f8f57fee223","Type":"ContainerDied","Data":"6196b4086c2cabe8a8a8002049dfbd58813d80a326c75bce08cc90f3626f6e45"}
Oct 30 13:30:21.309330 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:21.309309    4544 scope.go:122] "RemoveContainer" containerID="6196b4086c2cabe8a8a8002049dfbd58813d80a326c75bce08cc90f3626f6e45"
Oct 30 13:30:21.309960 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:30:21.309936    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c_odh-ai-gateway-infra(d6246b4d-a9f8-495d-8f1d-8f8f57fee223)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c" podUID="d6246b4d-a9f8-495d-8f1d-8f8f57fee223"
Oct 30 13:30:21.318092 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:21.318051    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k"]
Oct 30 13:30:21.329538 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:21.329499    4544 scope.go:122] "RemoveContainer" containerID="51a750784688719f19c1230e0088d86d3b1a96f188c604471780b0617ed6b695"
Oct 30 13:30:21.345625 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:21.345602    4544 scope.go:122] "RemoveContainer" containerID="5c9da85d34150b7e35bc36fd3864ad8b6b5e7e4e9692c8a3094644192a941b4a"
Oct 30 13:30:21.362673 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:21.362591    4544 scope.go:122] "RemoveContainer" containerID="18e0559709bcbb20645349e6c19d6589edfe26586b2c459d70caa551a2d56353"
Oct 30 13:30:21.378158 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:21.378116    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427"]
Oct 30 13:30:21.425626 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:21.425589    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c"]
Oct 30 13:30:21.485270 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:21.483303    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t"]
Oct 30 13:30:21.603088 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:21.603015    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427"
Oct 30 13:30:21.656304 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:21.656268    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c9fb4e3-399f-4870-943c-8744296adab5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c9fb4e3-399f-4870-943c-8744296adab5-maas-api-service-ca\") pod \"7c9fb4e3-399f-4870-943c-8744296adab5\" (UID: \"7c9fb4e3-399f-4870-943c-8744296adab5\") "
Oct 30 13:30:21.656483 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:21.656351    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c9fb4e3-399f-4870-943c-8744296adab5-kube-api-access-6ngb8\" (UniqueName: \"kubernetes.io/projected/7c9fb4e3-399f-4870-943c-8744296adab5-kube-api-access-6ngb8\") pod \"7c9fb4e3-399f-4870-943c-8744296adab5\" (UID: \"7c9fb4e3-399f-4870-943c-8744296adab5\") "
Oct 30 13:30:21.657885 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:21.657850    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c9fb4e3-399f-4870-943c-8744296adab5-maas-api-service-ca" pod "7c9fb4e3-399f-4870-943c-8744296adab5" (UID: "7c9fb4e3-399f-4870-943c-8744296adab5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:21.659566 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:21.659528    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c9fb4e3-399f-4870-943c-8744296adab5-kube-api-access-6ngb8" pod "7c9fb4e3-399f-4870-943c-8744296adab5" (UID: "7c9fb4e3-399f-4870-943c-8744296adab5"). InnerVolumeSpecName "kube-api-access-6ngb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:21.758728 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:21.758685    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c9fb4e3-399f-4870-943c-8744296adab5-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:30:21.758728 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:21.758722    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ngb8\" (UniqueName: \"kubernetes.io/projected/7c9fb4e3-399f-4870-943c-8744296adab5-kube-api-access-6ngb8\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:30:22.325008 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.324973    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427"
Oct 30 13:30:22.325008 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.324996    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427" event={"ID":"7c9fb4e3-399f-4870-943c-8744296adab5","Type":"ContainerDied","Data":"d65602685d9964b6d2e61dd0fa3b67de60118a6db57e18faa4bbe785f5f6d99a"}
Oct 30 13:30:22.325368 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.325064    4544 scope.go:122] "RemoveContainer" containerID="bf75bf53c41a76242b88910b4caec1da97833da8caa038d0875209cc59ee73e5"
Oct 30 13:30:22.525836 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.525805    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf"]
Oct 30 13:30:22.546900 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.546869    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427"]
Oct 30 13:30:22.557316 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.557281    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-bcd76c-29889450-8v427"]
Oct 30 13:30:22.581506 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.581481    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c"
Oct 30 13:30:22.625276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.622449    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t"
Oct 30 13:30:22.626959 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.626936    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k"
Oct 30 13:30:22.684360 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.684326    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcbe2566-fac2-4912-9f80-9e3dbb313226-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbe2566-fac2-4912-9f80-9e3dbb313226-maas-api-service-ca\") pod \"dcbe2566-fac2-4912-9f80-9e3dbb313226\" (UID: \"dcbe2566-fac2-4912-9f80-9e3dbb313226\") "
Oct 30 13:30:22.684658 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.684635    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0625ba18-6017-481b-9bf2-983358ae2123-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0625ba18-6017-481b-9bf2-983358ae2123-maas-api-service-ca\") pod \"0625ba18-6017-481b-9bf2-983358ae2123\" (UID: \"0625ba18-6017-481b-9bf2-983358ae2123\") "
Oct 30 13:30:22.684841 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.684823    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6246b4d-a9f8-495d-8f1d-8f8f57fee223-kube-api-access-qn7xt\" (UniqueName: \"kubernetes.io/projected/d6246b4d-a9f8-495d-8f1d-8f8f57fee223-kube-api-access-qn7xt\") pod \"d6246b4d-a9f8-495d-8f1d-8f8f57fee223\" (UID: \"d6246b4d-a9f8-495d-8f1d-8f8f57fee223\") "
Oct 30 13:30:22.685016 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.685000    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcbe2566-fac2-4912-9f80-9e3dbb313226-kube-api-access-d84pc\" (UniqueName: \"kubernetes.io/projected/dcbe2566-fac2-4912-9f80-9e3dbb313226-kube-api-access-d84pc\") pod \"dcbe2566-fac2-4912-9f80-9e3dbb313226\" (UID: \"dcbe2566-fac2-4912-9f80-9e3dbb313226\") "
Oct 30 13:30:22.685260 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.685242    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6246b4d-a9f8-495d-8f1d-8f8f57fee223-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6246b4d-a9f8-495d-8f1d-8f8f57fee223-maas-api-service-ca\") pod \"d6246b4d-a9f8-495d-8f1d-8f8f57fee223\" (UID: \"d6246b4d-a9f8-495d-8f1d-8f8f57fee223\") "
Oct 30 13:30:22.685542 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.685523    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0625ba18-6017-481b-9bf2-983358ae2123-kube-api-access-75xwr\" (UniqueName: \"kubernetes.io/projected/0625ba18-6017-481b-9bf2-983358ae2123-kube-api-access-75xwr\") pod \"0625ba18-6017-481b-9bf2-983358ae2123\" (UID: \"0625ba18-6017-481b-9bf2-983358ae2123\") "
Oct 30 13:30:22.686553 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.686003    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6246b4d-a9f8-495d-8f1d-8f8f57fee223-maas-api-service-ca" pod "d6246b4d-a9f8-495d-8f1d-8f8f57fee223" (UID: "d6246b4d-a9f8-495d-8f1d-8f8f57fee223"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:22.686553 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.686344    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0625ba18-6017-481b-9bf2-983358ae2123-maas-api-service-ca" pod "0625ba18-6017-481b-9bf2-983358ae2123" (UID: "0625ba18-6017-481b-9bf2-983358ae2123"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:22.686553 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.686502    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0625ba18-6017-481b-9bf2-983358ae2123-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:30:22.686553 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.686527    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6246b4d-a9f8-495d-8f1d-8f8f57fee223-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:30:22.692670 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.692637    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcbe2566-fac2-4912-9f80-9e3dbb313226-maas-api-service-ca" pod "dcbe2566-fac2-4912-9f80-9e3dbb313226" (UID: "dcbe2566-fac2-4912-9f80-9e3dbb313226"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:22.700836 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.700799    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcbe2566-fac2-4912-9f80-9e3dbb313226-kube-api-access-d84pc" pod "dcbe2566-fac2-4912-9f80-9e3dbb313226" (UID: "dcbe2566-fac2-4912-9f80-9e3dbb313226"). InnerVolumeSpecName "kube-api-access-d84pc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:22.701850 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.701809    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6246b4d-a9f8-495d-8f1d-8f8f57fee223-kube-api-access-qn7xt" pod "d6246b4d-a9f8-495d-8f1d-8f8f57fee223" (UID: "d6246b4d-a9f8-495d-8f1d-8f8f57fee223"). InnerVolumeSpecName "kube-api-access-qn7xt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:22.702737 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.702694    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0625ba18-6017-481b-9bf2-983358ae2123-kube-api-access-75xwr" pod "0625ba18-6017-481b-9bf2-983358ae2123" (UID: "0625ba18-6017-481b-9bf2-983358ae2123"). InnerVolumeSpecName "kube-api-access-75xwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:22.749290 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.748895    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf"
Oct 30 13:30:22.787193 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.787153    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1a04d8b-24e3-4321-b21c-4e2d698ea5d2-kube-api-access-7ms76\" (UniqueName: \"kubernetes.io/projected/e1a04d8b-24e3-4321-b21c-4e2d698ea5d2-kube-api-access-7ms76\") pod \"e1a04d8b-24e3-4321-b21c-4e2d698ea5d2\" (UID: \"e1a04d8b-24e3-4321-b21c-4e2d698ea5d2\") "
Oct 30 13:30:22.787434 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.787277    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1a04d8b-24e3-4321-b21c-4e2d698ea5d2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1a04d8b-24e3-4321-b21c-4e2d698ea5d2-maas-api-service-ca\") pod \"e1a04d8b-24e3-4321-b21c-4e2d698ea5d2\" (UID: \"e1a04d8b-24e3-4321-b21c-4e2d698ea5d2\") "
Oct 30 13:30:22.787668 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.787654    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbe2566-fac2-4912-9f80-9e3dbb313226-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:30:22.787743 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.787668    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn7xt\" (UniqueName: \"kubernetes.io/projected/d6246b4d-a9f8-495d-8f1d-8f8f57fee223-kube-api-access-qn7xt\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:30:22.787743 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.787679    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d84pc\" (UniqueName: \"kubernetes.io/projected/dcbe2566-fac2-4912-9f80-9e3dbb313226-kube-api-access-d84pc\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:30:22.787743 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.787689    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75xwr\" (UniqueName: \"kubernetes.io/projected/0625ba18-6017-481b-9bf2-983358ae2123-kube-api-access-75xwr\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:30:22.787861 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.787732    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1a04d8b-24e3-4321-b21c-4e2d698ea5d2-maas-api-service-ca" pod "e1a04d8b-24e3-4321-b21c-4e2d698ea5d2" (UID: "e1a04d8b-24e3-4321-b21c-4e2d698ea5d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:22.789405 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.789381    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1a04d8b-24e3-4321-b21c-4e2d698ea5d2-kube-api-access-7ms76" pod "e1a04d8b-24e3-4321-b21c-4e2d698ea5d2" (UID: "e1a04d8b-24e3-4321-b21c-4e2d698ea5d2"). InnerVolumeSpecName "kube-api-access-7ms76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:22.889015 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.888932    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ms76\" (UniqueName: \"kubernetes.io/projected/e1a04d8b-24e3-4321-b21c-4e2d698ea5d2-kube-api-access-7ms76\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:30:22.889015 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.888968    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1a04d8b-24e3-4321-b21c-4e2d698ea5d2-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:30:22.992815 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:22.992781    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:23.098650 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:23.098619    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c9fb4e3-399f-4870-943c-8744296adab5" path="/var/lib/kubelet/pods/7c9fb4e3-399f-4870-943c-8744296adab5/volumes"
Oct 30 13:30:23.344518 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:23.344485    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t"
Oct 30 13:30:23.344518 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:23.344493    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t" event={"ID":"0625ba18-6017-481b-9bf2-983358ae2123","Type":"ContainerDied","Data":"eff60e5cde7437a1241093498a42730a2cf6a848d1d43b0117d9a215586d40ec"}
Oct 30 13:30:23.344773 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:23.344563    4544 scope.go:122] "RemoveContainer" containerID="50e962420e0f35f8fe4af191a31a19a869639ea36fac60aafc6efbc052f6d544"
Oct 30 13:30:23.346207 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:23.346186    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf"
Oct 30 13:30:23.346373 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:23.346209    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf" event={"ID":"e1a04d8b-24e3-4321-b21c-4e2d698ea5d2","Type":"ContainerDied","Data":"06974cba6dec89ff790439f423a2c016040b192d62bee2008bfffe32a5849195"}
Oct 30 13:30:23.347897 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:23.347878    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k"
Oct 30 13:30:23.348351 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:23.347883    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k" event={"ID":"dcbe2566-fac2-4912-9f80-9e3dbb313226","Type":"ContainerDied","Data":"34289c39f411fb402aa60e662537df45905a721f2ccb92ae73ef064bcab5cd00"}
Oct 30 13:30:23.349632 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:23.349617    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c"
Oct 30 13:30:23.349730 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:23.349644    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c" event={"ID":"d6246b4d-a9f8-495d-8f1d-8f8f57fee223","Type":"ContainerDied","Data":"7496738145ef41725ade3909d2b2aed0acbc35d53abdacd248a6a25b09fed29b"}
Oct 30 13:30:23.357113 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:23.357093    4544 scope.go:122] "RemoveContainer" containerID="9f72dc706fb697d04c39fee90ba3b154137036683a855a390781e3e4049e40aa"
Oct 30 13:30:23.368371 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:23.368298    4544 scope.go:122] "RemoveContainer" containerID="ad195918dff29215fa59c6da2aa9b5cc5dda7e7d7f5cf98b2cf7e9cc2447b483"
Oct 30 13:30:23.380566 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:23.380545    4544 scope.go:122] "RemoveContainer" containerID="6196b4086c2cabe8a8a8002049dfbd58813d80a326c75bce08cc90f3626f6e45"
Oct 30 13:30:23.400376 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:23.400345    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t"]
Oct 30 13:30:23.407840 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:23.407786    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e9d641-29889450-7h85t"]
Oct 30 13:30:23.443096 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:23.443060    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf"]
Oct 30 13:30:23.449876 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:23.449850    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b46c9-29889450-xk8wf"]
Oct 30 13:30:23.476875 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:23.476840    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c"]
Oct 30 13:30:23.486337 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:23.486310    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-67f56e-29889450-cvg9c"]
Oct 30 13:30:23.523840 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:23.523801    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k"]
Oct 30 13:30:23.531701 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:23.531669    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c2ce75-29889450-r555k"]
Oct 30 13:30:24.956894 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:24.956852    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:25.089416 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:25.089370    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0625ba18-6017-481b-9bf2-983358ae2123" path="/var/lib/kubelet/pods/0625ba18-6017-481b-9bf2-983358ae2123/volumes"
Oct 30 13:30:25.090008 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:25.089983    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6246b4d-a9f8-495d-8f1d-8f8f57fee223" path="/var/lib/kubelet/pods/d6246b4d-a9f8-495d-8f1d-8f8f57fee223/volumes"
Oct 30 13:30:25.090569 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:25.090545    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcbe2566-fac2-4912-9f80-9e3dbb313226" path="/var/lib/kubelet/pods/dcbe2566-fac2-4912-9f80-9e3dbb313226/volumes"
Oct 30 13:30:25.091380 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:25.091358    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1a04d8b-24e3-4321-b21c-4e2d698ea5d2" path="/var/lib/kubelet/pods/e1a04d8b-24e3-4321-b21c-4e2d698ea5d2/volumes"
Oct 30 13:30:26.835980 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:26.835937    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:28.540915 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:28.540865    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:28.765795 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:28.765738    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf"
Oct 30 13:30:28.779162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:28.779135    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf"
Oct 30 13:30:29.010236 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:30:29.010184    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:30:29.435118 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:29.435084    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf"
Oct 30 13:30:30.373494 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:30.373460    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:32.099488 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:32.099448    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:32.914469 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:32.914435    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:33.866522 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:33.866482    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:35.762923 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:35.762882    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:37.335861 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.335814    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k"]
Oct 30 13:30:37.337142 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337111    4544 cpu_manager.go:414] "containerMap: removing container" podUID="dcbe2566-fac2-4912-9f80-9e3dbb313226" containerName="cleanup"
Oct 30 13:30:37.337142 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337141    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcbe2566-fac2-4912-9f80-9e3dbb313226" containerName="cleanup"
Oct 30 13:30:37.337391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337158    4544 cpu_manager.go:414] "containerMap: removing container" podUID="7c9fb4e3-399f-4870-943c-8744296adab5" containerName="cleanup"
Oct 30 13:30:37.337391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337166    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c9fb4e3-399f-4870-943c-8744296adab5" containerName="cleanup"
Oct 30 13:30:37.337391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337176    4544 cpu_manager.go:414] "containerMap: removing container" podUID="7c9fb4e3-399f-4870-943c-8744296adab5" containerName="cleanup"
Oct 30 13:30:37.337391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337183    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c9fb4e3-399f-4870-943c-8744296adab5" containerName="cleanup"
Oct 30 13:30:37.337391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337204    4544 cpu_manager.go:414] "containerMap: removing container" podUID="ebce006c-8200-434b-99e8-5934eadad9a3" containerName="cleanup"
Oct 30 13:30:37.337391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337212    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebce006c-8200-434b-99e8-5934eadad9a3" containerName="cleanup"
Oct 30 13:30:37.337391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337240    4544 cpu_manager.go:414] "containerMap: removing container" podUID="d6246b4d-a9f8-495d-8f1d-8f8f57fee223" containerName="cleanup"
Oct 30 13:30:37.337391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337248    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6246b4d-a9f8-495d-8f1d-8f8f57fee223" containerName="cleanup"
Oct 30 13:30:37.337391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337270    4544 cpu_manager.go:414] "containerMap: removing container" podUID="0625ba18-6017-481b-9bf2-983358ae2123" containerName="cleanup"
Oct 30 13:30:37.337391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337276    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0625ba18-6017-481b-9bf2-983358ae2123" containerName="cleanup"
Oct 30 13:30:37.337391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337309    4544 cpu_manager.go:414] "containerMap: removing container" podUID="00b65da6-a93a-4b46-8a6f-912049acda4e" containerName="cleanup"
Oct 30 13:30:37.337391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337316    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00b65da6-a93a-4b46-8a6f-912049acda4e" containerName="cleanup"
Oct 30 13:30:37.337391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337349    4544 cpu_manager.go:414] "containerMap: removing container" podUID="00b65da6-a93a-4b46-8a6f-912049acda4e" containerName="cleanup"
Oct 30 13:30:37.337391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337358    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00b65da6-a93a-4b46-8a6f-912049acda4e" containerName="cleanup"
Oct 30 13:30:37.337391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337368    4544 cpu_manager.go:414] "containerMap: removing container" podUID="e1a04d8b-24e3-4321-b21c-4e2d698ea5d2" containerName="cleanup"
Oct 30 13:30:37.337391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337378    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1a04d8b-24e3-4321-b21c-4e2d698ea5d2" containerName="cleanup"
Oct 30 13:30:37.337391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337390    4544 cpu_manager.go:414] "containerMap: removing container" podUID="d6246b4d-a9f8-495d-8f1d-8f8f57fee223" containerName="cleanup"
Oct 30 13:30:37.337391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337397    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6246b4d-a9f8-495d-8f1d-8f8f57fee223" containerName="cleanup"
Oct 30 13:30:37.338391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337416    4544 cpu_manager.go:414] "containerMap: removing container" podUID="00b65da6-a93a-4b46-8a6f-912049acda4e" containerName="cleanup"
Oct 30 13:30:37.338391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337423    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00b65da6-a93a-4b46-8a6f-912049acda4e" containerName="cleanup"
Oct 30 13:30:37.338391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337432    4544 cpu_manager.go:414] "containerMap: removing container" podUID="e1a04d8b-24e3-4321-b21c-4e2d698ea5d2" containerName="cleanup"
Oct 30 13:30:37.338391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337438    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1a04d8b-24e3-4321-b21c-4e2d698ea5d2" containerName="cleanup"
Oct 30 13:30:37.338391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337448    4544 cpu_manager.go:414] "containerMap: removing container" podUID="ebce006c-8200-434b-99e8-5934eadad9a3" containerName="cleanup"
Oct 30 13:30:37.338391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337454    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebce006c-8200-434b-99e8-5934eadad9a3" containerName="cleanup"
Oct 30 13:30:37.338391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337466    4544 cpu_manager.go:414] "containerMap: removing container" podUID="dcbe2566-fac2-4912-9f80-9e3dbb313226" containerName="cleanup"
Oct 30 13:30:37.338391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337472    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcbe2566-fac2-4912-9f80-9e3dbb313226" containerName="cleanup"
Oct 30 13:30:37.338391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337491    4544 cpu_manager.go:414] "containerMap: removing container" podUID="0625ba18-6017-481b-9bf2-983358ae2123" containerName="cleanup"
Oct 30 13:30:37.338391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337498    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0625ba18-6017-481b-9bf2-983358ae2123" containerName="cleanup"
Oct 30 13:30:37.338391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337527    4544 cpu_manager.go:414] "containerMap: removing container" podUID="0625ba18-6017-481b-9bf2-983358ae2123" containerName="cleanup"
Oct 30 13:30:37.338391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337534    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0625ba18-6017-481b-9bf2-983358ae2123" containerName="cleanup"
Oct 30 13:30:37.338391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337732    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6246b4d-a9f8-495d-8f1d-8f8f57fee223" containerName="cleanup"
Oct 30 13:30:37.338391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337747    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="0625ba18-6017-481b-9bf2-983358ae2123" containerName="cleanup"
Oct 30 13:30:37.338391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337766    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c9fb4e3-399f-4870-943c-8744296adab5" containerName="cleanup"
Oct 30 13:30:37.338391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337775    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1a04d8b-24e3-4321-b21c-4e2d698ea5d2" containerName="cleanup"
Oct 30 13:30:37.338391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337790    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcbe2566-fac2-4912-9f80-9e3dbb313226" containerName="cleanup"
Oct 30 13:30:37.338391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337797    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c9fb4e3-399f-4870-943c-8744296adab5" containerName="cleanup"
Oct 30 13:30:37.338391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337805    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c9fb4e3-399f-4870-943c-8744296adab5" containerName="cleanup"
Oct 30 13:30:37.338391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337815    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebce006c-8200-434b-99e8-5934eadad9a3" containerName="cleanup"
Oct 30 13:30:37.338391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337824    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="0625ba18-6017-481b-9bf2-983358ae2123" containerName="cleanup"
Oct 30 13:30:37.338391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337835    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebce006c-8200-434b-99e8-5934eadad9a3" containerName="cleanup"
Oct 30 13:30:37.338391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337844    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="00b65da6-a93a-4b46-8a6f-912049acda4e" containerName="cleanup"
Oct 30 13:30:37.338391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337854    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcbe2566-fac2-4912-9f80-9e3dbb313226" containerName="cleanup"
Oct 30 13:30:37.338391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337864    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="00b65da6-a93a-4b46-8a6f-912049acda4e" containerName="cleanup"
Oct 30 13:30:37.338391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337875    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6246b4d-a9f8-495d-8f1d-8f8f57fee223" containerName="cleanup"
Oct 30 13:30:37.338391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337887    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="00b65da6-a93a-4b46-8a6f-912049acda4e" containerName="cleanup"
Oct 30 13:30:37.338391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337896    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1a04d8b-24e3-4321-b21c-4e2d698ea5d2" containerName="cleanup"
Oct 30 13:30:37.338391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.337905    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="0625ba18-6017-481b-9bf2-983358ae2123" containerName="cleanup"
Oct 30 13:30:37.342581 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.342553    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k"
Oct 30 13:30:37.360988 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.360953    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k"]
Oct 30 13:30:37.366368 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.366341    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-e9d641\"/\"default-dockercfg-rwklk\""
Oct 30 13:30:37.366368 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.366336    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-e9d641\"/\"openshift-service-ca.crt\""
Oct 30 13:30:37.366635 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.366405    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-e9d641\"/\"kube-root-ca.crt\""
Oct 30 13:30:37.370667 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.370631    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-e9d641\"/\"rate-test-model-e9d641-kserve-self-signed-certs\""
Oct 30 13:30:37.389660 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.389626    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:37.456712 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.456666    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbeeada2-dca6-47fb-8b83-559703fc24f4-tmp-dir\") pod \"rate-test-model-e9d641-kserve-7d4f8b4865-swg9k\" (UID: \"dbeeada2-dca6-47fb-8b83-559703fc24f4\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k"
Oct 30 13:30:37.456882 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.456837    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhxxq\" (UniqueName: \"kubernetes.io/projected/dbeeada2-dca6-47fb-8b83-559703fc24f4-kube-api-access-qhxxq\") pod \"rate-test-model-e9d641-kserve-7d4f8b4865-swg9k\" (UID: \"dbeeada2-dca6-47fb-8b83-559703fc24f4\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k"
Oct 30 13:30:37.456928 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.456894    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbeeada2-dca6-47fb-8b83-559703fc24f4-tls-certs\") pod \"rate-test-model-e9d641-kserve-7d4f8b4865-swg9k\" (UID: \"dbeeada2-dca6-47fb-8b83-559703fc24f4\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k"
Oct 30 13:30:37.456998 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.456983    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbeeada2-dca6-47fb-8b83-559703fc24f4-model-cache\") pod \"rate-test-model-e9d641-kserve-7d4f8b4865-swg9k\" (UID: \"dbeeada2-dca6-47fb-8b83-559703fc24f4\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k"
Oct 30 13:30:37.457108 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.457089    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbeeada2-dca6-47fb-8b83-559703fc24f4-dshm\") pod \"rate-test-model-e9d641-kserve-7d4f8b4865-swg9k\" (UID: \"dbeeada2-dca6-47fb-8b83-559703fc24f4\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k"
Oct 30 13:30:37.457214 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.457196    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbeeada2-dca6-47fb-8b83-559703fc24f4-home\") pod \"rate-test-model-e9d641-kserve-7d4f8b4865-swg9k\" (UID: \"dbeeada2-dca6-47fb-8b83-559703fc24f4\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k"
Oct 30 13:30:37.510683 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.510645    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:37.558833 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.558791    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbeeada2-dca6-47fb-8b83-559703fc24f4-home\") pod \"rate-test-model-e9d641-kserve-7d4f8b4865-swg9k\" (UID: \"dbeeada2-dca6-47fb-8b83-559703fc24f4\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k"
Oct 30 13:30:37.559045 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.558872    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbeeada2-dca6-47fb-8b83-559703fc24f4-tmp-dir\") pod \"rate-test-model-e9d641-kserve-7d4f8b4865-swg9k\" (UID: \"dbeeada2-dca6-47fb-8b83-559703fc24f4\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k"
Oct 30 13:30:37.559045 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.558951    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhxxq\" (UniqueName: \"kubernetes.io/projected/dbeeada2-dca6-47fb-8b83-559703fc24f4-kube-api-access-qhxxq\") pod \"rate-test-model-e9d641-kserve-7d4f8b4865-swg9k\" (UID: \"dbeeada2-dca6-47fb-8b83-559703fc24f4\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k"
Oct 30 13:30:37.559045 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.558993    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbeeada2-dca6-47fb-8b83-559703fc24f4-tls-certs\") pod \"rate-test-model-e9d641-kserve-7d4f8b4865-swg9k\" (UID: \"dbeeada2-dca6-47fb-8b83-559703fc24f4\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k"
Oct 30 13:30:37.559045 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.559022    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbeeada2-dca6-47fb-8b83-559703fc24f4-model-cache\") pod \"rate-test-model-e9d641-kserve-7d4f8b4865-swg9k\" (UID: \"dbeeada2-dca6-47fb-8b83-559703fc24f4\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k"
Oct 30 13:30:37.559345 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.559086    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbeeada2-dca6-47fb-8b83-559703fc24f4-dshm\") pod \"rate-test-model-e9d641-kserve-7d4f8b4865-swg9k\" (UID: \"dbeeada2-dca6-47fb-8b83-559703fc24f4\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k"
Oct 30 13:30:37.559441 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.559414    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbeeada2-dca6-47fb-8b83-559703fc24f4-tmp-dir\") pod \"rate-test-model-e9d641-kserve-7d4f8b4865-swg9k\" (UID: \"dbeeada2-dca6-47fb-8b83-559703fc24f4\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k"
Oct 30 13:30:37.559531 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.559436    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbeeada2-dca6-47fb-8b83-559703fc24f4-home\") pod \"rate-test-model-e9d641-kserve-7d4f8b4865-swg9k\" (UID: \"dbeeada2-dca6-47fb-8b83-559703fc24f4\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k"
Oct 30 13:30:37.559919 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.559638    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbeeada2-dca6-47fb-8b83-559703fc24f4-model-cache\") pod \"rate-test-model-e9d641-kserve-7d4f8b4865-swg9k\" (UID: \"dbeeada2-dca6-47fb-8b83-559703fc24f4\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k"
Oct 30 13:30:37.562089 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.562065    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbeeada2-dca6-47fb-8b83-559703fc24f4-dshm\") pod \"rate-test-model-e9d641-kserve-7d4f8b4865-swg9k\" (UID: \"dbeeada2-dca6-47fb-8b83-559703fc24f4\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k"
Oct 30 13:30:37.562393 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.562372    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbeeada2-dca6-47fb-8b83-559703fc24f4-tls-certs\") pod \"rate-test-model-e9d641-kserve-7d4f8b4865-swg9k\" (UID: \"dbeeada2-dca6-47fb-8b83-559703fc24f4\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k"
Oct 30 13:30:37.623178 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.623024    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhxxq\" (UniqueName: \"kubernetes.io/projected/dbeeada2-dca6-47fb-8b83-559703fc24f4-kube-api-access-qhxxq\") pod \"rate-test-model-e9d641-kserve-7d4f8b4865-swg9k\" (UID: \"dbeeada2-dca6-47fb-8b83-559703fc24f4\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k"
Oct 30 13:30:37.662302 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:37.662196    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k"
Oct 30 13:30:38.027066 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:38.026981    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k"]
Oct 30 13:30:38.028679 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:30:38.028651    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbeeada2_dca6_47fb_8b83_559703fc24f4.slice/crio-195816db74933cdd7f9e5cb47bfd2c253f23727b8f943a8c3cdf0c508e2463ac WatchSource:0}: Error finding container 195816db74933cdd7f9e5cb47bfd2c253f23727b8f943a8c3cdf0c508e2463ac: Status 404 returned error can't find the container with id 195816db74933cdd7f9e5cb47bfd2c253f23727b8f943a8c3cdf0c508e2463ac
Oct 30 13:30:38.524619 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:38.524570    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k" event={"ID":"dbeeada2-dca6-47fb-8b83-559703fc24f4","Type":"ContainerStarted","Data":"54ae21511e7f6b6c16b8d856dff9432420915b131fa2ce61f00ad3709828dd94"}
Oct 30 13:30:38.525112 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:38.524627    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k" event={"ID":"dbeeada2-dca6-47fb-8b83-559703fc24f4","Type":"ContainerStarted","Data":"195816db74933cdd7f9e5cb47bfd2c253f23727b8f943a8c3cdf0c508e2463ac"}
Oct 30 13:30:38.592326 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:38.592276    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k" podStartSLOduration=1.592259925 podStartE2EDuration="1.592259925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:38.591397903 +0000 UTC m=+2733.638531858" watchObservedRunningTime="2026-10-30 13:30:38.592259925 +0000 UTC m=+2733.639393878"
Oct 30 13:30:38.663359 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:38.663317    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k"
Oct 30 13:30:39.275959 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:39.275920    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:39.887426 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:39.887390    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:40.546877 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:40.546831    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:41.092585 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:41.092550    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:41.204941 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:41.204900    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:43.034094 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:43.034060    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:43.096962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:43.096928    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8f8991c-pnl9v"]
Oct 30 13:30:43.102116 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:43.102074    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8f8991c-pnl9v"]
Oct 30 13:30:44.889245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:44.886525    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:45.134265 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:45.134211    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:45.185519 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:45.185428    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02995b0b-f05b-40a7-b390-4c59835aad3b" path="/var/lib/kubelet/pods/02995b0b-f05b-40a7-b390-4c59835aad3b/volumes"
Oct 30 13:30:45.466129 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:45.465986    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:30:45.998499 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:45.998466    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:46.783877 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:46.783834    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:46.892311 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:46.892233    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n"]
Oct 30 13:30:46.893081 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:46.893066    4544 cpu_manager.go:414] "containerMap: removing container" podUID="e1a04d8b-24e3-4321-b21c-4e2d698ea5d2" containerName="cleanup"
Oct 30 13:30:46.893127 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:46.893083    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1a04d8b-24e3-4321-b21c-4e2d698ea5d2" containerName="cleanup"
Oct 30 13:30:46.893127 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:46.893092    4544 cpu_manager.go:414] "containerMap: removing container" podUID="ebce006c-8200-434b-99e8-5934eadad9a3" containerName="cleanup"
Oct 30 13:30:46.893127 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:46.893098    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebce006c-8200-434b-99e8-5934eadad9a3" containerName="cleanup"
Oct 30 13:30:46.893242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:46.893127    4544 cpu_manager.go:414] "containerMap: removing container" podUID="dcbe2566-fac2-4912-9f80-9e3dbb313226" containerName="cleanup"
Oct 30 13:30:46.893242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:46.893133    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcbe2566-fac2-4912-9f80-9e3dbb313226" containerName="cleanup"
Oct 30 13:30:46.893242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:46.893141    4544 cpu_manager.go:414] "containerMap: removing container" podUID="7c9fb4e3-399f-4870-943c-8744296adab5" containerName="cleanup"
Oct 30 13:30:46.893242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:46.893146    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c9fb4e3-399f-4870-943c-8744296adab5" containerName="cleanup"
Oct 30 13:30:46.893242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:46.893160    4544 cpu_manager.go:414] "containerMap: removing container" podUID="d6246b4d-a9f8-495d-8f1d-8f8f57fee223" containerName="cleanup"
Oct 30 13:30:46.893242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:46.893164    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6246b4d-a9f8-495d-8f1d-8f8f57fee223" containerName="cleanup"
Oct 30 13:30:46.893914 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:46.893302    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6246b4d-a9f8-495d-8f1d-8f8f57fee223" containerName="cleanup"
Oct 30 13:30:46.893914 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:46.893311    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1a04d8b-24e3-4321-b21c-4e2d698ea5d2" containerName="cleanup"
Oct 30 13:30:46.893914 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:46.893322    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebce006c-8200-434b-99e8-5934eadad9a3" containerName="cleanup"
Oct 30 13:30:46.893914 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:46.893332    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcbe2566-fac2-4912-9f80-9e3dbb313226" containerName="cleanup"
Oct 30 13:30:46.898759 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:46.898737    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n"
Oct 30 13:30:46.916245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:46.912248    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:46.939504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:46.939469    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3c1b28\""
Oct 30 13:30:46.955631 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:46.955599    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n"]
Oct 30 13:30:46.976143 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:46.976104    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k"]
Oct 30 13:30:46.981509 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:46.981475    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k"
Oct 30 13:30:46.999897 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:46.999860    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3c1b28-dockercfg-46x4z\""
Oct 30 13:30:47.003167 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:47.003135    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k"]
Oct 30 13:30:47.091342 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:47.091296    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a5b15b6-da23-43f0-9505-01ef161423fa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n\" (UID: \"6a5b15b6-da23-43f0-9505-01ef161423fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n"
Oct 30 13:30:47.104248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:47.091782    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da80418-8188-4b8c-87ca-5b887c86b70a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k\" (UID: \"5da80418-8188-4b8c-87ca-5b887c86b70a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k"
Oct 30 13:30:47.104793 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:47.104752    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk4l6\" (UniqueName: \"kubernetes.io/projected/6a5b15b6-da23-43f0-9505-01ef161423fa-kube-api-access-vk4l6\") pod \"payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n\" (UID: \"6a5b15b6-da23-43f0-9505-01ef161423fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n"
Oct 30 13:30:47.105039 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:47.105016    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvkf7\" (UniqueName: \"kubernetes.io/projected/5da80418-8188-4b8c-87ca-5b887c86b70a-kube-api-access-fvkf7\") pod \"payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k\" (UID: \"5da80418-8188-4b8c-87ca-5b887c86b70a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k"
Oct 30 13:30:47.256246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:47.231144    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da80418-8188-4b8c-87ca-5b887c86b70a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k\" (UID: \"5da80418-8188-4b8c-87ca-5b887c86b70a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k"
Oct 30 13:30:47.256777 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:47.256718    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk4l6\" (UniqueName: \"kubernetes.io/projected/6a5b15b6-da23-43f0-9505-01ef161423fa-kube-api-access-vk4l6\") pod \"payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n\" (UID: \"6a5b15b6-da23-43f0-9505-01ef161423fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n"
Oct 30 13:30:47.257032 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:47.257014    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvkf7\" (UniqueName: \"kubernetes.io/projected/5da80418-8188-4b8c-87ca-5b887c86b70a-kube-api-access-fvkf7\") pod \"payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k\" (UID: \"5da80418-8188-4b8c-87ca-5b887c86b70a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k"
Oct 30 13:30:47.257332 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:47.257313    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a5b15b6-da23-43f0-9505-01ef161423fa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n\" (UID: \"6a5b15b6-da23-43f0-9505-01ef161423fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n"
Oct 30 13:30:47.257612 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:47.257065    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da80418-8188-4b8c-87ca-5b887c86b70a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k\" (UID: \"5da80418-8188-4b8c-87ca-5b887c86b70a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k"
Oct 30 13:30:47.269250 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:47.263303    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a5b15b6-da23-43f0-9505-01ef161423fa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n\" (UID: \"6a5b15b6-da23-43f0-9505-01ef161423fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n"
Oct 30 13:30:47.302710 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:47.302682    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk4l6\" (UniqueName: \"kubernetes.io/projected/6a5b15b6-da23-43f0-9505-01ef161423fa-kube-api-access-vk4l6\") pod \"payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n\" (UID: \"6a5b15b6-da23-43f0-9505-01ef161423fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n"
Oct 30 13:30:47.305771 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:47.305740    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvkf7\" (UniqueName: \"kubernetes.io/projected/5da80418-8188-4b8c-87ca-5b887c86b70a-kube-api-access-fvkf7\") pod \"payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k\" (UID: \"5da80418-8188-4b8c-87ca-5b887c86b70a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k"
Oct 30 13:30:47.543127 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:47.543037    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n"
Oct 30 13:30:47.605884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:47.605855    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k"
Oct 30 13:30:47.664905 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:47.664877    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k"
Oct 30 13:30:47.682501 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:47.682456    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k"
Oct 30 13:30:47.795641 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:47.795544    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k"
Oct 30 13:30:47.887356 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:47.886748    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n"]
Oct 30 13:30:47.890311 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:30:47.890259    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5b15b6_da23_43f0_9505_01ef161423fa.slice/crio-fcf433e78c89fa4134be51db48eaa68e929111c8de7bf63534c8f1037d517c70 WatchSource:0}: Error finding container fcf433e78c89fa4134be51db48eaa68e929111c8de7bf63534c8f1037d517c70: Status 404 returned error can't find the container with id fcf433e78c89fa4134be51db48eaa68e929111c8de7bf63534c8f1037d517c70
Oct 30 13:30:48.047715 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:48.046315    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k"]
Oct 30 13:30:48.054652 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:30:48.054395    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5da80418_8188_4b8c_87ca_5b887c86b70a.slice/crio-de3113e4a3491ddec31542a19ea11af94714fb1584367b13f012bbe5572db451 WatchSource:0}: Error finding container de3113e4a3491ddec31542a19ea11af94714fb1584367b13f012bbe5572db451: Status 404 returned error can't find the container with id de3113e4a3491ddec31542a19ea11af94714fb1584367b13f012bbe5572db451
Oct 30 13:30:48.562364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:48.562328    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:48.742095 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:48.741921    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:48.798924 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:48.798889    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n" event={"ID":"6a5b15b6-da23-43f0-9505-01ef161423fa","Type":"ContainerStarted","Data":"abc0dfba40c342891c2b24a45001f892f8bf40c59c1f4ec0a87f5899990f09e6"}
Oct 30 13:30:48.799667 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:48.799645    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n"
Oct 30 13:30:48.799865 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:48.799846    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n" event={"ID":"6a5b15b6-da23-43f0-9505-01ef161423fa","Type":"ContainerStarted","Data":"fcf433e78c89fa4134be51db48eaa68e929111c8de7bf63534c8f1037d517c70"}
Oct 30 13:30:48.802389 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:48.802363    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k" event={"ID":"5da80418-8188-4b8c-87ca-5b887c86b70a","Type":"ContainerStarted","Data":"2b9794c2768da090508af02fd4ba522a6fc4cf56bf42cc94cec065e9da2c54f3"}
Oct 30 13:30:48.802562 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:48.802550    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k"
Oct 30 13:30:48.802673 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:48.802660    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k" event={"ID":"5da80418-8188-4b8c-87ca-5b887c86b70a","Type":"ContainerStarted","Data":"de3113e4a3491ddec31542a19ea11af94714fb1584367b13f012bbe5572db451"}
Oct 30 13:30:48.857370 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:48.857311    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n" podStartSLOduration=2.857296011 podStartE2EDuration="2.857296011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:48.854968347 +0000 UTC m=+2743.902102297" watchObservedRunningTime="2026-10-30 13:30:48.857296011 +0000 UTC m=+2743.904429964"
Oct 30 13:30:48.899627 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:48.899573    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k" podStartSLOduration=2.89956107 podStartE2EDuration="2.89956107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:48.89888577 +0000 UTC m=+2743.946019724" watchObservedRunningTime="2026-10-30 13:30:48.89956107 +0000 UTC m=+2743.946695024"
Oct 30 13:30:50.576092 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:50.576041    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:52.653435 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:52.653393    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:53.031915 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:53.031829    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8089916f-mz2lj"]
Oct 30 13:30:53.043707 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:53.043670    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8089916f-mz2lj"]
Oct 30 13:30:53.093406 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:53.093364    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9682bb73-16ea-4b2b-b4d2-e88469657da3" path="/var/lib/kubelet/pods/9682bb73-16ea-4b2b-b4d2-e88469657da3/volumes"
Oct 30 13:30:53.974183 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:53.974136    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:30:53.974650 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:53.974198    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7" podUID="820333af-678e-40c7-a1b7-24cab9591ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:30:54.639715 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:54.639669    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:54.766497 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:54.764438    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:54.856780 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:54.856745    4544 generic.go:356] "Generic (PLEG): container finished" podID="820333af-678e-40c7-a1b7-24cab9591ea2" containerID="2c60a75236026d1d1f0f915292164c351adeebb4e0101cae02351b5b402b083f" exitCode=0
Oct 30 13:30:54.856968 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:54.856819    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7" event={"ID":"820333af-678e-40c7-a1b7-24cab9591ea2","Type":"ContainerDied","Data":"2c60a75236026d1d1f0f915292164c351adeebb4e0101cae02351b5b402b083f"}
Oct 30 13:30:54.857420 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:54.857394    4544 scope.go:122] "RemoveContainer" containerID="2c60a75236026d1d1f0f915292164c351adeebb4e0101cae02351b5b402b083f"
Oct 30 13:30:54.859439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:54.859408    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:30:54.859517 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:54.859486    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:30:54.859556 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:54.859519    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:30:55.866626 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:55.866592    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58"]
Oct 30 13:30:55.870628 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:55.870592    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7" event={"ID":"820333af-678e-40c7-a1b7-24cab9591ea2","Type":"ContainerStarted","Data":"c705b45706084fb81e7b52018a64159c3b77147b07870ef42a989b8cf6e09fd2"}
Oct 30 13:30:55.991707 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:55.991670    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:30:55.991917 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:55.991800    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 13:30:55.991917 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:55.991855    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7" podUID="820333af-678e-40c7-a1b7-24cab9591ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 13:30:55.993371 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:55.993338    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:56.131178 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:56.131070    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:56.259074 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:56.259008    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:56.725246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:56.722818    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:56.989585 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:56.989485    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 13:30:56.989585 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:56.989544    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7" podUID="820333af-678e-40c7-a1b7-24cab9591ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 13:30:57.235013 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:57.234973    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:57.989497 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:57.989457    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 13:30:57.989712 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:57.989529    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7" podUID="820333af-678e-40c7-a1b7-24cab9591ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 13:30:58.773419 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:58.773378    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:30:58.989704 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:58.989666    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 13:30:58.990116 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:58.989739    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7" podUID="820333af-678e-40c7-a1b7-24cab9591ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 13:30:59.019570 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:30:59.019460    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:30:59.837567 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:59.837528    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k"
Oct 30 13:30:59.988987 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:59.988946    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 13:30:59.989178 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:30:59.989033    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7" podUID="820333af-678e-40c7-a1b7-24cab9591ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 13:31:00.839038 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:00.838997    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n"
Oct 30 13:31:00.859759 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:00.859694    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:00.866970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:00.866930    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58" podUID="7dddf370-d1a3-44af-bb88-aad50dccee04" containerName="payload-pre-processing" containerID="cri-o://7195dbca2013e4d9b52302ffda80742fe2152416d6da696a00c54c7fd0042f7a" gracePeriod=25
Oct 30 13:31:00.989277 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:00.989238    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 13:31:00.989548 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:00.989521    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7" podUID="820333af-678e-40c7-a1b7-24cab9591ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 13:31:01.305285 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:01.305208    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58"
Oct 30 13:31:01.379487 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:01.379401    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dddf370-d1a3-44af-bb88-aad50dccee04-kube-api-access-p67cc\" (UniqueName: \"kubernetes.io/projected/7dddf370-d1a3-44af-bb88-aad50dccee04-kube-api-access-p67cc\") pod \"7dddf370-d1a3-44af-bb88-aad50dccee04\" (UID: \"7dddf370-d1a3-44af-bb88-aad50dccee04\") "
Oct 30 13:31:01.379790 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:01.379757    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dddf370-d1a3-44af-bb88-aad50dccee04-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dddf370-d1a3-44af-bb88-aad50dccee04-plugins-config-volume\") pod \"7dddf370-d1a3-44af-bb88-aad50dccee04\" (UID: \"7dddf370-d1a3-44af-bb88-aad50dccee04\") "
Oct 30 13:31:01.381054 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:01.381024    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dddf370-d1a3-44af-bb88-aad50dccee04-plugins-config-volume" pod "7dddf370-d1a3-44af-bb88-aad50dccee04" (UID: "7dddf370-d1a3-44af-bb88-aad50dccee04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:31:01.382779 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:01.382753    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dddf370-d1a3-44af-bb88-aad50dccee04-kube-api-access-p67cc" pod "7dddf370-d1a3-44af-bb88-aad50dccee04" (UID: "7dddf370-d1a3-44af-bb88-aad50dccee04"). InnerVolumeSpecName "kube-api-access-p67cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:31:01.401369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:01.401334    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:01.481694 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:01.481655    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dddf370-d1a3-44af-bb88-aad50dccee04-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:31:01.481694 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:01.481687    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p67cc\" (UniqueName: \"kubernetes.io/projected/7dddf370-d1a3-44af-bb88-aad50dccee04-kube-api-access-p67cc\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:31:01.989765 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:01.989717    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 13:31:01.990291 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:01.989784    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7" podUID="820333af-678e-40c7-a1b7-24cab9591ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 13:31:01.999275 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:01.999244    4544 generic.go:356] "Generic (PLEG): container finished" podID="7dddf370-d1a3-44af-bb88-aad50dccee04" containerID="7195dbca2013e4d9b52302ffda80742fe2152416d6da696a00c54c7fd0042f7a" exitCode=0
Oct 30 13:31:01.999437 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:01.999286    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58"
Oct 30 13:31:01.999437 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:01.999324    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58" event={"ID":"7dddf370-d1a3-44af-bb88-aad50dccee04","Type":"ContainerDied","Data":"7195dbca2013e4d9b52302ffda80742fe2152416d6da696a00c54c7fd0042f7a"}
Oct 30 13:31:01.999437 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:01.999386    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58" event={"ID":"7dddf370-d1a3-44af-bb88-aad50dccee04","Type":"ContainerDied","Data":"d5e916ad52ff063ad2db47359c58968f62a61ba23dfd10b9d30ee8c97d1d3fca"}
Oct 30 13:31:01.999437 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:01.999416    4544 scope.go:122] "RemoveContainer" containerID="7195dbca2013e4d9b52302ffda80742fe2152416d6da696a00c54c7fd0042f7a"
Oct 30 13:31:02.011667 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:02.011644    4544 scope.go:122] "RemoveContainer" containerID="7195dbca2013e4d9b52302ffda80742fe2152416d6da696a00c54c7fd0042f7a"
Oct 30 13:31:02.011962 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:31:02.011944    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7195dbca2013e4d9b52302ffda80742fe2152416d6da696a00c54c7fd0042f7a\": container with ID starting with 7195dbca2013e4d9b52302ffda80742fe2152416d6da696a00c54c7fd0042f7a not found: ID does not exist" containerID="7195dbca2013e4d9b52302ffda80742fe2152416d6da696a00c54c7fd0042f7a"
Oct 30 13:31:02.012013 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:02.011970    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7195dbca2013e4d9b52302ffda80742fe2152416d6da696a00c54c7fd0042f7a"} err="failed to get container status \"7195dbca2013e4d9b52302ffda80742fe2152416d6da696a00c54c7fd0042f7a\": rpc error: code = NotFound desc = could not find container \"7195dbca2013e4d9b52302ffda80742fe2152416d6da696a00c54c7fd0042f7a\": container with ID starting with 7195dbca2013e4d9b52302ffda80742fe2152416d6da696a00c54c7fd0042f7a not found: ID does not exist"
Oct 30 13:31:02.093863 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:02.093830    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58"]
Oct 30 13:31:02.101588 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:02.101556    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58"]
Oct 30 13:31:02.920768 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:02.920732    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:02.994790 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:02.994757    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:31:02.995261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:02.995111    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:31:02.995978 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:02.995958    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:31:03.092120 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:03.092080    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dddf370-d1a3-44af-bb88-aad50dccee04" path="/var/lib/kubelet/pods/7dddf370-d1a3-44af-bb88-aad50dccee04/volumes"
Oct 30 13:31:04.688507 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:04.688476    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:06.518502 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:06.518461    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:08.287813 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:08.287775    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:09.441814 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:09.441780    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:10.167778 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:10.167740    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:12.276249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:12.250988    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4"]
Oct 30 13:31:12.332885 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:12.332842    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:12.777156 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:12.777115    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:14.145683 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:14.145646    4544 scope.go:122] "RemoveContainer" containerID="2f0850854403dfd8f2340d0c15cb5f1e9a3b056e803c8ecd4592f7d94c7c4d50"
Oct 30 13:31:14.156598 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:14.156572    4544 scope.go:122] "RemoveContainer" containerID="b9ea0c17d77c80c3b8d423bbb510ec4c4b8101feea9db52cfd2997fb3d61091f"
Oct 30 13:31:14.166987 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:14.166959    4544 scope.go:122] "RemoveContainer" containerID="f62d0b6957524f6007ae278eaa650a014668a974592c82a49ffae083a801bbe4"
Oct 30 13:31:14.177252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:14.177213    4544 scope.go:122] "RemoveContainer" containerID="0553ce613f24bae9f1de2bd92b19b0799e347cc0594cd26d4d244251e06575ef"
Oct 30 13:31:14.188380 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:14.188358    4544 scope.go:122] "RemoveContainer" containerID="bc0143cd624a916ae8df24651d937acd4a25f27d512f7b29425d16ef1c75ab90"
Oct 30 13:31:14.892992 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:14.892962    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:16.707967 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:16.707924    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:18.425703 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:18.425645    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:20.121334 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.121284    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:20.513538 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.513453    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:20.641189 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.641155    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"]
Oct 30 13:31:20.642709 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.642680    4544 cpu_manager.go:414] "containerMap: removing container" podUID="7dddf370-d1a3-44af-bb88-aad50dccee04" containerName="payload-pre-processing"
Oct 30 13:31:20.642883 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.642869    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dddf370-d1a3-44af-bb88-aad50dccee04" containerName="payload-pre-processing"
Oct 30 13:31:20.643282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.643264    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dddf370-d1a3-44af-bb88-aad50dccee04" containerName="payload-pre-processing"
Oct 30 13:31:20.669650 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.669521    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"]
Oct 30 13:31:20.669883 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.669628    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:31:20.687353 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.687307    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-627dfa-openshift-default-dockercfg-mnpmk\""
Oct 30 13:31:20.767847 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.767758    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/265b2410-a8b7-4d99-9cda-32565416a1d6-istiod-ca-cert\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:31:20.767847 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.767804    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-istio-data\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:31:20.768095 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.767928    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-istio-envoy\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:31:20.768095 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.767958    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/265b2410-a8b7-4d99-9cda-32565416a1d6-istio-token\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:31:20.768212 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.768135    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-credential-socket\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:31:20.768284 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.768237    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-workload-certs\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:31:20.768373 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.768356    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mgns\" (UniqueName: \"kubernetes.io/projected/265b2410-a8b7-4d99-9cda-32565416a1d6-kube-api-access-7mgns\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:31:20.768424 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.768391    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-workload-socket\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:31:20.768515 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.768498    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/265b2410-a8b7-4d99-9cda-32565416a1d6-istio-podinfo\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:31:20.869860 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.869823    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-istio-envoy\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:31:20.870078 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.869881    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/265b2410-a8b7-4d99-9cda-32565416a1d6-istio-token\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:31:20.870078 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.869986    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-credential-socket\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:31:20.870078 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.870033    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-workload-certs\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:31:20.870311 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.870127    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mgns\" (UniqueName: \"kubernetes.io/projected/265b2410-a8b7-4d99-9cda-32565416a1d6-kube-api-access-7mgns\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:31:20.870311 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.870159    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-workload-socket\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:31:20.870311 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.870240    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/265b2410-a8b7-4d99-9cda-32565416a1d6-istio-podinfo\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:31:20.870492 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.870388    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/265b2410-a8b7-4d99-9cda-32565416a1d6-istiod-ca-cert\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:31:20.870492 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.870422    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-istio-data\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:31:20.870601 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.870532    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-workload-certs\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:31:20.870601 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.870572    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-credential-socket\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:31:20.870717 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.870635    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-workload-socket\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:31:20.870870 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.870853    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-istio-data\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:31:20.871154 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.871129    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/265b2410-a8b7-4d99-9cda-32565416a1d6-istiod-ca-cert\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:31:20.872925 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.872897    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-istio-envoy\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:31:20.873208 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.873189    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/265b2410-a8b7-4d99-9cda-32565416a1d6-istio-podinfo\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:31:20.894355 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.894310    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/265b2410-a8b7-4d99-9cda-32565416a1d6-istio-token\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:31:20.894513 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.894394    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mgns\" (UniqueName: \"kubernetes.io/projected/265b2410-a8b7-4d99-9cda-32565416a1d6-kube-api-access-7mgns\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:31:20.999118 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:20.998806    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:31:21.353024 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:21.352995    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"]
Oct 30 13:31:21.355198 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:31:21.355167    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod265b2410_a8b7_4d99_9cda_32565416a1d6.slice/crio-635cc0bbccac4257950bb8cc39eae52967c1a6d9bf7c4e250890bdcb578b38a2 WatchSource:0}: Error finding container 635cc0bbccac4257950bb8cc39eae52967c1a6d9bf7c4e250890bdcb578b38a2: Status 404 returned error can't find the container with id 635cc0bbccac4257950bb8cc39eae52967c1a6d9bf7c4e250890bdcb578b38a2
Oct 30 13:31:21.359341 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:21.359295    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:31:21.359449 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:21.359397    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:31:21.359499 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:21.359448    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:31:21.558240 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:21.557684    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5" event={"ID":"265b2410-a8b7-4d99-9cda-32565416a1d6","Type":"ContainerStarted","Data":"da9dd40a34d315792924f330b490886541402d55bdab34f8150adce630fa519d"}
Oct 30 13:31:21.558240 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:21.557753    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5" event={"ID":"265b2410-a8b7-4d99-9cda-32565416a1d6","Type":"ContainerStarted","Data":"635cc0bbccac4257950bb8cc39eae52967c1a6d9bf7c4e250890bdcb578b38a2"}
Oct 30 13:31:21.607166 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:21.607045    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5" podStartSLOduration=1.607022868 podStartE2EDuration="1.607022868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:21.603020165 +0000 UTC m=+2776.650154119" watchObservedRunningTime="2026-10-30 13:31:21.607022868 +0000 UTC m=+2776.654156821"
Oct 30 13:31:21.904143 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:21.904093    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:21.999508 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:21.999474    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:31:22.568201 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:22.568169    4544 generic.go:356] "Generic (PLEG): container finished" podID="68fa1f90-25f8-442f-96ff-12e6a716db84" containerID="1b5b1864e485d3ba2317e500e7781d058ddcd9712fb6a5eaaf154303bec16dd3" exitCode=0
Oct 30 13:31:22.568734 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:22.568247    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt" event={"ID":"68fa1f90-25f8-442f-96ff-12e6a716db84","Type":"ContainerDied","Data":"1b5b1864e485d3ba2317e500e7781d058ddcd9712fb6a5eaaf154303bec16dd3"}
Oct 30 13:31:22.569153 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:22.569133    4544 scope.go:122] "RemoveContainer" containerID="1b5b1864e485d3ba2317e500e7781d058ddcd9712fb6a5eaaf154303bec16dd3"
Oct 30 13:31:22.570904 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:22.570877    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:31:22.570993 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:22.570948    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:31:22.570993 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:22.570980    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:31:23.001884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:23.001846    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body=
Oct 30 13:31:23.002093 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:23.001906    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5" podUID="265b2410-a8b7-4d99-9cda-32565416a1d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused"
Oct 30 13:31:23.579383 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:23.579346    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt" event={"ID":"68fa1f90-25f8-442f-96ff-12e6a716db84","Type":"ContainerStarted","Data":"67f408b3ed9734ca22c1321171a4edb0cb40399d8e76a9cffe9c96bf93cb9bc2"}
Oct 30 13:31:23.770180 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:23.770121    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:24.000347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:24.000246    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body=
Oct 30 13:31:24.000347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:24.000312    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5" podUID="265b2410-a8b7-4d99-9cda-32565416a1d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused"
Oct 30 13:31:24.002519 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:24.002488    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:31:24.002726 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:24.002536    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Oct 30 13:31:24.002840 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:24.002823    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt" podUID="68fa1f90-25f8-442f-96ff-12e6a716db84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Oct 30 13:31:25.000718 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:25.000677    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body=
Oct 30 13:31:25.001150 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:25.000736    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5" podUID="265b2410-a8b7-4d99-9cda-32565416a1d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused"
Oct 30 13:31:25.003143 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:25.003114    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Oct 30 13:31:25.003237 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:25.003167    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt" podUID="68fa1f90-25f8-442f-96ff-12e6a716db84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Oct 30 13:31:25.929767 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:25.929719    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:26.005241 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:26.004596    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Oct 30 13:31:26.005241 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:26.004662    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt" podUID="68fa1f90-25f8-442f-96ff-12e6a716db84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Oct 30 13:31:26.011401 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:26.007315    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body=
Oct 30 13:31:26.011401 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:26.007372    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5" podUID="265b2410-a8b7-4d99-9cda-32565416a1d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused"
Oct 30 13:31:26.999924 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:26.999887    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body=
Oct 30 13:31:27.000116 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:26.999968    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5" podUID="265b2410-a8b7-4d99-9cda-32565416a1d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused"
Oct 30 13:31:27.003308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:27.003283    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Oct 30 13:31:27.003374 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:27.003333    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt" podUID="68fa1f90-25f8-442f-96ff-12e6a716db84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Oct 30 13:31:27.504607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:27.504565    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4" podUID="7bb9d35b-2245-418e-b133-e9007ee9ba66" containerName="main" containerID="cri-o://250b243f654a6d11ffa191bf9a673b51475999fafa5356351fc940f198d89e05" gracePeriod=45
Oct 30 13:31:27.947458 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:27.947412    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:27.989938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:27.981610    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4"
Oct 30 13:31:28.004004 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:28.000712    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body=
Oct 30 13:31:28.004004 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:28.000774    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5" podUID="265b2410-a8b7-4d99-9cda-32565416a1d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused"
Oct 30 13:31:28.004004 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:28.002391    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Oct 30 13:31:28.004004 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:28.002445    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt" podUID="68fa1f90-25f8-442f-96ff-12e6a716db84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Oct 30 13:31:28.076751 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:28.076708    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bb9d35b-2245-418e-b133-e9007ee9ba66-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bb9d35b-2245-418e-b133-e9007ee9ba66-model-cache\") pod \"7bb9d35b-2245-418e-b133-e9007ee9ba66\" (UID: \"7bb9d35b-2245-418e-b133-e9007ee9ba66\") "
Oct 30 13:31:28.076963 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:28.076784    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bb9d35b-2245-418e-b133-e9007ee9ba66-kube-api-access-swdkv\" (UniqueName: \"kubernetes.io/projected/7bb9d35b-2245-418e-b133-e9007ee9ba66-kube-api-access-swdkv\") pod \"7bb9d35b-2245-418e-b133-e9007ee9ba66\" (UID: \"7bb9d35b-2245-418e-b133-e9007ee9ba66\") "
Oct 30 13:31:28.076963 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:28.076824    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bb9d35b-2245-418e-b133-e9007ee9ba66-home\" (UniqueName: \"kubernetes.io/empty-dir/7bb9d35b-2245-418e-b133-e9007ee9ba66-home\") pod \"7bb9d35b-2245-418e-b133-e9007ee9ba66\" (UID: \"7bb9d35b-2245-418e-b133-e9007ee9ba66\") "
Oct 30 13:31:28.076963 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:28.076936    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bb9d35b-2245-418e-b133-e9007ee9ba66-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bb9d35b-2245-418e-b133-e9007ee9ba66-tmp-dir\") pod \"7bb9d35b-2245-418e-b133-e9007ee9ba66\" (UID: \"7bb9d35b-2245-418e-b133-e9007ee9ba66\") "
Oct 30 13:31:28.077160 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:28.077103    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bb9d35b-2245-418e-b133-e9007ee9ba66-tls-certs\" (UniqueName: \"kubernetes.io/secret/7bb9d35b-2245-418e-b133-e9007ee9ba66-tls-certs\") pod \"7bb9d35b-2245-418e-b133-e9007ee9ba66\" (UID: \"7bb9d35b-2245-418e-b133-e9007ee9ba66\") "
Oct 30 13:31:28.078970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:28.077369    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bb9d35b-2245-418e-b133-e9007ee9ba66-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bb9d35b-2245-418e-b133-e9007ee9ba66-dshm\") pod \"7bb9d35b-2245-418e-b133-e9007ee9ba66\" (UID: \"7bb9d35b-2245-418e-b133-e9007ee9ba66\") "
Oct 30 13:31:28.078970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:28.077703    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb9d35b-2245-418e-b133-e9007ee9ba66-home" pod "7bb9d35b-2245-418e-b133-e9007ee9ba66" (UID: "7bb9d35b-2245-418e-b133-e9007ee9ba66"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:31:28.078970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:28.078124    4544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bb9d35b-2245-418e-b133-e9007ee9ba66-home\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:31:28.078970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:28.078383    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb9d35b-2245-418e-b133-e9007ee9ba66-model-cache" pod "7bb9d35b-2245-418e-b133-e9007ee9ba66" (UID: "7bb9d35b-2245-418e-b133-e9007ee9ba66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:31:28.078970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:28.078639    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb9d35b-2245-418e-b133-e9007ee9ba66-tmp-dir" pod "7bb9d35b-2245-418e-b133-e9007ee9ba66" (UID: "7bb9d35b-2245-418e-b133-e9007ee9ba66"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:31:28.086080 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:28.085468    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bb9d35b-2245-418e-b133-e9007ee9ba66-tls-certs" pod "7bb9d35b-2245-418e-b133-e9007ee9ba66" (UID: "7bb9d35b-2245-418e-b133-e9007ee9ba66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:31:28.086080 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:28.085916    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bb9d35b-2245-418e-b133-e9007ee9ba66-kube-api-access-swdkv" pod "7bb9d35b-2245-418e-b133-e9007ee9ba66" (UID: "7bb9d35b-2245-418e-b133-e9007ee9ba66"). InnerVolumeSpecName "kube-api-access-swdkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:31:28.094773 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:28.093273    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb9d35b-2245-418e-b133-e9007ee9ba66-dshm" pod "7bb9d35b-2245-418e-b133-e9007ee9ba66" (UID: "7bb9d35b-2245-418e-b133-e9007ee9ba66"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:31:28.180356 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:28.180319    4544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bb9d35b-2245-418e-b133-e9007ee9ba66-dshm\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:31:28.180356 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:28.180350    4544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bb9d35b-2245-418e-b133-e9007ee9ba66-model-cache\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:31:28.180647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:28.180368    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swdkv\" (UniqueName: \"kubernetes.io/projected/7bb9d35b-2245-418e-b133-e9007ee9ba66-kube-api-access-swdkv\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:31:28.180647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:28.180383    4544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bb9d35b-2245-418e-b133-e9007ee9ba66-tmp-dir\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:31:28.180647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:28.180399    4544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bb9d35b-2245-418e-b133-e9007ee9ba66-tls-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:31:28.647128 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:28.647095    4544 generic.go:356] "Generic (PLEG): container finished" podID="7bb9d35b-2245-418e-b133-e9007ee9ba66" containerID="250b243f654a6d11ffa191bf9a673b51475999fafa5356351fc940f198d89e05" exitCode=0
Oct 30 13:31:28.647601 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:28.647142    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4"
Oct 30 13:31:28.647601 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:28.647187    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4" event={"ID":"7bb9d35b-2245-418e-b133-e9007ee9ba66","Type":"ContainerDied","Data":"250b243f654a6d11ffa191bf9a673b51475999fafa5356351fc940f198d89e05"}
Oct 30 13:31:28.647601 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:28.647254    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4" event={"ID":"7bb9d35b-2245-418e-b133-e9007ee9ba66","Type":"ContainerDied","Data":"612786a172faef8a8425a7b91fa953115f0480aa3100cd5e081c59e5002743ff"}
Oct 30 13:31:28.647601 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:28.647283    4544 scope.go:122] "RemoveContainer" containerID="250b243f654a6d11ffa191bf9a673b51475999fafa5356351fc940f198d89e05"
Oct 30 13:31:28.658877 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:28.658849    4544 scope.go:122] "RemoveContainer" containerID="250b243f654a6d11ffa191bf9a673b51475999fafa5356351fc940f198d89e05"
Oct 30 13:31:28.659280 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:31:28.659254    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"250b243f654a6d11ffa191bf9a673b51475999fafa5356351fc940f198d89e05\": container with ID starting with 250b243f654a6d11ffa191bf9a673b51475999fafa5356351fc940f198d89e05 not found: ID does not exist" containerID="250b243f654a6d11ffa191bf9a673b51475999fafa5356351fc940f198d89e05"
Oct 30 13:31:28.659400 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:28.659287    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"250b243f654a6d11ffa191bf9a673b51475999fafa5356351fc940f198d89e05"} err="failed to get container status \"250b243f654a6d11ffa191bf9a673b51475999fafa5356351fc940f198d89e05\": rpc error: code = NotFound desc = could not find container \"250b243f654a6d11ffa191bf9a673b51475999fafa5356351fc940f198d89e05\": container with ID starting with 250b243f654a6d11ffa191bf9a673b51475999fafa5356351fc940f198d89e05 not found: ID does not exist"
Oct 30 13:31:28.703763 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:28.703722    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4"]
Oct 30 13:31:28.738969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:28.738928    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-51288b/auth-test-model-51288b-kserve-5844f5b57c-fm8h4"]
Oct 30 13:31:29.002914 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:29.002742    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Oct 30 13:31:29.002914 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:29.002803    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt" podUID="68fa1f90-25f8-442f-96ff-12e6a716db84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Oct 30 13:31:29.005992 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:29.005964    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:31:29.006841 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:29.006820    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:31:29.007761 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:29.007739    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:31:29.011473 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:31:29.011445    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:31:29.089897 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:29.089854    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bb9d35b-2245-418e-b133-e9007ee9ba66" path="/var/lib/kubelet/pods/7bb9d35b-2245-418e-b133-e9007ee9ba66/volumes"
Oct 30 13:31:29.856577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:29.856536    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:30.008774 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:30.008741    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:31:30.009265 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:30.009206    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:31:30.010147 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:30.010130    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:31:31.527845 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:31.527794    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:33.232289 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:33.232253    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:33.359270 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:33.358975    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:35.312087 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:35.312036    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:36.340524 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:36.340485    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:36.503349 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:36.503308    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:37.156622 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:37.156585    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:39.119259 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:39.119207    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:39.516313 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:39.516209    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:40.693544 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:40.693496    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:40.881052 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:40.880930    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:41.117183 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:41.117150    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:41.938025 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:41.937947    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:42.008751 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.008710    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn"]
Oct 30 13:31:42.040984 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.040944    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w"]
Oct 30 13:31:42.070970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.070936    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:42.201068 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.200979    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:42.310049 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.310005    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:42.492747 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.492323    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"]
Oct 30 13:31:42.493877 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.493849    4544 cpu_manager.go:414] "containerMap: removing container" podUID="7bb9d35b-2245-418e-b133-e9007ee9ba66" containerName="main"
Oct 30 13:31:42.493877 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.493876    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bb9d35b-2245-418e-b133-e9007ee9ba66" containerName="main"
Oct 30 13:31:42.494078 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.494041    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bb9d35b-2245-418e-b133-e9007ee9ba66" containerName="main"
Oct 30 13:31:42.498276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.498245    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:31:42.533439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.533407    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-c91909-openshift-default-dockercfg-zz66t\""
Oct 30 13:31:42.539463 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.539426    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"]
Oct 30 13:31:42.574204 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.574149    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:42.579963 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.579933    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istio-data\") pod \"e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") " pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:31:42.580276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.580252    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-workload-socket\") pod \"e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") " pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:31:42.580377 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.580309    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-workload-certs\") pod \"e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") " pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:31:42.580465 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.580449    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nsbj\" (UniqueName: \"kubernetes.io/projected/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-kube-api-access-4nsbj\") pod \"e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") " pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:31:42.580563 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.580548    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istio-podinfo\") pod \"e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") " pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:31:42.580633 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.580598    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istiod-ca-cert\") pod \"e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") " pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:31:42.580723 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.580710    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istio-envoy\") pod \"e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") " pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:31:42.580780 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.580742    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istio-token\") pod \"e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") " pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:31:42.580853 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.580838    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-credential-socket\") pod \"e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") " pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:31:42.682635 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.682599    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istio-podinfo\") pod \"e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") " pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:31:42.682852 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.682650    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istiod-ca-cert\") pod \"e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") " pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:31:42.682852 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.682722    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istio-envoy\") pod \"e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") " pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:31:42.682852 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.682741    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istio-token\") pod \"e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") " pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:31:42.682852 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.682793    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-credential-socket\") pod \"e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") " pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:31:42.682852 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.682826    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istio-data\") pod \"e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") " pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:31:42.683126 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.682882    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-workload-socket\") pod \"e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") " pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:31:42.683126 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.682918    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-workload-certs\") pod \"e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") " pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:31:42.683126 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.682969    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nsbj\" (UniqueName: \"kubernetes.io/projected/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-kube-api-access-4nsbj\") pod \"e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") " pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:31:42.683360 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.683309    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-credential-socket\") pod \"e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") " pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:31:42.683360 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.683329    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-workload-socket\") pod \"e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") " pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:31:42.683603 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.683577    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-workload-certs\") pod \"e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") " pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:31:42.683666 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.683650    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istiod-ca-cert\") pod \"e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") " pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:31:42.683749 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.683727    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istio-data\") pod \"e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") " pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:31:42.685280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.685256    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istio-envoy\") pod \"e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") " pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:31:42.685512 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.685490    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istio-podinfo\") pod \"e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") " pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:31:42.706836 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.706799    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istio-token\") pod \"e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") " pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:31:42.706983 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.706891    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nsbj\" (UniqueName: \"kubernetes.io/projected/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-kube-api-access-4nsbj\") pod \"e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") " pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:31:42.756701 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.756601    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:42.846245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:42.839525    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:31:43.366991 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:43.366953    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"]
Oct 30 13:31:43.368638 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:31:43.368608    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4400fc6b_5d1c_4f3f_93be_7ef8540e8d79.slice/crio-74fe47361d6fb0176fbcd9d3f2931e1d4540b6ee551005f9538872d914e3d488 WatchSource:0}: Error finding container 74fe47361d6fb0176fbcd9d3f2931e1d4540b6ee551005f9538872d914e3d488: Status 404 returned error can't find the container with id 74fe47361d6fb0176fbcd9d3f2931e1d4540b6ee551005f9538872d914e3d488
Oct 30 13:31:43.371880 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:43.371837    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:31:43.371998 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:43.371957    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:31:43.372071 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:43.372028    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:31:43.909549 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:43.909509    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:43.930790 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:43.930752    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz" event={"ID":"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79","Type":"ContainerStarted","Data":"d6d4b0cad2e96d7bf655fc4d0c2d3d91862bd6ff7b91b1be789b66aeb860b672"}
Oct 30 13:31:43.930976 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:43.930816    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz" event={"ID":"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79","Type":"ContainerStarted","Data":"74fe47361d6fb0176fbcd9d3f2931e1d4540b6ee551005f9538872d914e3d488"}
Oct 30 13:31:43.995703 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:43.995646    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz" podStartSLOduration=1.995626342 podStartE2EDuration="1.995626342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:43.983941094 +0000 UTC m=+2799.031075049" watchObservedRunningTime="2026-10-30 13:31:43.995626342 +0000 UTC m=+2799.042760296"
Oct 30 13:31:44.840330 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:44.840288    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:31:44.846750 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:44.846719    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:31:44.940235 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:44.940187    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:31:44.941140 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:44.941118    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:31:46.085158 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:46.085111    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:46.442321 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:46.442237    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:46.562568 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:46.562535    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:47.010298 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:47.010259    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn" podUID="39386eb4-a783-44ca-b8f0-ecf162582aee" containerName="payload-processing" containerID="cri-o://3f1f6210850e23bc3f4e9c6a26c8db6c40fb6dc0daf0b87fd65c5f0721b2bd26" gracePeriod=25
Oct 30 13:31:47.041877 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:47.041842    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w" podUID="25c89b7d-ea4e-425b-b35f-9bc061ecffdb" containerName="payload-pre-processing" containerID="cri-o://ba362cd24f4fb79cbbf3d03979d7cfe699d54e7c893155241f5bfa41b4523a2b" gracePeriod=25
Oct 30 13:31:47.368499 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:47.368475    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn"
Oct 30 13:31:47.372188 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:47.372160    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w"
Oct 30 13:31:47.376968 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:47.376701    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:47.414436 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:47.414401    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25c89b7d-ea4e-425b-b35f-9bc061ecffdb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c89b7d-ea4e-425b-b35f-9bc061ecffdb-plugins-config-volume\") pod \"25c89b7d-ea4e-425b-b35f-9bc061ecffdb\" (UID: \"25c89b7d-ea4e-425b-b35f-9bc061ecffdb\") "
Oct 30 13:31:47.414683 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:47.414489    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39386eb4-a783-44ca-b8f0-ecf162582aee-kube-api-access-swxzr\" (UniqueName: \"kubernetes.io/projected/39386eb4-a783-44ca-b8f0-ecf162582aee-kube-api-access-swxzr\") pod \"39386eb4-a783-44ca-b8f0-ecf162582aee\" (UID: \"39386eb4-a783-44ca-b8f0-ecf162582aee\") "
Oct 30 13:31:47.414683 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:47.414551    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39386eb4-a783-44ca-b8f0-ecf162582aee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39386eb4-a783-44ca-b8f0-ecf162582aee-plugins-config-volume\") pod \"39386eb4-a783-44ca-b8f0-ecf162582aee\" (UID: \"39386eb4-a783-44ca-b8f0-ecf162582aee\") "
Oct 30 13:31:47.414683 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:47.414648    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25c89b7d-ea4e-425b-b35f-9bc061ecffdb-kube-api-access-vgxf8\" (UniqueName: \"kubernetes.io/projected/25c89b7d-ea4e-425b-b35f-9bc061ecffdb-kube-api-access-vgxf8\") pod \"25c89b7d-ea4e-425b-b35f-9bc061ecffdb\" (UID: \"25c89b7d-ea4e-425b-b35f-9bc061ecffdb\") "
Oct 30 13:31:47.416065 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:47.416036    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39386eb4-a783-44ca-b8f0-ecf162582aee-plugins-config-volume" pod "39386eb4-a783-44ca-b8f0-ecf162582aee" (UID: "39386eb4-a783-44ca-b8f0-ecf162582aee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:31:47.416503 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:47.416477    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25c89b7d-ea4e-425b-b35f-9bc061ecffdb-plugins-config-volume" pod "25c89b7d-ea4e-425b-b35f-9bc061ecffdb" (UID: "25c89b7d-ea4e-425b-b35f-9bc061ecffdb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:31:47.417688 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:47.417663    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25c89b7d-ea4e-425b-b35f-9bc061ecffdb-kube-api-access-vgxf8" pod "25c89b7d-ea4e-425b-b35f-9bc061ecffdb" (UID: "25c89b7d-ea4e-425b-b35f-9bc061ecffdb"). InnerVolumeSpecName "kube-api-access-vgxf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:31:47.418263 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:47.418237    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39386eb4-a783-44ca-b8f0-ecf162582aee-kube-api-access-swxzr" pod "39386eb4-a783-44ca-b8f0-ecf162582aee" (UID: "39386eb4-a783-44ca-b8f0-ecf162582aee"). InnerVolumeSpecName "kube-api-access-swxzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:31:47.517477 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:47.517433    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgxf8\" (UniqueName: \"kubernetes.io/projected/25c89b7d-ea4e-425b-b35f-9bc061ecffdb-kube-api-access-vgxf8\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:31:47.517477 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:47.517478    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c89b7d-ea4e-425b-b35f-9bc061ecffdb-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:31:47.517699 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:47.517497    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swxzr\" (UniqueName: \"kubernetes.io/projected/39386eb4-a783-44ca-b8f0-ecf162582aee-kube-api-access-swxzr\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:31:47.517699 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:47.517512    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39386eb4-a783-44ca-b8f0-ecf162582aee-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:31:47.964481 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:47.964450    4544 generic.go:356] "Generic (PLEG): container finished" podID="39386eb4-a783-44ca-b8f0-ecf162582aee" containerID="3f1f6210850e23bc3f4e9c6a26c8db6c40fb6dc0daf0b87fd65c5f0721b2bd26" exitCode=0
Oct 30 13:31:47.964702 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:47.964520    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn"
Oct 30 13:31:47.964702 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:47.964533    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn" event={"ID":"39386eb4-a783-44ca-b8f0-ecf162582aee","Type":"ContainerDied","Data":"3f1f6210850e23bc3f4e9c6a26c8db6c40fb6dc0daf0b87fd65c5f0721b2bd26"}
Oct 30 13:31:47.964702 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:47.964613    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn" event={"ID":"39386eb4-a783-44ca-b8f0-ecf162582aee","Type":"ContainerDied","Data":"f6467cea6a07e6718b94026761560099f38c4121eb0caf745afb1d561708ec58"}
Oct 30 13:31:47.964702 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:47.964648    4544 scope.go:122] "RemoveContainer" containerID="3f1f6210850e23bc3f4e9c6a26c8db6c40fb6dc0daf0b87fd65c5f0721b2bd26"
Oct 30 13:31:47.966191 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:47.966161    4544 generic.go:356] "Generic (PLEG): container finished" podID="25c89b7d-ea4e-425b-b35f-9bc061ecffdb" containerID="ba362cd24f4fb79cbbf3d03979d7cfe699d54e7c893155241f5bfa41b4523a2b" exitCode=0
Oct 30 13:31:47.966346 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:47.966203    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w" event={"ID":"25c89b7d-ea4e-425b-b35f-9bc061ecffdb","Type":"ContainerDied","Data":"ba362cd24f4fb79cbbf3d03979d7cfe699d54e7c893155241f5bfa41b4523a2b"}
Oct 30 13:31:47.966346 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:47.966263    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w"
Oct 30 13:31:47.966346 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:47.966280    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w" event={"ID":"25c89b7d-ea4e-425b-b35f-9bc061ecffdb","Type":"ContainerDied","Data":"b4c0e72f61379e71e2cfa592079d9f781dbb818658014fd9d8b64566e542a638"}
Oct 30 13:31:47.976568 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:47.976553    4544 scope.go:122] "RemoveContainer" containerID="3f1f6210850e23bc3f4e9c6a26c8db6c40fb6dc0daf0b87fd65c5f0721b2bd26"
Oct 30 13:31:47.976851 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:31:47.976832    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f1f6210850e23bc3f4e9c6a26c8db6c40fb6dc0daf0b87fd65c5f0721b2bd26\": container with ID starting with 3f1f6210850e23bc3f4e9c6a26c8db6c40fb6dc0daf0b87fd65c5f0721b2bd26 not found: ID does not exist" containerID="3f1f6210850e23bc3f4e9c6a26c8db6c40fb6dc0daf0b87fd65c5f0721b2bd26"
Oct 30 13:31:47.976940 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:47.976858    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f1f6210850e23bc3f4e9c6a26c8db6c40fb6dc0daf0b87fd65c5f0721b2bd26"} err="failed to get container status \"3f1f6210850e23bc3f4e9c6a26c8db6c40fb6dc0daf0b87fd65c5f0721b2bd26\": rpc error: code = NotFound desc = could not find container \"3f1f6210850e23bc3f4e9c6a26c8db6c40fb6dc0daf0b87fd65c5f0721b2bd26\": container with ID starting with 3f1f6210850e23bc3f4e9c6a26c8db6c40fb6dc0daf0b87fd65c5f0721b2bd26 not found: ID does not exist"
Oct 30 13:31:47.976940 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:47.976873    4544 scope.go:122] "RemoveContainer" containerID="ba362cd24f4fb79cbbf3d03979d7cfe699d54e7c893155241f5bfa41b4523a2b"
Oct 30 13:31:47.987731 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:47.987713    4544 scope.go:122] "RemoveContainer" containerID="ba362cd24f4fb79cbbf3d03979d7cfe699d54e7c893155241f5bfa41b4523a2b"
Oct 30 13:31:47.988046 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:31:47.988025    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba362cd24f4fb79cbbf3d03979d7cfe699d54e7c893155241f5bfa41b4523a2b\": container with ID starting with ba362cd24f4fb79cbbf3d03979d7cfe699d54e7c893155241f5bfa41b4523a2b not found: ID does not exist" containerID="ba362cd24f4fb79cbbf3d03979d7cfe699d54e7c893155241f5bfa41b4523a2b"
Oct 30 13:31:47.988115 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:47.988057    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba362cd24f4fb79cbbf3d03979d7cfe699d54e7c893155241f5bfa41b4523a2b"} err="failed to get container status \"ba362cd24f4fb79cbbf3d03979d7cfe699d54e7c893155241f5bfa41b4523a2b\": rpc error: code = NotFound desc = could not find container \"ba362cd24f4fb79cbbf3d03979d7cfe699d54e7c893155241f5bfa41b4523a2b\": container with ID starting with ba362cd24f4fb79cbbf3d03979d7cfe699d54e7c893155241f5bfa41b4523a2b not found: ID does not exist"
Oct 30 13:31:48.028982 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:48.028427    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn"]
Oct 30 13:31:48.033709 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:48.033682    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn"]
Oct 30 13:31:48.061290 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:48.061215    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:48.068486 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:48.068458    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w"]
Oct 30 13:31:48.075718 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:48.075695    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w"]
Oct 30 13:31:48.178669 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:48.178636    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:48.317957 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:48.317872    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:48.374981 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:48.374941    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r"]
Oct 30 13:31:48.376011 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:48.375978    4544 cpu_manager.go:414] "containerMap: removing container" podUID="25c89b7d-ea4e-425b-b35f-9bc061ecffdb" containerName="payload-pre-processing"
Oct 30 13:31:48.376011 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:48.376001    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25c89b7d-ea4e-425b-b35f-9bc061ecffdb" containerName="payload-pre-processing"
Oct 30 13:31:48.376198 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:48.376023    4544 cpu_manager.go:414] "containerMap: removing container" podUID="39386eb4-a783-44ca-b8f0-ecf162582aee" containerName="payload-processing"
Oct 30 13:31:48.376198 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:48.376032    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39386eb4-a783-44ca-b8f0-ecf162582aee" containerName="payload-processing"
Oct 30 13:31:48.376325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:48.376264    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="25c89b7d-ea4e-425b-b35f-9bc061ecffdb" containerName="payload-pre-processing"
Oct 30 13:31:48.376325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:48.376277    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="39386eb4-a783-44ca-b8f0-ecf162582aee" containerName="payload-processing"
Oct 30 13:31:48.382412 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:48.382386    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r"
Oct 30 13:31:48.397264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:48.397209    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c91909\""
Oct 30 13:31:48.397426 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:48.397279    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c91909-dockercfg-zhv6c\""
Oct 30 13:31:48.401792 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:48.401755    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r"]
Oct 30 13:31:48.457320 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:48.435407    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knffk\" (UniqueName: \"kubernetes.io/projected/be7872c4-5356-451c-bc78-767629d8a6ea-kube-api-access-knffk\") pod \"payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r\" (UID: \"be7872c4-5356-451c-bc78-767629d8a6ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r"
Oct 30 13:31:48.457320 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:48.435489    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be7872c4-5356-451c-bc78-767629d8a6ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r\" (UID: \"be7872c4-5356-451c-bc78-767629d8a6ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r"
Oct 30 13:31:48.538243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:48.536765    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knffk\" (UniqueName: \"kubernetes.io/projected/be7872c4-5356-451c-bc78-767629d8a6ea-kube-api-access-knffk\") pod \"payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r\" (UID: \"be7872c4-5356-451c-bc78-767629d8a6ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r"
Oct 30 13:31:48.538243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:48.536837    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be7872c4-5356-451c-bc78-767629d8a6ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r\" (UID: \"be7872c4-5356-451c-bc78-767629d8a6ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r"
Oct 30 13:31:48.538243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:48.537819    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be7872c4-5356-451c-bc78-767629d8a6ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r\" (UID: \"be7872c4-5356-451c-bc78-767629d8a6ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r"
Oct 30 13:31:48.700305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:48.700240    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s"]
Oct 30 13:31:48.707502 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:48.707466    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s"
Oct 30 13:31:48.709448 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:48.709415    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knffk\" (UniqueName: \"kubernetes.io/projected/be7872c4-5356-451c-bc78-767629d8a6ea-kube-api-access-knffk\") pod \"payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r\" (UID: \"be7872c4-5356-451c-bc78-767629d8a6ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r"
Oct 30 13:31:48.716318 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:48.716118    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s"]
Oct 30 13:31:48.845272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:48.845198    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdsmm\" (UniqueName: \"kubernetes.io/projected/53bc0c74-61e1-4713-94e4-bab1b71e8fe8-kube-api-access-kdsmm\") pod \"payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s\" (UID: \"53bc0c74-61e1-4713-94e4-bab1b71e8fe8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s"
Oct 30 13:31:48.845451 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:48.845407    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53bc0c74-61e1-4713-94e4-bab1b71e8fe8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s\" (UID: \"53bc0c74-61e1-4713-94e4-bab1b71e8fe8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s"
Oct 30 13:31:48.946933 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:48.946897    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53bc0c74-61e1-4713-94e4-bab1b71e8fe8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s\" (UID: \"53bc0c74-61e1-4713-94e4-bab1b71e8fe8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s"
Oct 30 13:31:48.947150 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:48.947100    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdsmm\" (UniqueName: \"kubernetes.io/projected/53bc0c74-61e1-4713-94e4-bab1b71e8fe8-kube-api-access-kdsmm\") pod \"payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s\" (UID: \"53bc0c74-61e1-4713-94e4-bab1b71e8fe8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s"
Oct 30 13:31:48.947704 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:48.947678    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53bc0c74-61e1-4713-94e4-bab1b71e8fe8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s\" (UID: \"53bc0c74-61e1-4713-94e4-bab1b71e8fe8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s"
Oct 30 13:31:48.970614 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:48.970544    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdsmm\" (UniqueName: \"kubernetes.io/projected/53bc0c74-61e1-4713-94e4-bab1b71e8fe8-kube-api-access-kdsmm\") pod \"payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s\" (UID: \"53bc0c74-61e1-4713-94e4-bab1b71e8fe8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s"
Oct 30 13:31:48.998628 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:48.998598    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r"
Oct 30 13:31:49.037338 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:49.037307    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s"
Oct 30 13:31:49.103784 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:49.102063    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25c89b7d-ea4e-425b-b35f-9bc061ecffdb" path="/var/lib/kubelet/pods/25c89b7d-ea4e-425b-b35f-9bc061ecffdb/volumes"
Oct 30 13:31:49.103784 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:49.102819    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39386eb4-a783-44ca-b8f0-ecf162582aee" path="/var/lib/kubelet/pods/39386eb4-a783-44ca-b8f0-ecf162582aee/volumes"
Oct 30 13:31:49.318961 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:49.318931    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r"]
Oct 30 13:31:49.322610 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:31:49.322581    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe7872c4_5356_451c_bc78_767629d8a6ea.slice/crio-f3cfa0bcbe1cbcf79e0757fecfe33763a937f10874ec97d8e88fd932f1cbcc60 WatchSource:0}: Error finding container f3cfa0bcbe1cbcf79e0757fecfe33763a937f10874ec97d8e88fd932f1cbcc60: Status 404 returned error can't find the container with id f3cfa0bcbe1cbcf79e0757fecfe33763a937f10874ec97d8e88fd932f1cbcc60
Oct 30 13:31:49.367980 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:49.367954    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s"]
Oct 30 13:31:49.373840 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:31:49.373811    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53bc0c74_61e1_4713_94e4_bab1b71e8fe8.slice/crio-24b447179757ea00ca1bbd87b6e720ce1d1e9fdeda17bca7c9c4ba34f117ac39 WatchSource:0}: Error finding container 24b447179757ea00ca1bbd87b6e720ce1d1e9fdeda17bca7c9c4ba34f117ac39: Status 404 returned error can't find the container with id 24b447179757ea00ca1bbd87b6e720ce1d1e9fdeda17bca7c9c4ba34f117ac39
Oct 30 13:31:49.939503 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:49.939474    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:49.986322 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:49.986282    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s" event={"ID":"53bc0c74-61e1-4713-94e4-bab1b71e8fe8","Type":"ContainerStarted","Data":"53fc884765100f3bdfbbdae43c4fbd440cc95da1da0a2ac1dedd3c85ee7cd34e"}
Oct 30 13:31:49.986523 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:49.986345    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s" event={"ID":"53bc0c74-61e1-4713-94e4-bab1b71e8fe8","Type":"ContainerStarted","Data":"24b447179757ea00ca1bbd87b6e720ce1d1e9fdeda17bca7c9c4ba34f117ac39"}
Oct 30 13:31:49.986523 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:49.986376    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s"
Oct 30 13:31:49.988274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:49.988213    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r" event={"ID":"be7872c4-5356-451c-bc78-767629d8a6ea","Type":"ContainerStarted","Data":"90cebdd128c1e8c9376ad244813f25a5d2e6b8170dc0c7e28bf2dada859a41b8"}
Oct 30 13:31:49.988399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:49.988297    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r" event={"ID":"be7872c4-5356-451c-bc78-767629d8a6ea","Type":"ContainerStarted","Data":"f3cfa0bcbe1cbcf79e0757fecfe33763a937f10874ec97d8e88fd932f1cbcc60"}
Oct 30 13:31:49.988399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:49.988318    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r"
Oct 30 13:31:50.061787 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:50.060953    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s" podStartSLOduration=2.060931131 podStartE2EDuration="2.060931131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:50.035091867 +0000 UTC m=+2805.082225846" watchObservedRunningTime="2026-10-30 13:31:50.060931131 +0000 UTC m=+2805.108065086"
Oct 30 13:31:50.092322 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:50.092268    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r" podStartSLOduration=2.092215066 podStartE2EDuration="2.092215066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:50.090645842 +0000 UTC m=+2805.137779792" watchObservedRunningTime="2026-10-30 13:31:50.092215066 +0000 UTC m=+2805.139349044"
Oct 30 13:31:52.133649 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:52.133612    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:52.678941 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:52.678905    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:53.968476 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:53.968434    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:55.859591 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:55.859555    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:57.366720 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:57.366682    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:58.009178 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:58.009141    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:59.009375 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:31:59.009342    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:31:59.209846 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:59.209800    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:59.738055 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:59.738016    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:31:59.895001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:31:59.894961    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:00.088621 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:00.088583    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:01.005189 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:01.005154    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r"
Oct 30 13:32:01.005615 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:01.005595    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s"
Oct 30 13:32:01.568771 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:01.568734    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:03.528735 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:03.528703    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:05.024538 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:05.024497    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:05.213447 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:05.213408    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:05.742699 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:05.742664    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:06.882917 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:06.882879    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:07.061576 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:07.061534    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d"]
Oct 30 13:32:07.066783 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:07.066754    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d"
Oct 30 13:32:07.078486 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:07.078450    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:07.110726 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:07.110684    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d"]
Oct 30 13:32:07.114343 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:07.114294    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1c8130-dockercfg-9894g\""
Oct 30 13:32:07.114343 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:07.114307    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1c8130\""
Oct 30 13:32:07.144817 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:07.144710    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c1d897-b96c-45dd-a6cd-341b8abc28fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d\" (UID: \"29c1d897-b96c-45dd-a6cd-341b8abc28fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d"
Oct 30 13:32:07.145006 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:07.144954    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd9xm\" (UniqueName: \"kubernetes.io/projected/29c1d897-b96c-45dd-a6cd-341b8abc28fe-kube-api-access-bd9xm\") pod \"payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d\" (UID: \"29c1d897-b96c-45dd-a6cd-341b8abc28fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d"
Oct 30 13:32:07.154993 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:07.154940    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp"]
Oct 30 13:32:07.275244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:07.202028    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp"
Oct 30 13:32:07.275244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:07.270903    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a5e1569-11e1-4247-ae2e-4507f8826474-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp\" (UID: \"0a5e1569-11e1-4247-ae2e-4507f8826474\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp"
Oct 30 13:32:07.275244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:07.271051    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd9xm\" (UniqueName: \"kubernetes.io/projected/29c1d897-b96c-45dd-a6cd-341b8abc28fe-kube-api-access-bd9xm\") pod \"payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d\" (UID: \"29c1d897-b96c-45dd-a6cd-341b8abc28fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d"
Oct 30 13:32:07.275244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:07.271126    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqxrn\" (UniqueName: \"kubernetes.io/projected/0a5e1569-11e1-4247-ae2e-4507f8826474-kube-api-access-wqxrn\") pod \"payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp\" (UID: \"0a5e1569-11e1-4247-ae2e-4507f8826474\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp"
Oct 30 13:32:07.275244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:07.271516    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c1d897-b96c-45dd-a6cd-341b8abc28fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d\" (UID: \"29c1d897-b96c-45dd-a6cd-341b8abc28fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d"
Oct 30 13:32:07.300692 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:07.300654    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c1d897-b96c-45dd-a6cd-341b8abc28fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d\" (UID: \"29c1d897-b96c-45dd-a6cd-341b8abc28fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d"
Oct 30 13:32:07.347186 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:07.346590    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd9xm\" (UniqueName: \"kubernetes.io/projected/29c1d897-b96c-45dd-a6cd-341b8abc28fe-kube-api-access-bd9xm\") pod \"payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d\" (UID: \"29c1d897-b96c-45dd-a6cd-341b8abc28fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d"
Oct 30 13:32:07.357360 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:07.357285    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp"]
Oct 30 13:32:07.376837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:07.376796    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a5e1569-11e1-4247-ae2e-4507f8826474-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp\" (UID: \"0a5e1569-11e1-4247-ae2e-4507f8826474\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp"
Oct 30 13:32:07.377233 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:07.377192    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqxrn\" (UniqueName: \"kubernetes.io/projected/0a5e1569-11e1-4247-ae2e-4507f8826474-kube-api-access-wqxrn\") pod \"payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp\" (UID: \"0a5e1569-11e1-4247-ae2e-4507f8826474\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp"
Oct 30 13:32:07.379143 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:07.379118    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a5e1569-11e1-4247-ae2e-4507f8826474-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp\" (UID: \"0a5e1569-11e1-4247-ae2e-4507f8826474\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp"
Oct 30 13:32:07.386955 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:07.386903    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d"
Oct 30 13:32:07.417241 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:07.417089    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqxrn\" (UniqueName: \"kubernetes.io/projected/0a5e1569-11e1-4247-ae2e-4507f8826474-kube-api-access-wqxrn\") pod \"payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp\" (UID: \"0a5e1569-11e1-4247-ae2e-4507f8826474\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp"
Oct 30 13:32:07.708235 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:07.708123    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp"
Oct 30 13:32:07.743775 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:07.743744    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d"]
Oct 30 13:32:07.747965 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:32:07.745056    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c1d897_b96c_45dd_a6cd_341b8abc28fe.slice/crio-eb80b18c555d873a9e6bb326e406ca01be6632d54334d5c86776a710a3aea730 WatchSource:0}: Error finding container eb80b18c555d873a9e6bb326e406ca01be6632d54334d5c86776a710a3aea730: Status 404 returned error can't find the container with id eb80b18c555d873a9e6bb326e406ca01be6632d54334d5c86776a710a3aea730
Oct 30 13:32:08.015700 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:08.015610    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:08.068196 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:08.068071    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp"]
Oct 30 13:32:08.077624 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:32:08.077196    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5e1569_11e1_4247_ae2e_4507f8826474.slice/crio-5dcc06e08d7c7fc923f896e4ed77efc679a8df0935c0a643cfaf06df7c444386 WatchSource:0}: Error finding container 5dcc06e08d7c7fc923f896e4ed77efc679a8df0935c0a643cfaf06df7c444386: Status 404 returned error can't find the container with id 5dcc06e08d7c7fc923f896e4ed77efc679a8df0935c0a643cfaf06df7c444386
Oct 30 13:32:08.365615 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:08.365563    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp" event={"ID":"0a5e1569-11e1-4247-ae2e-4507f8826474","Type":"ContainerStarted","Data":"c84e8c806c8b026494947f3bc392f06a8663dc03c4f73bfec7813ea841e41ab6"}
Oct 30 13:32:08.365851 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:08.365670    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp" event={"ID":"0a5e1569-11e1-4247-ae2e-4507f8826474","Type":"ContainerStarted","Data":"5dcc06e08d7c7fc923f896e4ed77efc679a8df0935c0a643cfaf06df7c444386"}
Oct 30 13:32:08.365851 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:08.365741    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp"
Oct 30 13:32:08.367667 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:08.367637    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d" event={"ID":"29c1d897-b96c-45dd-a6cd-341b8abc28fe","Type":"ContainerStarted","Data":"f714a4fe8eea47da1571266e6dd5919ae1d876af94d56d36f4588697a7cc7bae"}
Oct 30 13:32:08.367809 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:08.367684    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d" event={"ID":"29c1d897-b96c-45dd-a6cd-341b8abc28fe","Type":"ContainerStarted","Data":"eb80b18c555d873a9e6bb326e406ca01be6632d54334d5c86776a710a3aea730"}
Oct 30 13:32:08.367809 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:08.367744    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d"
Oct 30 13:32:08.410611 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:08.410556    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp" podStartSLOduration=1.410541601 podStartE2EDuration="1.410541601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:08.409521298 +0000 UTC m=+2823.456655251" watchObservedRunningTime="2026-10-30 13:32:08.410541601 +0000 UTC m=+2823.457675554"
Oct 30 13:32:08.453204 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:08.453143    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d" podStartSLOduration=1.453126627 podStartE2EDuration="1.453126627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:08.45115027 +0000 UTC m=+2823.498284223" watchObservedRunningTime="2026-10-30 13:32:08.453126627 +0000 UTC m=+2823.500260580"
Oct 30 13:32:09.886564 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:09.886523    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:11.480133 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:11.480098    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"]
Oct 30 13:32:11.485307 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:11.485288    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:32:11.505753 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:11.505733    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3c1b28-openshift-default-dockercfg-rbqkl\""
Oct 30 13:32:11.522287 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:11.522261    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"]
Oct 30 13:32:11.568557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:11.568528    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10885581-4aa8-4607-b81a-158066a7dd27-istio-token\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:32:11.568717 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:11.568561    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdr7h\" (UniqueName: \"kubernetes.io/projected/10885581-4aa8-4607-b81a-158066a7dd27-kube-api-access-vdr7h\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:32:11.568761 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:11.568714    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-credential-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:32:11.568841 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:11.568823    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10885581-4aa8-4607-b81a-158066a7dd27-istiod-ca-cert\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:32:11.568882 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:11.568862    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-workload-certs\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:32:11.569005 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:11.568993    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10885581-4aa8-4607-b81a-158066a7dd27-istio-podinfo\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:32:11.569044 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:11.569023    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-istio-envoy\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:32:11.569044 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:11.569040    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-istio-data\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:32:11.569124 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:11.569105    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-workload-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:32:11.728247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:11.714895    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10885581-4aa8-4607-b81a-158066a7dd27-istio-token\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:32:11.728247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:11.714985    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdr7h\" (UniqueName: \"kubernetes.io/projected/10885581-4aa8-4607-b81a-158066a7dd27-kube-api-access-vdr7h\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:32:11.728247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:11.715178    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-credential-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:32:11.728247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:11.715347    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10885581-4aa8-4607-b81a-158066a7dd27-istiod-ca-cert\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:32:11.728247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:11.715406    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-workload-certs\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:32:11.728247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:11.715583    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10885581-4aa8-4607-b81a-158066a7dd27-istio-podinfo\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:32:11.728247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:11.715633    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-istio-envoy\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:32:11.728247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:11.715664    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-istio-data\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:32:11.728247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:11.715749    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-workload-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:32:11.728247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:11.716510    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-workload-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:32:11.728247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:11.717590    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-workload-certs\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:32:11.728247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:11.717620    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10885581-4aa8-4607-b81a-158066a7dd27-istiod-ca-cert\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:32:11.728247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:11.718951    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-credential-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:32:11.728247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:11.719647    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-istio-data\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:32:11.734261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:11.734139    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10885581-4aa8-4607-b81a-158066a7dd27-istio-podinfo\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:32:11.750207 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:11.749779    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-istio-envoy\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:32:11.750827 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:11.750804    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdr7h\" (UniqueName: \"kubernetes.io/projected/10885581-4aa8-4607-b81a-158066a7dd27-kube-api-access-vdr7h\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:32:11.751485 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:11.751461    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10885581-4aa8-4607-b81a-158066a7dd27-istio-token\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:32:11.764273 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:11.764212    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:11.801915 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:11.801885    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:32:12.130399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:12.130371    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"]
Oct 30 13:32:12.131902 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:32:12.131876    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10885581_4aa8_4607_b81a_158066a7dd27.slice/crio-25632e4e32d734b218454f88459d0b1cd6f98e0dacce0a0ee0b8d423c8872c0e WatchSource:0}: Error finding container 25632e4e32d734b218454f88459d0b1cd6f98e0dacce0a0ee0b8d423c8872c0e: Status 404 returned error can't find the container with id 25632e4e32d734b218454f88459d0b1cd6f98e0dacce0a0ee0b8d423c8872c0e
Oct 30 13:32:12.134729 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:12.134693    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:32:12.134809 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:12.134762    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:32:12.134809 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:12.134794    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:32:12.403131 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:12.403046    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt" event={"ID":"10885581-4aa8-4607-b81a-158066a7dd27","Type":"ContainerStarted","Data":"b1b6e8f55f2d229518c72cc3c354d25fbe7cf48ea872859b6dc54fecebbaf3ec"}
Oct 30 13:32:12.403131 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:12.403102    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt" event={"ID":"10885581-4aa8-4607-b81a-158066a7dd27","Type":"ContainerStarted","Data":"25632e4e32d734b218454f88459d0b1cd6f98e0dacce0a0ee0b8d423c8872c0e"}
Oct 30 13:32:12.450511 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:12.450458    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt" podStartSLOduration=1.450444261 podStartE2EDuration="1.450444261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:12.449900371 +0000 UTC m=+2827.497034325" watchObservedRunningTime="2026-10-30 13:32:12.450444261 +0000 UTC m=+2827.497578213"
Oct 30 13:32:12.802201 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:12.802119    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:32:13.644345 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:13.632893    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:13.804595 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:13.804554    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body=
Oct 30 13:32:13.805164 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:13.804616    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt" podUID="10885581-4aa8-4607-b81a-158066a7dd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused"
Oct 30 13:32:14.342510 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:14.342471    4544 scope.go:122] "RemoveContainer" containerID="58dfca7d7b51d2ad7683d60b02fa6c1e37e6390795f66617421c30c830fde1cd"
Oct 30 13:32:14.353834 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:14.353811    4544 scope.go:122] "RemoveContainer" containerID="7b2fd25e9bbd5d7118b09b9c812a1dcf2229709314ad855a8cb0212f4d9a91f7"
Oct 30 13:32:14.802514 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:14.802436    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body=
Oct 30 13:32:14.802514 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:14.802491    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt" podUID="10885581-4aa8-4607-b81a-158066a7dd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused"
Oct 30 13:32:15.722248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:15.718785    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:15.802883 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:15.802850    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body=
Oct 30 13:32:15.803068 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:15.802912    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt" podUID="10885581-4aa8-4607-b81a-158066a7dd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused"
Oct 30 13:32:16.802787 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:16.802756    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body=
Oct 30 13:32:16.803194 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:16.802807    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt" podUID="10885581-4aa8-4607-b81a-158066a7dd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused"
Oct 30 13:32:17.802667 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:17.802628    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body=
Oct 30 13:32:17.802875 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:17.802701    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt" podUID="10885581-4aa8-4607-b81a-158066a7dd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused"
Oct 30 13:32:17.994623 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:17.994586    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:18.803196 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:18.803164    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body=
Oct 30 13:32:18.803694 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:18.803213    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt" podUID="10885581-4aa8-4607-b81a-158066a7dd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused"
Oct 30 13:32:19.379143 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:19.379108    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp"
Oct 30 13:32:19.379374 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:19.379192    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d"
Oct 30 13:32:19.807376 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:19.807294    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:32:19.807867 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:19.807763    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:32:19.808444 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:19.808422    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:32:19.871963 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:19.871925    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:21.610795 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:21.610759    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"]
Oct 30 13:32:21.618282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:21.618256    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:32:21.638870 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:21.638848    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-03b5b3-openshift-default-dockercfg-dp4wq\""
Oct 30 13:32:21.652333 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:21.652303    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"]
Oct 30 13:32:21.695007 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:21.694958    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a60c9a2-4e23-4e4e-be07-09609b79f023-istio-podinfo\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:32:21.695245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:21.695042    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-workload-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:32:21.695245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:21.695104    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-credential-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:32:21.695245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:21.695165    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-istio-data\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:32:21.695245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:21.695198    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-workload-certs\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:32:21.695520 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:21.695249    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a60c9a2-4e23-4e4e-be07-09609b79f023-istiod-ca-cert\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:32:21.695520 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:21.695379    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-istio-envoy\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:32:21.695520 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:21.695463    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a60c9a2-4e23-4e4e-be07-09609b79f023-istio-token\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:32:21.695687 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:21.695612    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgpk8\" (UniqueName: \"kubernetes.io/projected/5a60c9a2-4e23-4e4e-be07-09609b79f023-kube-api-access-hgpk8\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:32:21.766406 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:21.766371    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:21.841196 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:21.841154    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-credential-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:32:21.841567 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:21.841539    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-istio-data\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:32:21.842003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:21.841717    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-credential-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:32:21.842003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:21.841984    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-istio-data\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:32:21.842177 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:21.841862    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-workload-certs\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:32:21.842280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:21.842236    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a60c9a2-4e23-4e4e-be07-09609b79f023-istiod-ca-cert\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:32:21.842369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:21.842352    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-istio-envoy\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:32:21.842433 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:21.842409    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-workload-certs\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:32:21.842498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:21.842436    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a60c9a2-4e23-4e4e-be07-09609b79f023-istio-token\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:32:21.842578 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:21.842561    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgpk8\" (UniqueName: \"kubernetes.io/projected/5a60c9a2-4e23-4e4e-be07-09609b79f023-kube-api-access-hgpk8\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:32:21.842994 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:21.842958    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a60c9a2-4e23-4e4e-be07-09609b79f023-istio-podinfo\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:32:21.843109 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:21.843003    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a60c9a2-4e23-4e4e-be07-09609b79f023-istiod-ca-cert\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:32:21.843109 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:21.843043    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-workload-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:32:21.843630 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:21.843597    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-workload-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:32:21.845910 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:21.845881    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-istio-envoy\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:32:21.846444 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:21.846351    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a60c9a2-4e23-4e4e-be07-09609b79f023-istio-podinfo\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:32:21.866427 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:21.866362    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a60c9a2-4e23-4e4e-be07-09609b79f023-istio-token\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:32:21.866427 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:21.866416    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgpk8\" (UniqueName: \"kubernetes.io/projected/5a60c9a2-4e23-4e4e-be07-09609b79f023-kube-api-access-hgpk8\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:32:21.935110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:21.935066    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:32:22.264961 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:22.264934    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"]
Oct 30 13:32:22.265952 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:32:22.265928    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a60c9a2_4e23_4e4e_be07_09609b79f023.slice/crio-d6ff3cd1995e2295bf36c296f324a3ddc27ea2a491c0b7f32df677a3c9a19952 WatchSource:0}: Error finding container d6ff3cd1995e2295bf36c296f324a3ddc27ea2a491c0b7f32df677a3c9a19952: Status 404 returned error can't find the container with id d6ff3cd1995e2295bf36c296f324a3ddc27ea2a491c0b7f32df677a3c9a19952
Oct 30 13:32:22.268947 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:22.268914    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:32:22.269036 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:22.268984    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:32:22.269036 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:22.269013    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:32:22.513802 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:22.513706    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2" event={"ID":"5a60c9a2-4e23-4e4e-be07-09609b79f023","Type":"ContainerStarted","Data":"2773c3435f6f1ba8746bfb50ec5af3da37ee191eecb9e36e167615f6e1e96ae5"}
Oct 30 13:32:22.513802 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:22.513782    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2" event={"ID":"5a60c9a2-4e23-4e4e-be07-09609b79f023","Type":"ContainerStarted","Data":"d6ff3cd1995e2295bf36c296f324a3ddc27ea2a491c0b7f32df677a3c9a19952"}
Oct 30 13:32:22.564403 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:22.564358    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2" podStartSLOduration=1.564336771 podStartE2EDuration="1.564336771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:22.559264924 +0000 UTC m=+2837.606398878" watchObservedRunningTime="2026-10-30 13:32:22.564336771 +0000 UTC m=+2837.611470724"
Oct 30 13:32:22.937462 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:22.937419    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:32:23.505772 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:23.505715    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:23.939003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:23.938963    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body=
Oct 30 13:32:23.939478 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:23.939026    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2" podUID="5a60c9a2-4e23-4e4e-be07-09609b79f023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused"
Oct 30 13:32:24.937833 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:24.937769    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body=
Oct 30 13:32:24.938201 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:24.938138    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2" podUID="5a60c9a2-4e23-4e4e-be07-09609b79f023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused"
Oct 30 13:32:25.433246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:25.433049    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:25.938246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:25.937719    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body=
Oct 30 13:32:25.938246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:25.937793    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2" podUID="5a60c9a2-4e23-4e4e-be07-09609b79f023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused"
Oct 30 13:32:26.938926 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:26.938644    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body=
Oct 30 13:32:26.938926 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:26.938710    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2" podUID="5a60c9a2-4e23-4e4e-be07-09609b79f023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused"
Oct 30 13:32:27.399916 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:27.399836    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n"]
Oct 30 13:32:27.442931 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:27.442836    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k"]
Oct 30 13:32:27.568573 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:27.568541    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:27.688656 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:27.688574    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:27.847258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:27.844902    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:27.938778 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:27.938577    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body=
Oct 30 13:32:27.938778 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:27.938636    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2" podUID="5a60c9a2-4e23-4e4e-be07-09609b79f023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused"
Oct 30 13:32:28.241010 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:28.240896    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:28.940703 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:28.940670    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:32:28.941190 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:28.941175    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:32:28.942026 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:28.942005    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:32:29.009139 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:32:29.009105    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:32:29.289773 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:29.289687    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:29.545373 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:29.544589    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:30.737722 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:30.737692    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:32.400631 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:32.400574    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n" podUID="6a5b15b6-da23-43f0-9505-01ef161423fa" containerName="payload-processing" containerID="cri-o://abc0dfba40c342891c2b24a45001f892f8bf40c59c1f4ec0a87f5899990f09e6" gracePeriod=25
Oct 30 13:32:32.443592 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:32.443553    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k" podUID="5da80418-8188-4b8c-87ca-5b887c86b70a" containerName="payload-pre-processing" containerID="cri-o://2b9794c2768da090508af02fd4ba522a6fc4cf56bf42cc94cec065e9da2c54f3" gracePeriod=25
Oct 30 13:32:32.662746 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:32.662611    4544 generic.go:356] "Generic (PLEG): container finished" podID="5da80418-8188-4b8c-87ca-5b887c86b70a" containerID="2b9794c2768da090508af02fd4ba522a6fc4cf56bf42cc94cec065e9da2c54f3" exitCode=0
Oct 30 13:32:32.662924 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:32.662870    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k" event={"ID":"5da80418-8188-4b8c-87ca-5b887c86b70a","Type":"ContainerDied","Data":"2b9794c2768da090508af02fd4ba522a6fc4cf56bf42cc94cec065e9da2c54f3"}
Oct 30 13:32:32.668691 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:32.667453    4544 generic.go:356] "Generic (PLEG): container finished" podID="6a5b15b6-da23-43f0-9505-01ef161423fa" containerID="abc0dfba40c342891c2b24a45001f892f8bf40c59c1f4ec0a87f5899990f09e6" exitCode=0
Oct 30 13:32:32.668691 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:32.667588    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n" event={"ID":"6a5b15b6-da23-43f0-9505-01ef161423fa","Type":"ContainerDied","Data":"abc0dfba40c342891c2b24a45001f892f8bf40c59c1f4ec0a87f5899990f09e6"}
Oct 30 13:32:32.759317 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:32.759286    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n"
Oct 30 13:32:32.818778 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:32.818748    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k"
Oct 30 13:32:32.874238 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:32.874185    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a5b15b6-da23-43f0-9505-01ef161423fa-kube-api-access-vk4l6\" (UniqueName: \"kubernetes.io/projected/6a5b15b6-da23-43f0-9505-01ef161423fa-kube-api-access-vk4l6\") pod \"6a5b15b6-da23-43f0-9505-01ef161423fa\" (UID: \"6a5b15b6-da23-43f0-9505-01ef161423fa\") "
Oct 30 13:32:32.874437 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:32.874391    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5da80418-8188-4b8c-87ca-5b887c86b70a-kube-api-access-fvkf7\" (UniqueName: \"kubernetes.io/projected/5da80418-8188-4b8c-87ca-5b887c86b70a-kube-api-access-fvkf7\") pod \"5da80418-8188-4b8c-87ca-5b887c86b70a\" (UID: \"5da80418-8188-4b8c-87ca-5b887c86b70a\") "
Oct 30 13:32:32.874521 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:32.874468    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5da80418-8188-4b8c-87ca-5b887c86b70a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da80418-8188-4b8c-87ca-5b887c86b70a-plugins-config-volume\") pod \"5da80418-8188-4b8c-87ca-5b887c86b70a\" (UID: \"5da80418-8188-4b8c-87ca-5b887c86b70a\") "
Oct 30 13:32:32.874730 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:32.874667    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a5b15b6-da23-43f0-9505-01ef161423fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a5b15b6-da23-43f0-9505-01ef161423fa-plugins-config-volume\") pod \"6a5b15b6-da23-43f0-9505-01ef161423fa\" (UID: \"6a5b15b6-da23-43f0-9505-01ef161423fa\") "
Oct 30 13:32:32.874883 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:32.874857    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5da80418-8188-4b8c-87ca-5b887c86b70a-plugins-config-volume" pod "5da80418-8188-4b8c-87ca-5b887c86b70a" (UID: "5da80418-8188-4b8c-87ca-5b887c86b70a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:32:32.877264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:32.875520    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a5b15b6-da23-43f0-9505-01ef161423fa-plugins-config-volume" pod "6a5b15b6-da23-43f0-9505-01ef161423fa" (UID: "6a5b15b6-da23-43f0-9505-01ef161423fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:32:32.877264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:32.875527    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da80418-8188-4b8c-87ca-5b887c86b70a-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:32:32.877264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:32.876809    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5da80418-8188-4b8c-87ca-5b887c86b70a-kube-api-access-fvkf7" pod "5da80418-8188-4b8c-87ca-5b887c86b70a" (UID: "5da80418-8188-4b8c-87ca-5b887c86b70a"). InnerVolumeSpecName "kube-api-access-fvkf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:32:32.877264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:32.876872    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a5b15b6-da23-43f0-9505-01ef161423fa-kube-api-access-vk4l6" pod "6a5b15b6-da23-43f0-9505-01ef161423fa" (UID: "6a5b15b6-da23-43f0-9505-01ef161423fa"). InnerVolumeSpecName "kube-api-access-vk4l6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:32:32.976687 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:32.976599    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvkf7\" (UniqueName: \"kubernetes.io/projected/5da80418-8188-4b8c-87ca-5b887c86b70a-kube-api-access-fvkf7\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:32:32.976687 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:32.976629    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a5b15b6-da23-43f0-9505-01ef161423fa-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:32:32.976687 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:32.976638    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk4l6\" (UniqueName: \"kubernetes.io/projected/6a5b15b6-da23-43f0-9505-01ef161423fa-kube-api-access-vk4l6\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:32:33.339059 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:33.339026    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:33.676860 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:33.676781    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n"
Oct 30 13:32:33.676860 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:33.676812    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n" event={"ID":"6a5b15b6-da23-43f0-9505-01ef161423fa","Type":"ContainerDied","Data":"fcf433e78c89fa4134be51db48eaa68e929111c8de7bf63534c8f1037d517c70"}
Oct 30 13:32:33.677571 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:33.676878    4544 scope.go:122] "RemoveContainer" containerID="abc0dfba40c342891c2b24a45001f892f8bf40c59c1f4ec0a87f5899990f09e6"
Oct 30 13:32:33.678925 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:33.678902    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k"
Oct 30 13:32:33.679059 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:33.678977    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k" event={"ID":"5da80418-8188-4b8c-87ca-5b887c86b70a","Type":"ContainerDied","Data":"de3113e4a3491ddec31542a19ea11af94714fb1584367b13f012bbe5572db451"}
Oct 30 13:32:33.688970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:33.688951    4544 scope.go:122] "RemoveContainer" containerID="2b9794c2768da090508af02fd4ba522a6fc4cf56bf42cc94cec065e9da2c54f3"
Oct 30 13:32:33.753889 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:33.753855    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n"]
Oct 30 13:32:33.765447 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:33.765416    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n"]
Oct 30 13:32:33.810402 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:33.810364    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k"]
Oct 30 13:32:33.819501 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:33.819476    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k"]
Oct 30 13:32:35.089672 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:35.089639    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5da80418-8188-4b8c-87ca-5b887c86b70a" path="/var/lib/kubelet/pods/5da80418-8188-4b8c-87ca-5b887c86b70a/volumes"
Oct 30 13:32:35.090133 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:35.090118    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a5b15b6-da23-43f0-9505-01ef161423fa" path="/var/lib/kubelet/pods/6a5b15b6-da23-43f0-9505-01ef161423fa/volumes"
Oct 30 13:32:35.190538 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:35.190505    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:36.849144 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:36.849107    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk"]
Oct 30 13:32:36.850013 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:36.849995    4544 cpu_manager.go:414] "containerMap: removing container" podUID="5da80418-8188-4b8c-87ca-5b887c86b70a" containerName="payload-pre-processing"
Oct 30 13:32:36.850071 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:36.850013    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5da80418-8188-4b8c-87ca-5b887c86b70a" containerName="payload-pre-processing"
Oct 30 13:32:36.850071 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:36.850034    4544 cpu_manager.go:414] "containerMap: removing container" podUID="6a5b15b6-da23-43f0-9505-01ef161423fa" containerName="payload-processing"
Oct 30 13:32:36.850071 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:36.850040    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a5b15b6-da23-43f0-9505-01ef161423fa" containerName="payload-processing"
Oct 30 13:32:36.850187 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:36.850177    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a5b15b6-da23-43f0-9505-01ef161423fa" containerName="payload-processing"
Oct 30 13:32:36.850260 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:36.850197    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="5da80418-8188-4b8c-87ca-5b887c86b70a" containerName="payload-pre-processing"
Oct 30 13:32:36.854421 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:36.854402    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk"
Oct 30 13:32:36.875264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:36.875212    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-c91909-serving-cert\""
Oct 30 13:32:36.904930 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:36.904889    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk"]
Oct 30 13:32:36.936173 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:36.936137    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:36.948105 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:36.948072    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw7gb\" (UniqueName: \"kubernetes.io/projected/d4fc9668-7314-46ee-bfef-6092e5b58d61-kube-api-access-sw7gb\") pod \"maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk\" (UID: \"d4fc9668-7314-46ee-bfef-6092e5b58d61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk"
Oct 30 13:32:36.948324 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:36.948122    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4fc9668-7314-46ee-bfef-6092e5b58d61-metrics-certs\") pod \"maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk\" (UID: \"d4fc9668-7314-46ee-bfef-6092e5b58d61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk"
Oct 30 13:32:36.948449 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:36.948426    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fc9668-7314-46ee-bfef-6092e5b58d61-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk\" (UID: \"d4fc9668-7314-46ee-bfef-6092e5b58d61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk"
Oct 30 13:32:36.948520 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:36.948472    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4fc9668-7314-46ee-bfef-6092e5b58d61-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk\" (UID: \"d4fc9668-7314-46ee-bfef-6092e5b58d61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk"
Oct 30 13:32:37.050555 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:37.050511    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fc9668-7314-46ee-bfef-6092e5b58d61-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk\" (UID: \"d4fc9668-7314-46ee-bfef-6092e5b58d61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk"
Oct 30 13:32:37.050760 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:37.050591    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4fc9668-7314-46ee-bfef-6092e5b58d61-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk\" (UID: \"d4fc9668-7314-46ee-bfef-6092e5b58d61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk"
Oct 30 13:32:37.050760 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:37.050677    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw7gb\" (UniqueName: \"kubernetes.io/projected/d4fc9668-7314-46ee-bfef-6092e5b58d61-kube-api-access-sw7gb\") pod \"maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk\" (UID: \"d4fc9668-7314-46ee-bfef-6092e5b58d61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk"
Oct 30 13:32:37.050760 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:37.050726    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4fc9668-7314-46ee-bfef-6092e5b58d61-metrics-certs\") pod \"maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk\" (UID: \"d4fc9668-7314-46ee-bfef-6092e5b58d61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk"
Oct 30 13:32:37.051300 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:37.051266    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fc9668-7314-46ee-bfef-6092e5b58d61-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk\" (UID: \"d4fc9668-7314-46ee-bfef-6092e5b58d61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk"
Oct 30 13:32:37.053152 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:37.053124    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:37.053526 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:37.053503    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4fc9668-7314-46ee-bfef-6092e5b58d61-metrics-certs\") pod \"maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk\" (UID: \"d4fc9668-7314-46ee-bfef-6092e5b58d61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk"
Oct 30 13:32:37.053613 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:37.053537    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4fc9668-7314-46ee-bfef-6092e5b58d61-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk\" (UID: \"d4fc9668-7314-46ee-bfef-6092e5b58d61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk"
Oct 30 13:32:37.098524 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:37.096249    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw7gb\" (UniqueName: \"kubernetes.io/projected/d4fc9668-7314-46ee-bfef-6092e5b58d61-kube-api-access-sw7gb\") pod \"maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk\" (UID: \"d4fc9668-7314-46ee-bfef-6092e5b58d61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk"
Oct 30 13:32:37.173293 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:37.172400    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk"
Oct 30 13:32:37.486105 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:37.486079    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk"]
Oct 30 13:32:37.487572 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:32:37.487541    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4fc9668_7314_46ee_bfef_6092e5b58d61.slice/crio-b0532979304701f3ef4b0b386b684b36f1b5bf58037d67e68915b3609513f3a3 WatchSource:0}: Error finding container b0532979304701f3ef4b0b386b684b36f1b5bf58037d67e68915b3609513f3a3: Status 404 returned error can't find the container with id b0532979304701f3ef4b0b386b684b36f1b5bf58037d67e68915b3609513f3a3
Oct 30 13:32:37.489035 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:37.489017    4544 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:32:37.721000 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:37.720967    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk" event={"ID":"d4fc9668-7314-46ee-bfef-6092e5b58d61","Type":"ContainerStarted","Data":"b0532979304701f3ef4b0b386b684b36f1b5bf58037d67e68915b3609513f3a3"}
Oct 30 13:32:38.731016 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:38.730969    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk" event={"ID":"d4fc9668-7314-46ee-bfef-6092e5b58d61","Type":"ContainerStarted","Data":"b613d9d7edcf38bd9e93fbe10be08a613bc31eaf3a8f18f6df35d0883081c6cc"}
Oct 30 13:32:38.731541 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:38.731038    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk"
Oct 30 13:32:38.776362 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:38.776309    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk" podStartSLOduration=2.452835826 podStartE2EDuration="2.776294297s" totalImagesPullingTime="323.458471ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:38.77282495 +0000 UTC m=+2853.819958904" watchObservedRunningTime="2026-10-30 13:32:38.776294297 +0000 UTC m=+2853.823428250"
Oct 30 13:32:38.906216 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:38.906176    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:40.769293 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:40.768501    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:42.485288 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:42.485249    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:44.824243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:44.823786    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk"
Oct 30 13:32:45.090379 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:45.084947    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:46.864114 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:46.864071    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:48.774794 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:48.774760    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:50.770140 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:50.770086    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:52.474287 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:52.474187    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:54.294229 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:54.294196    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:56.068374 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.068337    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:56.507302 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.507265    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"]
Oct 30 13:32:56.514044 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.514012    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:32:56.530003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.529966    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"]
Oct 30 13:32:56.534973 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.534941    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:32:56.560192 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.560155    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"]
Oct 30 13:32:56.566965 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.566924    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47dxc\" (UniqueName: \"kubernetes.io/projected/44a8f645-e783-438c-9efe-4196f9e97dc7-kube-api-access-47dxc\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:32:56.567272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.567242    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"]
Oct 30 13:32:56.567379 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.567248    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-workload-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:32:56.567613 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.567583    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a8f645-e783-438c-9efe-4196f9e97dc7-istiod-ca-cert\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:32:56.567754 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.567635    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istio-data\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:32:56.567754 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.567668    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istio-envoy\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:32:56.567754 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.567732    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-workload-certs\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:32:56.569417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.567833    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istio-podinfo\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:32:56.569417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.567879    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-istio-envoy\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:32:56.569417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.567937    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-workload-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:32:56.569417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.567966    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxs68\" (UniqueName: \"kubernetes.io/projected/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-kube-api-access-zxs68\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:32:56.569417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.568013    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a8f645-e783-438c-9efe-4196f9e97dc7-istio-token\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:32:56.569417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.568073    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istio-token\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:32:56.569417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.568112    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a8f645-e783-438c-9efe-4196f9e97dc7-istio-podinfo\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:32:56.569417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.568160    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-workload-certs\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:32:56.569417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.568273    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-credential-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:32:56.569417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.568365    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-istio-data\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:32:56.569417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.568484    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-credential-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:32:56.569417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.568614    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istiod-ca-cert\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:32:56.670266 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.670177    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-istio-data\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:32:56.670477 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.670284    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-credential-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:32:56.670477 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.670333    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istiod-ca-cert\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:32:56.670477 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.670393    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47dxc\" (UniqueName: \"kubernetes.io/projected/44a8f645-e783-438c-9efe-4196f9e97dc7-kube-api-access-47dxc\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:32:56.670477 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.670414    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-workload-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:32:56.670724 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.670481    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a8f645-e783-438c-9efe-4196f9e97dc7-istiod-ca-cert\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:32:56.670724 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.670523    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istio-data\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:32:56.670724 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.670556    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istio-envoy\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:32:56.670724 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.670621    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-workload-certs\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:32:56.670724 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.670720    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istio-podinfo\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:32:56.671001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.670721    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-istio-data\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:32:56.671001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.670729    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-credential-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:32:56.671001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.670763    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-istio-envoy\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:32:56.671001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.670827    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-workload-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:32:56.671001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.670856    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxs68\" (UniqueName: \"kubernetes.io/projected/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-kube-api-access-zxs68\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:32:56.671001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.670902    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a8f645-e783-438c-9efe-4196f9e97dc7-istio-token\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:32:56.671001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.670938    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-workload-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:32:56.671001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.670969    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istio-token\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:32:56.671475 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.671010    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a8f645-e783-438c-9efe-4196f9e97dc7-istio-podinfo\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:32:56.671475 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.671068    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-workload-certs\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:32:56.671475 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.671069    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-workload-certs\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:32:56.671475 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.671163    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-credential-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:32:56.671475 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.671208    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-workload-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:32:56.671475 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.671261    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istiod-ca-cert\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:32:56.671475 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.671012    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istio-data\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:32:56.671475 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.671375    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-workload-certs\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:32:56.671918 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.671625    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-credential-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:32:56.671918 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.671875    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a8f645-e783-438c-9efe-4196f9e97dc7-istiod-ca-cert\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:32:56.673489 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.673465    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istio-envoy\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:32:56.673611 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.673486    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-istio-envoy\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:32:56.673762 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.673725    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istio-podinfo\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:32:56.674131 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.674111    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a8f645-e783-438c-9efe-4196f9e97dc7-istio-podinfo\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:32:56.695267 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.695193    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a8f645-e783-438c-9efe-4196f9e97dc7-istio-token\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:32:56.695439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.695414    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istio-token\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:32:56.695512 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.695448    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47dxc\" (UniqueName: \"kubernetes.io/projected/44a8f645-e783-438c-9efe-4196f9e97dc7-kube-api-access-47dxc\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:32:56.695896 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.695871    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxs68\" (UniqueName: \"kubernetes.io/projected/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-kube-api-access-zxs68\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:32:56.833067 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.833020    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:32:56.851337 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:56.851300    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:32:57.182842 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:32:57.182809    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a8f645_e783_438c_9efe_4196f9e97dc7.slice/crio-65acde9178e2bfecb53e4300830dc5d50bb5dbe76ee2261b8e0c5053ff8e430c WatchSource:0}: Error finding container 65acde9178e2bfecb53e4300830dc5d50bb5dbe76ee2261b8e0c5053ff8e430c: Status 404 returned error can't find the container with id 65acde9178e2bfecb53e4300830dc5d50bb5dbe76ee2261b8e0c5053ff8e430c
Oct 30 13:32:57.183694 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:57.183669    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"]
Oct 30 13:32:57.186728 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:32:57.186701    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a9deb6_b7b1_4870_adc9_8025aa6b49e2.slice/crio-5f4c670861820d5e6e404d494832f970a54502ffe037f7b5e2974c1c161181db WatchSource:0}: Error finding container 5f4c670861820d5e6e404d494832f970a54502ffe037f7b5e2974c1c161181db: Status 404 returned error can't find the container with id 5f4c670861820d5e6e404d494832f970a54502ffe037f7b5e2974c1c161181db
Oct 30 13:32:57.186882 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:57.186756    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:32:57.186882 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:57.186849    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:32:57.187014 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:57.186891    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:32:57.199245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:57.190772    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"]
Oct 30 13:32:57.202946 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:57.202869    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:32:57.203163 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:57.203055    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:32:57.203372 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:57.203187    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:32:57.480900 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:57.480801    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:57.936318 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:57.936285    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:58.067260 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:58.056758    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:32:58.196971 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:58.196874    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x" event={"ID":"44a8f645-e783-438c-9efe-4196f9e97dc7","Type":"ContainerStarted","Data":"f3ac95f6bc180f39f26ead4d3d19b71a830374a693482cf2e833f7112fccbfd8"}
Oct 30 13:32:58.196971 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:58.196941    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x" event={"ID":"44a8f645-e783-438c-9efe-4196f9e97dc7","Type":"ContainerStarted","Data":"65acde9178e2bfecb53e4300830dc5d50bb5dbe76ee2261b8e0c5053ff8e430c"}
Oct 30 13:32:58.198749 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:58.198723    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s" event={"ID":"00a9deb6-b7b1-4870-adc9-8025aa6b49e2","Type":"ContainerStarted","Data":"206b025f477102937bf636d950a52c8e69f3541117ad2e48489780ad47d50f96"}
Oct 30 13:32:58.198926 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:58.198780    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s" event={"ID":"00a9deb6-b7b1-4870-adc9-8025aa6b49e2","Type":"ContainerStarted","Data":"5f4c670861820d5e6e404d494832f970a54502ffe037f7b5e2974c1c161181db"}
Oct 30 13:32:58.255542 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:58.255480    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x" podStartSLOduration=2.255465601 podStartE2EDuration="2.255465601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:58.253628685 +0000 UTC m=+2873.300762639" watchObservedRunningTime="2026-10-30 13:32:58.255465601 +0000 UTC m=+2873.302599658"
Oct 30 13:32:58.299139 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:58.299070    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s" podStartSLOduration=2.299054106 podStartE2EDuration="2.299054106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:58.298033558 +0000 UTC m=+2873.345167512" watchObservedRunningTime="2026-10-30 13:32:58.299054106 +0000 UTC m=+2873.346188059"
Oct 30 13:32:58.837258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:58.835113    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:32:58.837258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:58.835317    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body=
Oct 30 13:32:58.837258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:58.835381    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x" podUID="44a8f645-e783-438c-9efe-4196f9e97dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused"
Oct 30 13:32:58.851699 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:58.851659    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:32:58.853769 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:58.853722    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body=
Oct 30 13:32:58.853928 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:58.853797    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s" podUID="00a9deb6-b7b1-4870-adc9-8025aa6b49e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused"
Oct 30 13:32:59.011467 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:32:59.011429    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:32:59.847823 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:59.847783    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body=
Oct 30 13:32:59.848476 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:59.848437    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x" podUID="44a8f645-e783-438c-9efe-4196f9e97dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused"
Oct 30 13:32:59.858855 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:59.858819    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body=
Oct 30 13:32:59.859124 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:59.859097    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s" podUID="00a9deb6-b7b1-4870-adc9-8025aa6b49e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused"
Oct 30 13:32:59.963376 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:32:59.961076    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:00.082828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:00.082788    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:00.865304 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:00.850354    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body=
Oct 30 13:33:00.865304 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:00.850420    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x" podUID="44a8f645-e783-438c-9efe-4196f9e97dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused"
Oct 30 13:33:00.865304 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:00.854638    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body=
Oct 30 13:33:00.865304 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:00.854713    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s" podUID="00a9deb6-b7b1-4870-adc9-8025aa6b49e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused"
Oct 30 13:33:01.836246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:01.835763    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body=
Oct 30 13:33:01.836246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:01.835835    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x" podUID="44a8f645-e783-438c-9efe-4196f9e97dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused"
Oct 30 13:33:01.860245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:01.857877    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body=
Oct 30 13:33:01.860245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:01.857941    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s" podUID="00a9deb6-b7b1-4870-adc9-8025aa6b49e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused"
Oct 30 13:33:02.541242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:02.540483    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:02.837246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:02.833722    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body=
Oct 30 13:33:02.837246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:02.833801    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x" podUID="44a8f645-e783-438c-9efe-4196f9e97dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused"
Oct 30 13:33:02.862247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:02.861763    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body=
Oct 30 13:33:02.862247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:02.861835    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s" podUID="00a9deb6-b7b1-4870-adc9-8025aa6b49e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused"
Oct 30 13:33:03.839246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:03.838650    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body=
Oct 30 13:33:03.839246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:03.838717    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x" podUID="44a8f645-e783-438c-9efe-4196f9e97dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused"
Oct 30 13:33:03.858244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:03.854588    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body=
Oct 30 13:33:03.858244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:03.854648    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s" podUID="00a9deb6-b7b1-4870-adc9-8025aa6b49e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused"
Oct 30 13:33:04.501091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:04.501059    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:04.837247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:04.835157    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body=
Oct 30 13:33:04.837247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:04.835253    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x" podUID="44a8f645-e783-438c-9efe-4196f9e97dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused"
Oct 30 13:33:04.857028 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:04.856997    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:33:04.857486 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:04.857340    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:33:04.858014 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:04.857997    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:33:05.838712 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:05.838681    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:33:05.839191 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:05.839169    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:33:05.840102 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:05.840083    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:33:06.317861 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.317779    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:06.652180 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.652080    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:06.768075 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.768029    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"]
Oct 30 13:33:06.777334 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.777299    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:33:06.780210 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.780174    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"]
Oct 30 13:33:06.791303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.787458    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:33:06.800580 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.800545    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"]
Oct 30 13:33:06.820048 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.816849    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"]
Oct 30 13:33:06.836399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.835417    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-istio-data\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:33:06.836399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.835479    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6f4e05-5105-4596-b248-bc018f59ce81-istiod-ca-cert\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:33:06.836399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.835518    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f90bdf7e-ba66-4f37-9585-5173e85566fb-istio-token\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:33:06.836399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.835558    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-istio-data\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:33:06.836399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.835583    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f90bdf7e-ba66-4f37-9585-5173e85566fb-istio-podinfo\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:33:06.836399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.835616    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b6f4e05-5105-4596-b248-bc018f59ce81-istio-token\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:33:06.836399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.835645    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-credential-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:33:06.836399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.835688    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-workload-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:33:06.836399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.835741    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc2xs\" (UniqueName: \"kubernetes.io/projected/8b6f4e05-5105-4596-b248-bc018f59ce81-kube-api-access-kc2xs\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:33:06.836399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.835955    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f90bdf7e-ba66-4f37-9585-5173e85566fb-istiod-ca-cert\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:33:06.836399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.835983    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9967\" (UniqueName: \"kubernetes.io/projected/f90bdf7e-ba66-4f37-9585-5173e85566fb-kube-api-access-k9967\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:33:06.836399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.836034    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-workload-certs\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:33:06.836399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.836117    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-istio-envoy\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:33:06.836399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.836160    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-workload-certs\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:33:06.836399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.836215    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-credential-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:33:06.836399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.836281    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-workload-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:33:06.837590 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.836448    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-istio-envoy\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:33:06.837590 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.836478    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6f4e05-5105-4596-b248-bc018f59ce81-istio-podinfo\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:33:06.938429 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.938345    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-credential-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:33:06.938429 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.938398    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-workload-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:33:06.938694 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.938491    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-istio-envoy\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:33:06.938694 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.938510    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6f4e05-5105-4596-b248-bc018f59ce81-istio-podinfo\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:33:06.938694 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.938557    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-istio-data\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:33:06.938694 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.938596    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6f4e05-5105-4596-b248-bc018f59ce81-istiod-ca-cert\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:33:06.938694 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.938650    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f90bdf7e-ba66-4f37-9585-5173e85566fb-istio-token\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:33:06.938969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.938701    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-istio-data\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:33:06.938969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.938724    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f90bdf7e-ba66-4f37-9585-5173e85566fb-istio-podinfo\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:33:06.938969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.938760    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b6f4e05-5105-4596-b248-bc018f59ce81-istio-token\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:33:06.938969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.938795    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-credential-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:33:06.938969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.938822    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-credential-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:33:06.938969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.938844    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-workload-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:33:06.938969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.938929    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc2xs\" (UniqueName: \"kubernetes.io/projected/8b6f4e05-5105-4596-b248-bc018f59ce81-kube-api-access-kc2xs\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:33:06.939371 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.939017    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f90bdf7e-ba66-4f37-9585-5173e85566fb-istiod-ca-cert\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:33:06.939371 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.939028    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-istio-data\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:33:06.939371 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.939048    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9967\" (UniqueName: \"kubernetes.io/projected/f90bdf7e-ba66-4f37-9585-5173e85566fb-kube-api-access-k9967\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:33:06.939371 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.939117    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-workload-certs\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:33:06.939371 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.939162    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-workload-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:33:06.939371 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.939165    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-istio-envoy\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:33:06.939371 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.939302    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-workload-certs\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:33:06.940052 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.939704    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6f4e05-5105-4596-b248-bc018f59ce81-istiod-ca-cert\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:33:06.940259 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.939873    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-credential-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:33:06.940412 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.940027    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-workload-certs\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:33:06.940937 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.940887    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f90bdf7e-ba66-4f37-9585-5173e85566fb-istiod-ca-cert\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:33:06.941040 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.940977    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-workload-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:33:06.941474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.941428    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-istio-envoy\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:33:06.941591 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.941567    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-istio-envoy\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:33:06.941790 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.941768    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-workload-certs\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:33:06.941850 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.941822    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-istio-data\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:33:06.941978 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.941961    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6f4e05-5105-4596-b248-bc018f59ce81-istio-podinfo\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:33:06.942200 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.942181    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f90bdf7e-ba66-4f37-9585-5173e85566fb-istio-podinfo\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:33:06.964617 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.964581    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f90bdf7e-ba66-4f37-9585-5173e85566fb-istio-token\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:33:06.964812 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.964600    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9967\" (UniqueName: \"kubernetes.io/projected/f90bdf7e-ba66-4f37-9585-5173e85566fb-kube-api-access-k9967\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:33:06.964812 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.964652    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc2xs\" (UniqueName: \"kubernetes.io/projected/8b6f4e05-5105-4596-b248-bc018f59ce81-kube-api-access-kc2xs\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:33:06.965274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:06.965256    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b6f4e05-5105-4596-b248-bc018f59ce81-istio-token\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:33:07.122955 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:07.122919    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:33:07.138619 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:07.138586    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:33:07.463264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:07.463235    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"]
Oct 30 13:33:07.464956 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:33:07.464923    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b6f4e05_5105_4596_b248_bc018f59ce81.slice/crio-b299268c320da03565c480f057a8553a7ab2071612ff52b9fdc481b84d826e0f WatchSource:0}: Error finding container b299268c320da03565c480f057a8553a7ab2071612ff52b9fdc481b84d826e0f: Status 404 returned error can't find the container with id b299268c320da03565c480f057a8553a7ab2071612ff52b9fdc481b84d826e0f
Oct 30 13:33:07.468270 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:07.468192    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:33:07.468374 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:07.468309    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:33:07.468374 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:07.468342    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:33:07.507720 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:07.507696    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"]
Oct 30 13:33:07.510810 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:33:07.510780    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90bdf7e_ba66_4f37_9585_5173e85566fb.slice/crio-90c3040c250e658afdc45b34137ff9839d9217dc9864200fb265e068aacef586 WatchSource:0}: Error finding container 90c3040c250e658afdc45b34137ff9839d9217dc9864200fb265e068aacef586: Status 404 returned error can't find the container with id 90c3040c250e658afdc45b34137ff9839d9217dc9864200fb265e068aacef586
Oct 30 13:33:07.514689 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:07.514644    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:33:07.514792 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:07.514743    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:33:07.514838 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:07.514787    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:33:08.230989 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:08.230943    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:08.417242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:08.412412    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g" event={"ID":"8b6f4e05-5105-4596-b248-bc018f59ce81","Type":"ContainerStarted","Data":"b80f1275112c4ee5ff483cf8352a3d6cd269f111fde61e65075c0648ed69b0a5"}
Oct 30 13:33:08.417242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:08.412504    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g" event={"ID":"8b6f4e05-5105-4596-b248-bc018f59ce81","Type":"ContainerStarted","Data":"b299268c320da03565c480f057a8553a7ab2071612ff52b9fdc481b84d826e0f"}
Oct 30 13:33:08.483780 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:08.483673    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp" event={"ID":"f90bdf7e-ba66-4f37-9585-5173e85566fb","Type":"ContainerStarted","Data":"29bc0e6821304515e028dab5ad10551f3b28ec78e428dd737391a7098f100555"}
Oct 30 13:33:08.483780 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:08.483731    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp" event={"ID":"f90bdf7e-ba66-4f37-9585-5173e85566fb","Type":"ContainerStarted","Data":"90c3040c250e658afdc45b34137ff9839d9217dc9864200fb265e068aacef586"}
Oct 30 13:33:08.539245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:08.512622    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g" podStartSLOduration=2.512602559 podStartE2EDuration="2.512602559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:08.507736202 +0000 UTC m=+2883.554870155" watchObservedRunningTime="2026-10-30 13:33:08.512602559 +0000 UTC m=+2883.559736512"
Oct 30 13:33:08.576835 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:08.576781    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp" podStartSLOduration=2.576765243 podStartE2EDuration="2.576765243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:08.570999857 +0000 UTC m=+2883.618133812" watchObservedRunningTime="2026-10-30 13:33:08.576765243 +0000 UTC m=+2883.623899195"
Oct 30 13:33:09.131245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:09.125505    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:33:09.131245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:09.125650    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" start-of-body=
Oct 30 13:33:09.131245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:09.125694    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g" podUID="8b6f4e05-5105-4596-b248-bc018f59ce81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused"
Oct 30 13:33:09.161538 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:09.151713    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:33:09.168872 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:09.168261    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body=
Oct 30 13:33:09.168872 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:09.168344    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp" podUID="f90bdf7e-ba66-4f37-9585-5173e85566fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused"
Oct 30 13:33:10.009256 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:09.989553    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:10.126253 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:10.126147    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" start-of-body=
Oct 30 13:33:10.126253 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:10.126212    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g" podUID="8b6f4e05-5105-4596-b248-bc018f59ce81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused"
Oct 30 13:33:10.171259 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:10.168911    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body=
Oct 30 13:33:10.171259 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:10.168968    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp" podUID="f90bdf7e-ba66-4f37-9585-5173e85566fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused"
Oct 30 13:33:10.531304 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:10.527705    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:11.135247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:11.126724    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" start-of-body=
Oct 30 13:33:11.135247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:11.126793    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g" podUID="8b6f4e05-5105-4596-b248-bc018f59ce81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused"
Oct 30 13:33:11.142247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:11.141310    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body=
Oct 30 13:33:11.142247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:11.141373    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp" podUID="f90bdf7e-ba66-4f37-9585-5173e85566fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused"
Oct 30 13:33:11.667730 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:11.667645    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"]
Oct 30 13:33:11.752245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:11.729593    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:33:11.838548 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:11.837643    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66d450d8-b5a4-406f-a427-015307e5a9f6-istiod-ca-cert\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:33:11.838548 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:11.837726    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-credential-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:33:11.838548 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:11.837883    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqm7d\" (UniqueName: \"kubernetes.io/projected/66d450d8-b5a4-406f-a427-015307e5a9f6-kube-api-access-tqm7d\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:33:11.838548 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:11.837929    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66d450d8-b5a4-406f-a427-015307e5a9f6-istio-token\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:33:11.838548 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:11.837974    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-workload-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:33:11.838548 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:11.838012    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66d450d8-b5a4-406f-a427-015307e5a9f6-istio-podinfo\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:33:11.838548 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:11.838103    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-workload-certs\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:33:11.838548 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:11.838158    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-istio-envoy\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:33:11.838548 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:11.838210    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-istio-data\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:33:11.972640 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:11.945673    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66d450d8-b5a4-406f-a427-015307e5a9f6-istiod-ca-cert\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:33:11.972640 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:11.945771    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-credential-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:33:11.972640 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:11.945905    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqm7d\" (UniqueName: \"kubernetes.io/projected/66d450d8-b5a4-406f-a427-015307e5a9f6-kube-api-access-tqm7d\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:33:11.972640 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:11.945949    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66d450d8-b5a4-406f-a427-015307e5a9f6-istio-token\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:33:11.972640 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:11.945993    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-workload-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:33:11.972640 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:11.946048    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66d450d8-b5a4-406f-a427-015307e5a9f6-istio-podinfo\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:33:11.972640 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:11.946137    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-workload-certs\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:33:11.972640 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:11.946185    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-istio-envoy\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:33:11.972640 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:11.946257    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-istio-data\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:33:11.972640 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:11.947032    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-istio-data\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:33:11.972640 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:11.950067    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"]
Oct 30 13:33:11.972640 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:11.950498    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-workload-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:33:11.972640 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:11.950741    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-credential-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:33:11.972640 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:11.951507    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-workload-certs\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:33:12.033246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:12.032610    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66d450d8-b5a4-406f-a427-015307e5a9f6-istiod-ca-cert\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:33:12.151248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:12.139352    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" start-of-body=
Oct 30 13:33:12.151248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:12.139424    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g" podUID="8b6f4e05-5105-4596-b248-bc018f59ce81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused"
Oct 30 13:33:12.188154 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:12.188116    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body=
Oct 30 13:33:12.188484 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:12.188450    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp" podUID="f90bdf7e-ba66-4f37-9585-5173e85566fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused"
Oct 30 13:33:12.273312 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:12.257118    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-istio-envoy\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:33:12.273312 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:12.257659    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66d450d8-b5a4-406f-a427-015307e5a9f6-istio-podinfo\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:33:12.296569 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:12.296532    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66d450d8-b5a4-406f-a427-015307e5a9f6-istio-token\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:33:12.297678 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:12.297650    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqm7d\" (UniqueName: \"kubernetes.io/projected/66d450d8-b5a4-406f-a427-015307e5a9f6-kube-api-access-tqm7d\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:33:12.585247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:12.583627    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:33:13.074857 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.074822    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:13.079390 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:33:13.079347    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d450d8_b5a4_406f_a427_015307e5a9f6.slice/crio-35c90626575ec46d7fe770062f403f36264db7e0722af6033c128381531cfea4 WatchSource:0}: Error finding container 35c90626575ec46d7fe770062f403f36264db7e0722af6033c128381531cfea4: Status 404 returned error can't find the container with id 35c90626575ec46d7fe770062f403f36264db7e0722af6033c128381531cfea4
Oct 30 13:33:13.084731 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.083857    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:33:13.084731 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.083959    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:33:13.084731 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.084009    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:33:13.148633 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.108511    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"]
Oct 30 13:33:13.159426 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.152627    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" start-of-body=
Oct 30 13:33:13.159426 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.152690    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g" podUID="8b6f4e05-5105-4596-b248-bc018f59ce81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused"
Oct 30 13:33:13.233643 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.227810    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body=
Oct 30 13:33:13.233643 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.227885    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp" podUID="f90bdf7e-ba66-4f37-9585-5173e85566fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused"
Oct 30 13:33:13.660693 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.660568    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"]
Oct 30 13:33:13.683749 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.683716    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:33:13.687521 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.686527    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"]
Oct 30 13:33:13.703256 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.701281    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-51288b-openshift-default-dockercfg-bbqtx\""
Oct 30 13:33:13.788244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.788055    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istiod-ca-cert\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:33:13.788244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.788188    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istio-envoy\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:33:13.788244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.788248    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istio-podinfo\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:33:13.788609 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.788291    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istio-token\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:33:13.788609 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.788375    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-workload-certs\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:33:13.788609 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.788408    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-credential-socket\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:33:13.788609 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.788440    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkt22\" (UniqueName: \"kubernetes.io/projected/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-kube-api-access-mkt22\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:33:13.788609 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.788541    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-workload-socket\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:33:13.788904 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.788613    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istio-data\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:33:13.890663 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.890487    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istiod-ca-cert\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:33:13.890663 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.890613    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istio-envoy\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:33:13.890663 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.890651    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istio-podinfo\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:33:13.891017 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.890695    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istio-token\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:33:13.891017 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.890788    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-workload-certs\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:33:13.891017 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.890825    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-credential-socket\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:33:13.891017 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.890854    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkt22\" (UniqueName: \"kubernetes.io/projected/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-kube-api-access-mkt22\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:33:13.891017 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.890957    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-workload-socket\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:33:13.891355 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.891025    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istio-data\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:33:13.891912 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.891745    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istio-data\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:33:13.894391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.894354    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istiod-ca-cert\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:33:13.897249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.894789    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-workload-certs\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:33:13.897249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.896121    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istio-podinfo\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:33:13.897249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.896490    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-credential-socket\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:33:13.897249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.896631    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-workload-socket\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:33:13.954472 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.946840    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkt22\" (UniqueName: \"kubernetes.io/projected/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-kube-api-access-mkt22\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:33:13.989065 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.988919    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istio-envoy\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:33:13.992112 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:13.992052    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istio-token\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:33:14.008147 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:14.008103    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj" event={"ID":"66d450d8-b5a4-406f-a427-015307e5a9f6","Type":"ContainerStarted","Data":"a5d7aabf9d2045725d76d8bf5c5bea75194491daa70b774fbc0d32e810bfb8fe"}
Oct 30 13:33:14.008447 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:14.008428    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj" event={"ID":"66d450d8-b5a4-406f-a427-015307e5a9f6","Type":"ContainerStarted","Data":"35c90626575ec46d7fe770062f403f36264db7e0722af6033c128381531cfea4"}
Oct 30 13:33:14.008537 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:14.008471    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:33:14.100156 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:14.098595    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj" podStartSLOduration=3.098570954 podStartE2EDuration="3.098570954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:14.092864463 +0000 UTC m=+2889.139998439" watchObservedRunningTime="2026-10-30 13:33:14.098570954 +0000 UTC m=+2889.145704908"
Oct 30 13:33:14.125731 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:14.125188    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" start-of-body=
Oct 30 13:33:14.125731 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:14.125279    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g" podUID="8b6f4e05-5105-4596-b248-bc018f59ce81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused"
Oct 30 13:33:14.161957 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:14.161424    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body=
Oct 30 13:33:14.161957 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:14.161490    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp" podUID="f90bdf7e-ba66-4f37-9585-5173e85566fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused"
Oct 30 13:33:14.591998 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:14.589696    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:33:14.592861 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:14.592821    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Oct 30 13:33:14.593022 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:14.592906    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj" podUID="66d450d8-b5a4-406f-a427-015307e5a9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Oct 30 13:33:14.682490 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:14.682450    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"]
Oct 30 13:33:14.686165 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:33:14.686123    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode16e9bc8_8d01_4ab3_a840_190ffa478e8f.slice/crio-925fb155516771ad0d1fc0267875634720dc2d2cef51541ff27c714809fcd213 WatchSource:0}: Error finding container 925fb155516771ad0d1fc0267875634720dc2d2cef51541ff27c714809fcd213: Status 404 returned error can't find the container with id 925fb155516771ad0d1fc0267875634720dc2d2cef51541ff27c714809fcd213
Oct 30 13:33:14.690499 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:14.690455    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:33:14.690613 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:14.690538    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:33:14.690613 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:14.690580    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:33:15.083209 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:15.083168    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z" event={"ID":"e16e9bc8-8d01-4ab3-a840-190ffa478e8f","Type":"ContainerStarted","Data":"122ac7f78ea0efc1fe7d89b1ddb88301a5a1cd431b202e93246e578c2a98aa6e"}
Oct 30 13:33:15.083497 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:15.083471    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z" event={"ID":"e16e9bc8-8d01-4ab3-a840-190ffa478e8f","Type":"ContainerStarted","Data":"925fb155516771ad0d1fc0267875634720dc2d2cef51541ff27c714809fcd213"}
Oct 30 13:33:15.160791 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:15.160748    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:33:15.162808 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:15.162540    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:33:15.162808 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:15.162708    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z" podStartSLOduration=2.162692893 podStartE2EDuration="2.162692893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:15.161373316 +0000 UTC m=+2890.208507271" watchObservedRunningTime="2026-10-30 13:33:15.162692893 +0000 UTC m=+2890.209826848"
Oct 30 13:33:15.165447 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:15.165421    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:33:15.293939 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:15.293903    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:33:15.295668 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:15.295480    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:33:15.297782 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:15.297753    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:33:15.586205 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:15.586168    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Oct 30 13:33:15.587172 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:15.586317    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj" podUID="66d450d8-b5a4-406f-a427-015307e5a9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Oct 30 13:33:15.644164 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:15.644105    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:16.009452 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:16.009362    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:33:16.011017 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:16.010991    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body=
Oct 30 13:33:16.011105 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:16.011044    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z" podUID="e16e9bc8-8d01-4ab3-a840-190ffa478e8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused"
Oct 30 13:33:16.586018 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:16.585982    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Oct 30 13:33:16.586541 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:16.586040    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj" podUID="66d450d8-b5a4-406f-a427-015307e5a9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Oct 30 13:33:17.009172 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:17.009074    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body=
Oct 30 13:33:17.009172 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:17.009134    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z" podUID="e16e9bc8-8d01-4ab3-a840-190ffa478e8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused"
Oct 30 13:33:17.586841 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:17.586671    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Oct 30 13:33:17.586841 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:17.586734    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj" podUID="66d450d8-b5a4-406f-a427-015307e5a9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Oct 30 13:33:17.666459 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:17.666419    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:18.009423 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:18.009323    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body=
Oct 30 13:33:18.009423 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:18.009395    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z" podUID="e16e9bc8-8d01-4ab3-a840-190ffa478e8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused"
Oct 30 13:33:18.587363 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:18.587323    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Oct 30 13:33:18.587803 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:18.587406    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj" podUID="66d450d8-b5a4-406f-a427-015307e5a9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Oct 30 13:33:19.009776 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:19.009665    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body=
Oct 30 13:33:19.009776 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:19.009736    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z" podUID="e16e9bc8-8d01-4ab3-a840-190ffa478e8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused"
Oct 30 13:33:19.586439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:19.586405    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Oct 30 13:33:19.586627 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:19.586455    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj" podUID="66d450d8-b5a4-406f-a427-015307e5a9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Oct 30 13:33:19.890670 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:19.890573    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:20.014247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:20.013723    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body=
Oct 30 13:33:20.014247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:20.013811    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z" podUID="e16e9bc8-8d01-4ab3-a840-190ffa478e8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused"
Oct 30 13:33:20.591777 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:20.591742    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:33:20.592285 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:20.592261    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:33:20.592985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:20.592965    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:33:21.009742 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:21.009624    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body=
Oct 30 13:33:21.009742 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:21.009704    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z" podUID="e16e9bc8-8d01-4ab3-a840-190ffa478e8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused"
Oct 30 13:33:21.341381 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:21.341342    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:21.523828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:21.523785    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:22.161770 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:22.161722    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body=
Oct 30 13:33:22.162405 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:22.162368    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z" podUID="e16e9bc8-8d01-4ab3-a840-190ffa478e8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused"
Oct 30 13:33:22.965749 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:22.965702    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:23.046787 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:23.046685    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:33:23.055454 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:23.052695    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:33:23.057576 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:23.057503    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:33:24.760665 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:24.760612    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:27.308201 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:27.308154    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:29.012605 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:33:29.012563    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:33:29.671921 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:29.671881    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:31.512058 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:31.512021    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:32.138047 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:32.137988    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:33.502583 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:33.502540    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:35.360292 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:35.360250    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:35.844965 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:35.844912    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:36.295834 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:36.295748    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"]
Oct 30 13:33:36.302560 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:36.302529    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:33:36.360328 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:36.357358    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-72208f-openshift-default-dockercfg-hps7z\""
Oct 30 13:33:36.366394 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:36.366355    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"]
Oct 30 13:33:36.386064 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:36.386032    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:36.519567 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:36.516031    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-workload-socket\") pod \"e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") " pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:33:36.519567 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:36.516168    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-workload-certs\") pod \"e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") " pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:33:36.519567 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:36.516198    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-istio-data\") pod \"e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") " pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:33:36.519567 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:36.516270    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c713af1f-1695-4ad5-a564-25420ba533a6-istio-podinfo\") pod \"e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") " pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:33:36.519567 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:36.516394    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-credential-socket\") pod \"e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") " pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:33:36.519567 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:36.516446    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5zj6\" (UniqueName: \"kubernetes.io/projected/c713af1f-1695-4ad5-a564-25420ba533a6-kube-api-access-w5zj6\") pod \"e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") " pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:33:36.519567 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:36.516556    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c713af1f-1695-4ad5-a564-25420ba533a6-istiod-ca-cert\") pod \"e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") " pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:33:36.519567 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:36.516590    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-istio-envoy\") pod \"e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") " pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:33:36.519567 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:36.516665    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c713af1f-1695-4ad5-a564-25420ba533a6-istio-token\") pod \"e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") " pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:33:36.555298 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:36.538160    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:36.621243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:36.621046    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-workload-socket\") pod \"e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") " pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:33:36.621243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:36.621190    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-workload-certs\") pod \"e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") " pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:33:36.621243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:36.621238    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-istio-data\") pod \"e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") " pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:33:36.621576 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:36.621289    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c713af1f-1695-4ad5-a564-25420ba533a6-istio-podinfo\") pod \"e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") " pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:33:36.621576 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:36.621425    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-credential-socket\") pod \"e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") " pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:33:36.621576 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:36.621480    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5zj6\" (UniqueName: \"kubernetes.io/projected/c713af1f-1695-4ad5-a564-25420ba533a6-kube-api-access-w5zj6\") pod \"e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") " pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:33:36.621741 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:36.621622    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c713af1f-1695-4ad5-a564-25420ba533a6-istiod-ca-cert\") pod \"e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") " pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:33:36.621741 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:36.621654    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-istio-envoy\") pod \"e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") " pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:33:36.621864 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:36.621740    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c713af1f-1695-4ad5-a564-25420ba533a6-istio-token\") pod \"e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") " pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:33:36.623877 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:36.623013    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-workload-socket\") pod \"e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") " pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:33:36.623877 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:36.623263    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-workload-certs\") pod \"e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") " pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:33:36.623877 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:36.623473    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-istio-data\") pod \"e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") " pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:33:36.634300 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:36.627903    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-credential-socket\") pod \"e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") " pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:33:36.634300 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:36.628628    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c713af1f-1695-4ad5-a564-25420ba533a6-istiod-ca-cert\") pod \"e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") " pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:33:36.706257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:36.681735    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c713af1f-1695-4ad5-a564-25420ba533a6-istio-token\") pod \"e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") " pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:33:36.706257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:36.686028    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c713af1f-1695-4ad5-a564-25420ba533a6-istio-podinfo\") pod \"e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") " pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:33:36.706257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:36.686424    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-istio-envoy\") pod \"e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") " pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:33:36.706257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:36.688773    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5zj6\" (UniqueName: \"kubernetes.io/projected/c713af1f-1695-4ad5-a564-25420ba533a6-kube-api-access-w5zj6\") pod \"e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") " pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:33:36.962284 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:36.962205    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:33:37.346040 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:37.346004    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf"]
Oct 30 13:33:37.356387 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:37.356338    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:33:37.356501 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:37.356445    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:33:37.356547 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:37.356496    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:33:37.357066 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:37.357040    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"]
Oct 30 13:33:37.439246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:37.436081    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:37.690921 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:37.690823    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd" event={"ID":"c713af1f-1695-4ad5-a564-25420ba533a6","Type":"ContainerStarted","Data":"d46ca10012c4d4bb329228b670d7d9cfc03f6103aefeef2e06b797c58a8be510"}
Oct 30 13:33:37.978650 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:37.978555    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:38.287957 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:38.287682    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:38.813251 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:38.789879    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:38.848258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:38.843706    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd" event={"ID":"c713af1f-1695-4ad5-a564-25420ba533a6","Type":"ContainerStarted","Data":"89b11d4dfe81eb377fa0657c55a2f4f775b613b4b6f4a86b7e6a1f139cffb103"}
Oct 30 13:33:38.965243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:38.963964    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:33:39.023248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:39.020870    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:39.023248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:39.022695    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:33:39.023248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:39.022889    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd" podStartSLOduration=3.022869322 podStartE2EDuration="3.022869322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:38.962745178 +0000 UTC m=+2914.009879140" watchObservedRunningTime="2026-10-30 13:33:39.022869322 +0000 UTC m=+2914.070003277"
Oct 30 13:33:39.423246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:39.422632    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s"]
Oct 30 13:33:39.461266 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:39.460893    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r"]
Oct 30 13:33:39.601255 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:39.598129    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:39.758426 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:39.758212    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:39.885898 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:39.881344    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:33:39.885898 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:39.882938    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:33:40.731243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:40.731164    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:43.050246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:43.036106    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:43.399029 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:43.398231    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:43.703118 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:43.703018    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:44.395245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:44.394319    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:44.425727 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:44.424907    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s" podUID="53bc0c74-61e1-4713-94e4-bab1b71e8fe8" containerName="payload-processing" containerID="cri-o://53fc884765100f3bdfbbdae43c4fbd440cc95da1da0a2ac1dedd3c85ee7cd34e" gracePeriod=25
Oct 30 13:33:44.464376 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:44.464310    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r" podUID="be7872c4-5356-451c-bc78-767629d8a6ea" containerName="payload-pre-processing" containerID="cri-o://90cebdd128c1e8c9376ad244813f25a5d2e6b8170dc0c7e28bf2dada859a41b8" gracePeriod=25
Oct 30 13:33:44.795866 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:44.795822    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s"
Oct 30 13:33:44.833419 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:44.833362    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53bc0c74-61e1-4713-94e4-bab1b71e8fe8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53bc0c74-61e1-4713-94e4-bab1b71e8fe8-plugins-config-volume\") pod \"53bc0c74-61e1-4713-94e4-bab1b71e8fe8\" (UID: \"53bc0c74-61e1-4713-94e4-bab1b71e8fe8\") "
Oct 30 13:33:44.833795 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:44.833771    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53bc0c74-61e1-4713-94e4-bab1b71e8fe8-kube-api-access-kdsmm\" (UniqueName: \"kubernetes.io/projected/53bc0c74-61e1-4713-94e4-bab1b71e8fe8-kube-api-access-kdsmm\") pod \"53bc0c74-61e1-4713-94e4-bab1b71e8fe8\" (UID: \"53bc0c74-61e1-4713-94e4-bab1b71e8fe8\") "
Oct 30 13:33:44.833890 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:44.833870    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53bc0c74-61e1-4713-94e4-bab1b71e8fe8-plugins-config-volume" pod "53bc0c74-61e1-4713-94e4-bab1b71e8fe8" (UID: "53bc0c74-61e1-4713-94e4-bab1b71e8fe8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:33:44.834771 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:44.834743    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53bc0c74-61e1-4713-94e4-bab1b71e8fe8-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:33:44.836520 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:44.836416    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53bc0c74-61e1-4713-94e4-bab1b71e8fe8-kube-api-access-kdsmm" pod "53bc0c74-61e1-4713-94e4-bab1b71e8fe8" (UID: "53bc0c74-61e1-4713-94e4-bab1b71e8fe8"). InnerVolumeSpecName "kube-api-access-kdsmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:33:44.860348 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:44.860300    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r"
Oct 30 13:33:44.936822 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:44.936787    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be7872c4-5356-451c-bc78-767629d8a6ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be7872c4-5356-451c-bc78-767629d8a6ea-plugins-config-volume\") pod \"be7872c4-5356-451c-bc78-767629d8a6ea\" (UID: \"be7872c4-5356-451c-bc78-767629d8a6ea\") "
Oct 30 13:33:44.937018 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:44.936944    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be7872c4-5356-451c-bc78-767629d8a6ea-kube-api-access-knffk\" (UniqueName: \"kubernetes.io/projected/be7872c4-5356-451c-bc78-767629d8a6ea-kube-api-access-knffk\") pod \"be7872c4-5356-451c-bc78-767629d8a6ea\" (UID: \"be7872c4-5356-451c-bc78-767629d8a6ea\") "
Oct 30 13:33:44.937261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:44.937239    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be7872c4-5356-451c-bc78-767629d8a6ea-plugins-config-volume" pod "be7872c4-5356-451c-bc78-767629d8a6ea" (UID: "be7872c4-5356-451c-bc78-767629d8a6ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:33:44.937610 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:44.937595    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdsmm\" (UniqueName: \"kubernetes.io/projected/53bc0c74-61e1-4713-94e4-bab1b71e8fe8-kube-api-access-kdsmm\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:33:44.937655 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:44.937614    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be7872c4-5356-451c-bc78-767629d8a6ea-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:33:44.939136 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:44.939117    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be7872c4-5356-451c-bc78-767629d8a6ea-kube-api-access-knffk" pod "be7872c4-5356-451c-bc78-767629d8a6ea" (UID: "be7872c4-5356-451c-bc78-767629d8a6ea"). InnerVolumeSpecName "kube-api-access-knffk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:33:45.015863 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.015770    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:45.038296 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.038266    4544 generic.go:356] "Generic (PLEG): container finished" podID="be7872c4-5356-451c-bc78-767629d8a6ea" containerID="90cebdd128c1e8c9376ad244813f25a5d2e6b8170dc0c7e28bf2dada859a41b8" exitCode=0
Oct 30 13:33:45.038521 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.038397    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r" event={"ID":"be7872c4-5356-451c-bc78-767629d8a6ea","Type":"ContainerDied","Data":"90cebdd128c1e8c9376ad244813f25a5d2e6b8170dc0c7e28bf2dada859a41b8"}
Oct 30 13:33:45.038521 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.038487    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r" event={"ID":"be7872c4-5356-451c-bc78-767629d8a6ea","Type":"ContainerDied","Data":"f3cfa0bcbe1cbcf79e0757fecfe33763a937f10874ec97d8e88fd932f1cbcc60"}
Oct 30 13:33:45.038521 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.038520    4544 scope.go:122] "RemoveContainer" containerID="90cebdd128c1e8c9376ad244813f25a5d2e6b8170dc0c7e28bf2dada859a41b8"
Oct 30 13:33:45.038703 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.038419    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r"
Oct 30 13:33:45.039720 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.039695    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knffk\" (UniqueName: \"kubernetes.io/projected/be7872c4-5356-451c-bc78-767629d8a6ea-kube-api-access-knffk\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:33:45.040500 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.040477    4544 generic.go:356] "Generic (PLEG): container finished" podID="53bc0c74-61e1-4713-94e4-bab1b71e8fe8" containerID="53fc884765100f3bdfbbdae43c4fbd440cc95da1da0a2ac1dedd3c85ee7cd34e" exitCode=0
Oct 30 13:33:45.040580 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.040525    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s"
Oct 30 13:33:45.040634 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.040569    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s" event={"ID":"53bc0c74-61e1-4713-94e4-bab1b71e8fe8","Type":"ContainerDied","Data":"53fc884765100f3bdfbbdae43c4fbd440cc95da1da0a2ac1dedd3c85ee7cd34e"}
Oct 30 13:33:45.040634 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.040627    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s" event={"ID":"53bc0c74-61e1-4713-94e4-bab1b71e8fe8","Type":"ContainerDied","Data":"24b447179757ea00ca1bbd87b6e720ce1d1e9fdeda17bca7c9c4ba34f117ac39"}
Oct 30 13:33:45.053641 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.053615    4544 scope.go:122] "RemoveContainer" containerID="90cebdd128c1e8c9376ad244813f25a5d2e6b8170dc0c7e28bf2dada859a41b8"
Oct 30 13:33:45.053985 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:33:45.053966    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90cebdd128c1e8c9376ad244813f25a5d2e6b8170dc0c7e28bf2dada859a41b8\": container with ID starting with 90cebdd128c1e8c9376ad244813f25a5d2e6b8170dc0c7e28bf2dada859a41b8 not found: ID does not exist" containerID="90cebdd128c1e8c9376ad244813f25a5d2e6b8170dc0c7e28bf2dada859a41b8"
Oct 30 13:33:45.054050 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.053994    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90cebdd128c1e8c9376ad244813f25a5d2e6b8170dc0c7e28bf2dada859a41b8"} err="failed to get container status \"90cebdd128c1e8c9376ad244813f25a5d2e6b8170dc0c7e28bf2dada859a41b8\": rpc error: code = NotFound desc = could not find container \"90cebdd128c1e8c9376ad244813f25a5d2e6b8170dc0c7e28bf2dada859a41b8\": container with ID starting with 90cebdd128c1e8c9376ad244813f25a5d2e6b8170dc0c7e28bf2dada859a41b8 not found: ID does not exist"
Oct 30 13:33:45.054050 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.054013    4544 scope.go:122] "RemoveContainer" containerID="53fc884765100f3bdfbbdae43c4fbd440cc95da1da0a2ac1dedd3c85ee7cd34e"
Oct 30 13:33:45.067172 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.067142    4544 scope.go:122] "RemoveContainer" containerID="53fc884765100f3bdfbbdae43c4fbd440cc95da1da0a2ac1dedd3c85ee7cd34e"
Oct 30 13:33:45.067601 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:33:45.067577    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53fc884765100f3bdfbbdae43c4fbd440cc95da1da0a2ac1dedd3c85ee7cd34e\": container with ID starting with 53fc884765100f3bdfbbdae43c4fbd440cc95da1da0a2ac1dedd3c85ee7cd34e not found: ID does not exist" containerID="53fc884765100f3bdfbbdae43c4fbd440cc95da1da0a2ac1dedd3c85ee7cd34e"
Oct 30 13:33:45.067702 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.067614    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53fc884765100f3bdfbbdae43c4fbd440cc95da1da0a2ac1dedd3c85ee7cd34e"} err="failed to get container status \"53fc884765100f3bdfbbdae43c4fbd440cc95da1da0a2ac1dedd3c85ee7cd34e\": rpc error: code = NotFound desc = could not find container \"53fc884765100f3bdfbbdae43c4fbd440cc95da1da0a2ac1dedd3c85ee7cd34e\": container with ID starting with 53fc884765100f3bdfbbdae43c4fbd440cc95da1da0a2ac1dedd3c85ee7cd34e not found: ID does not exist"
Oct 30 13:33:45.121145 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.121106    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s"]
Oct 30 13:33:45.132624 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.132586    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s"]
Oct 30 13:33:45.181440 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.181395    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl"]
Oct 30 13:33:45.182813 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.182786    4544 cpu_manager.go:414] "containerMap: removing container" podUID="53bc0c74-61e1-4713-94e4-bab1b71e8fe8" containerName="payload-processing"
Oct 30 13:33:45.182813 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.182813    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53bc0c74-61e1-4713-94e4-bab1b71e8fe8" containerName="payload-processing"
Oct 30 13:33:45.183051 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.182878    4544 cpu_manager.go:414] "containerMap: removing container" podUID="be7872c4-5356-451c-bc78-767629d8a6ea" containerName="payload-pre-processing"
Oct 30 13:33:45.183051 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.182886    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be7872c4-5356-451c-bc78-767629d8a6ea" containerName="payload-pre-processing"
Oct 30 13:33:45.183147 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.183091    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="53bc0c74-61e1-4713-94e4-bab1b71e8fe8" containerName="payload-processing"
Oct 30 13:33:45.183147 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.183118    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="be7872c4-5356-451c-bc78-767629d8a6ea" containerName="payload-pre-processing"
Oct 30 13:33:45.187924 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.187898    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl"
Oct 30 13:33:45.191627 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.191582    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r"]
Oct 30 13:33:45.202809 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.202780    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r"]
Oct 30 13:33:45.226569 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.226539    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-72208f\""
Oct 30 13:33:45.243620 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.243565    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67zv4\" (UniqueName: \"kubernetes.io/projected/d232746e-9562-4578-b95f-09895eece465-kube-api-access-67zv4\") pod \"payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl\" (UID: \"d232746e-9562-4578-b95f-09895eece465\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl"
Oct 30 13:33:45.243833 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.243695    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d232746e-9562-4578-b95f-09895eece465-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl\" (UID: \"d232746e-9562-4578-b95f-09895eece465\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl"
Oct 30 13:33:45.251358 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.251319    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl"]
Oct 30 13:33:45.255716 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.255680    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4"]
Oct 30 13:33:45.264944 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.264908    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4"
Oct 30 13:33:45.290771 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.290559    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-72208f-dockercfg-7t5d4\""
Oct 30 13:33:45.296597 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.296538    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4"]
Oct 30 13:33:45.310693 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.310656    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:45.372248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.347206    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aba97076-557c-4b61-9c1a-374341fbe152-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4\" (UID: \"aba97076-557c-4b61-9c1a-374341fbe152\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4"
Oct 30 13:33:45.372248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.347386    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67zv4\" (UniqueName: \"kubernetes.io/projected/d232746e-9562-4578-b95f-09895eece465-kube-api-access-67zv4\") pod \"payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl\" (UID: \"d232746e-9562-4578-b95f-09895eece465\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl"
Oct 30 13:33:45.372248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.347493    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d232746e-9562-4578-b95f-09895eece465-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl\" (UID: \"d232746e-9562-4578-b95f-09895eece465\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl"
Oct 30 13:33:45.372248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.347802    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhjz8\" (UniqueName: \"kubernetes.io/projected/aba97076-557c-4b61-9c1a-374341fbe152-kube-api-access-nhjz8\") pod \"payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4\" (UID: \"aba97076-557c-4b61-9c1a-374341fbe152\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4"
Oct 30 13:33:45.372248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.348840    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d232746e-9562-4578-b95f-09895eece465-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl\" (UID: \"d232746e-9562-4578-b95f-09895eece465\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl"
Oct 30 13:33:45.458248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.453050    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhjz8\" (UniqueName: \"kubernetes.io/projected/aba97076-557c-4b61-9c1a-374341fbe152-kube-api-access-nhjz8\") pod \"payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4\" (UID: \"aba97076-557c-4b61-9c1a-374341fbe152\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4"
Oct 30 13:33:45.458248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.453260    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aba97076-557c-4b61-9c1a-374341fbe152-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4\" (UID: \"aba97076-557c-4b61-9c1a-374341fbe152\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4"
Oct 30 13:33:45.458248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.454553    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aba97076-557c-4b61-9c1a-374341fbe152-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4\" (UID: \"aba97076-557c-4b61-9c1a-374341fbe152\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4"
Oct 30 13:33:45.528051 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.528004    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67zv4\" (UniqueName: \"kubernetes.io/projected/d232746e-9562-4578-b95f-09895eece465-kube-api-access-67zv4\") pod \"payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl\" (UID: \"d232746e-9562-4578-b95f-09895eece465\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl"
Oct 30 13:33:45.528983 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.528904    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhjz8\" (UniqueName: \"kubernetes.io/projected/aba97076-557c-4b61-9c1a-374341fbe152-kube-api-access-nhjz8\") pod \"payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4\" (UID: \"aba97076-557c-4b61-9c1a-374341fbe152\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4"
Oct 30 13:33:45.709353 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.707434    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4"
Oct 30 13:33:45.806807 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.806691    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl"
Oct 30 13:33:45.984020 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.981400    4544 patch_prober.go:29] interesting pod/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:33:45.984020 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:45.981473    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz" podUID="4400fc6b-5d1c-4f3f-93be-7ef8540e8d79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:33:46.094662 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:46.094630    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4"]
Oct 30 13:33:46.096592 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:33:46.096562    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba97076_557c_4b61_9c1a_374341fbe152.slice/crio-ae7adad940036cc1aed8f095cf489b301f04a2fa7c7f393e4185ca5657433088 WatchSource:0}: Error finding container ae7adad940036cc1aed8f095cf489b301f04a2fa7c7f393e4185ca5657433088: Status 404 returned error can't find the container with id ae7adad940036cc1aed8f095cf489b301f04a2fa7c7f393e4185ca5657433088
Oct 30 13:33:46.173566 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:33:46.173522    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd232746e_9562_4578_b95f_09895eece465.slice/crio-6e58582a32cf08261188d2a5af5654273131a3b9fdc017aab32615b246907054 WatchSource:0}: Error finding container 6e58582a32cf08261188d2a5af5654273131a3b9fdc017aab32615b246907054: Status 404 returned error can't find the container with id 6e58582a32cf08261188d2a5af5654273131a3b9fdc017aab32615b246907054
Oct 30 13:33:46.174721 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:46.174689    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl"]
Oct 30 13:33:47.093174 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:47.093074    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53bc0c74-61e1-4713-94e4-bab1b71e8fe8" path="/var/lib/kubelet/pods/53bc0c74-61e1-4713-94e4-bab1b71e8fe8/volumes"
Oct 30 13:33:47.093821 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:47.093801    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be7872c4-5356-451c-bc78-767629d8a6ea" path="/var/lib/kubelet/pods/be7872c4-5356-451c-bc78-767629d8a6ea/volumes"
Oct 30 13:33:47.094394 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:47.094370    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4" event={"ID":"aba97076-557c-4b61-9c1a-374341fbe152","Type":"ContainerStarted","Data":"447e5c02621aded4a7bf2b3daabe0916be0cf13244f6150dcc39e435693966c9"}
Oct 30 13:33:47.094466 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:47.094439    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4" event={"ID":"aba97076-557c-4b61-9c1a-374341fbe152","Type":"ContainerStarted","Data":"ae7adad940036cc1aed8f095cf489b301f04a2fa7c7f393e4185ca5657433088"}
Oct 30 13:33:47.094506 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:47.094473    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4"
Oct 30 13:33:47.094506 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:47.094497    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl"
Oct 30 13:33:47.094609 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:47.094521    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl" event={"ID":"d232746e-9562-4578-b95f-09895eece465","Type":"ContainerStarted","Data":"3f363054c1f041fb06b97602f9739316f56be031b99aa81eebe5c6aa2f58d2dc"}
Oct 30 13:33:47.094609 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:47.094547    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl" event={"ID":"d232746e-9562-4578-b95f-09895eece465","Type":"ContainerStarted","Data":"6e58582a32cf08261188d2a5af5654273131a3b9fdc017aab32615b246907054"}
Oct 30 13:33:47.221007 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:47.220971    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:47.266193 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:47.266137    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4" podStartSLOduration=2.266119781 podStartE2EDuration="2.266119781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:47.181588862 +0000 UTC m=+2922.228722815" watchObservedRunningTime="2026-10-30 13:33:47.266119781 +0000 UTC m=+2922.313253807"
Oct 30 13:33:47.266844 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:47.266809    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl" podStartSLOduration=2.266798117 podStartE2EDuration="2.266798117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:47.26665627 +0000 UTC m=+2922.313790222" watchObservedRunningTime="2026-10-30 13:33:47.266798117 +0000 UTC m=+2922.313932070"
Oct 30 13:33:47.845804 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:47.845759    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg"]
Oct 30 13:33:47.854440 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:47.854412    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg"
Oct 30 13:33:47.868621 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:47.868585    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:47.880934 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:47.880898    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-78543b\""
Oct 30 13:33:47.894279 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:47.892584    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg"]
Oct 30 13:33:47.953036 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:47.952984    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgjxn\" (UniqueName: \"kubernetes.io/projected/45260af2-724d-4065-b966-542af12813a6-kube-api-access-bgjxn\") pod \"payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg\" (UID: \"45260af2-724d-4065-b966-542af12813a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg"
Oct 30 13:33:47.953282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:47.953251    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45260af2-724d-4065-b966-542af12813a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg\" (UID: \"45260af2-724d-4065-b966-542af12813a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg"
Oct 30 13:33:48.064267 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:48.058665    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgjxn\" (UniqueName: \"kubernetes.io/projected/45260af2-724d-4065-b966-542af12813a6-kube-api-access-bgjxn\") pod \"payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg\" (UID: \"45260af2-724d-4065-b966-542af12813a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg"
Oct 30 13:33:48.064267 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:48.058830    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45260af2-724d-4065-b966-542af12813a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg\" (UID: \"45260af2-724d-4065-b966-542af12813a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg"
Oct 30 13:33:48.064267 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:48.060030    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45260af2-724d-4065-b966-542af12813a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg\" (UID: \"45260af2-724d-4065-b966-542af12813a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg"
Oct 30 13:33:48.248302 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:48.247920    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgjxn\" (UniqueName: \"kubernetes.io/projected/45260af2-724d-4065-b966-542af12813a6-kube-api-access-bgjxn\") pod \"payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg\" (UID: \"45260af2-724d-4065-b966-542af12813a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg"
Oct 30 13:33:48.295104 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:48.294943    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:48.473299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:48.473262    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg"
Oct 30 13:33:48.808621 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:48.808592    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg"]
Oct 30 13:33:48.810331 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:33:48.810301    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45260af2_724d_4065_b966_542af12813a6.slice/crio-1c30d580a00f0f6b3ef63c294e0bbf6e3222d9a8213c424b8353d666daf86495 WatchSource:0}: Error finding container 1c30d580a00f0f6b3ef63c294e0bbf6e3222d9a8213c424b8353d666daf86495: Status 404 returned error can't find the container with id 1c30d580a00f0f6b3ef63c294e0bbf6e3222d9a8213c424b8353d666daf86495
Oct 30 13:33:49.294336 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:49.294294    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg" event={"ID":"45260af2-724d-4065-b966-542af12813a6","Type":"ContainerStarted","Data":"915424c5beba64e95cb93c60770d6a155b8792482cba716a18ecfe771f1cabdf"}
Oct 30 13:33:49.294888 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:49.294363    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg" event={"ID":"45260af2-724d-4065-b966-542af12813a6","Type":"ContainerStarted","Data":"1c30d580a00f0f6b3ef63c294e0bbf6e3222d9a8213c424b8353d666daf86495"}
Oct 30 13:33:49.294888 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:49.294393    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg"
Oct 30 13:33:49.296032 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:49.296006    4544 generic.go:356] "Generic (PLEG): container finished" podID="4400fc6b-5d1c-4f3f-93be-7ef8540e8d79" containerID="d6d4b0cad2e96d7bf655fc4d0c2d3d91862bd6ff7b91b1be789b66aeb860b672" exitCode=0
Oct 30 13:33:49.296190 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:49.296083    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz" event={"ID":"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79","Type":"ContainerDied","Data":"d6d4b0cad2e96d7bf655fc4d0c2d3d91862bd6ff7b91b1be789b66aeb860b672"}
Oct 30 13:33:49.296498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:49.296481    4544 scope.go:122] "RemoveContainer" containerID="d6d4b0cad2e96d7bf655fc4d0c2d3d91862bd6ff7b91b1be789b66aeb860b672"
Oct 30 13:33:49.301179 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:49.301026    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:33:49.301270 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:49.301241    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:33:49.315373 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:49.301370    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:33:49.343317 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:49.341000    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg" podStartSLOduration=2.340981541 podStartE2EDuration="2.340981541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:49.338175008 +0000 UTC m=+2924.385308961" watchObservedRunningTime="2026-10-30 13:33:49.340981541 +0000 UTC m=+2924.388115495"
Oct 30 13:33:49.618561 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:49.618524    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:50.307963 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:50.307918    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz" event={"ID":"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79","Type":"ContainerStarted","Data":"2cb39dca7182c76aa1d05a555653c8aa056776ba7487c90250ed1b74ff785902"}
Oct 30 13:33:50.845600 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:50.844808    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:33:50.845600 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:50.844939    4544 patch_prober.go:29] interesting pod/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body=
Oct 30 13:33:50.845600 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:50.844983    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz" podUID="4400fc6b-5d1c-4f3f-93be-7ef8540e8d79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused"
Oct 30 13:33:50.854726 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:50.854676    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k"]
Oct 30 13:33:50.934245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:50.913104    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:51.598957 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:51.598925    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:51.840515 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:51.840475    4544 patch_prober.go:29] interesting pod/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body=
Oct 30 13:33:51.840738 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:51.840541    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz" podUID="4400fc6b-5d1c-4f3f-93be-7ef8540e8d79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused"
Oct 30 13:33:51.939567 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:51.939457    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:52.259305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:52.252167    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:52.409305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:52.409215    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf" podUID="05286a92-510f-4ddf-acdc-1dab447686f1" containerName="main" containerID="cri-o://df24c80262d782620e2a635604d43ea6d7e2caaeba877419a49af5f7534cb600" gracePeriod=45
Oct 30 13:33:52.510999 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:52.510882    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf" podUID="05286a92-510f-4ddf-acdc-1dab447686f1" containerName="main" probeResult="failure" output="Get \"https://10.133.1.26:8000/ready\": read tcp 10.133.0.2:44888->10.133.1.26:8000: read: connection reset by peer"
Oct 30 13:33:52.861884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:52.858993    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:52.861884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:52.859323    4544 patch_prober.go:29] interesting pod/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body=
Oct 30 13:33:52.861884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:52.859378    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz" podUID="4400fc6b-5d1c-4f3f-93be-7ef8540e8d79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused"
Oct 30 13:33:52.889502 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:52.889460    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf"
Oct 30 13:33:52.988362 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:52.988328    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05286a92-510f-4ddf-acdc-1dab447686f1-tls-certs\" (UniqueName: \"kubernetes.io/secret/05286a92-510f-4ddf-acdc-1dab447686f1-tls-certs\") pod \"05286a92-510f-4ddf-acdc-1dab447686f1\" (UID: \"05286a92-510f-4ddf-acdc-1dab447686f1\") "
Oct 30 13:33:52.988528 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:52.988459    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05286a92-510f-4ddf-acdc-1dab447686f1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/05286a92-510f-4ddf-acdc-1dab447686f1-dshm\") pod \"05286a92-510f-4ddf-acdc-1dab447686f1\" (UID: \"05286a92-510f-4ddf-acdc-1dab447686f1\") "
Oct 30 13:33:52.988528 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:52.988490    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05286a92-510f-4ddf-acdc-1dab447686f1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05286a92-510f-4ddf-acdc-1dab447686f1-model-cache\") pod \"05286a92-510f-4ddf-acdc-1dab447686f1\" (UID: \"05286a92-510f-4ddf-acdc-1dab447686f1\") "
Oct 30 13:33:52.988630 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:52.988557    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05286a92-510f-4ddf-acdc-1dab447686f1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05286a92-510f-4ddf-acdc-1dab447686f1-tmp-dir\") pod \"05286a92-510f-4ddf-acdc-1dab447686f1\" (UID: \"05286a92-510f-4ddf-acdc-1dab447686f1\") "
Oct 30 13:33:52.988630 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:52.988577    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05286a92-510f-4ddf-acdc-1dab447686f1-home\" (UniqueName: \"kubernetes.io/empty-dir/05286a92-510f-4ddf-acdc-1dab447686f1-home\") pod \"05286a92-510f-4ddf-acdc-1dab447686f1\" (UID: \"05286a92-510f-4ddf-acdc-1dab447686f1\") "
Oct 30 13:33:52.988781 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:52.988755    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05286a92-510f-4ddf-acdc-1dab447686f1-kube-api-access-hp8h5\" (UniqueName: \"kubernetes.io/projected/05286a92-510f-4ddf-acdc-1dab447686f1-kube-api-access-hp8h5\") pod \"05286a92-510f-4ddf-acdc-1dab447686f1\" (UID: \"05286a92-510f-4ddf-acdc-1dab447686f1\") "
Oct 30 13:33:52.988896 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:52.988835    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05286a92-510f-4ddf-acdc-1dab447686f1-tmp-dir" pod "05286a92-510f-4ddf-acdc-1dab447686f1" (UID: "05286a92-510f-4ddf-acdc-1dab447686f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:33:52.989022 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:52.988926    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05286a92-510f-4ddf-acdc-1dab447686f1-home" pod "05286a92-510f-4ddf-acdc-1dab447686f1" (UID: "05286a92-510f-4ddf-acdc-1dab447686f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:33:52.989115 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:52.989090    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05286a92-510f-4ddf-acdc-1dab447686f1-model-cache" pod "05286a92-510f-4ddf-acdc-1dab447686f1" (UID: "05286a92-510f-4ddf-acdc-1dab447686f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:33:52.989576 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:52.989551    4544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05286a92-510f-4ddf-acdc-1dab447686f1-model-cache\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:33:52.989576 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:52.989578    4544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05286a92-510f-4ddf-acdc-1dab447686f1-tmp-dir\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:33:52.989771 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:52.989591    4544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05286a92-510f-4ddf-acdc-1dab447686f1-home\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:33:52.990776 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:52.990753    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05286a92-510f-4ddf-acdc-1dab447686f1-tls-certs" pod "05286a92-510f-4ddf-acdc-1dab447686f1" (UID: "05286a92-510f-4ddf-acdc-1dab447686f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:33:52.991391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:52.991364    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05286a92-510f-4ddf-acdc-1dab447686f1-dshm" pod "05286a92-510f-4ddf-acdc-1dab447686f1" (UID: "05286a92-510f-4ddf-acdc-1dab447686f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:33:52.991391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:52.991377    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05286a92-510f-4ddf-acdc-1dab447686f1-kube-api-access-hp8h5" pod "05286a92-510f-4ddf-acdc-1dab447686f1" (UID: "05286a92-510f-4ddf-acdc-1dab447686f1"). InnerVolumeSpecName "kube-api-access-hp8h5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:33:53.093480 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:53.093441    4544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05286a92-510f-4ddf-acdc-1dab447686f1-tls-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:33:53.093480 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:53.093485    4544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05286a92-510f-4ddf-acdc-1dab447686f1-dshm\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:33:53.093762 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:53.093513    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp8h5\" (UniqueName: \"kubernetes.io/projected/05286a92-510f-4ddf-acdc-1dab447686f1-kube-api-access-hp8h5\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:33:53.452173 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:53.452081    4544 generic.go:356] "Generic (PLEG): container finished" podID="05286a92-510f-4ddf-acdc-1dab447686f1" containerID="df24c80262d782620e2a635604d43ea6d7e2caaeba877419a49af5f7534cb600" exitCode=0
Oct 30 13:33:53.452173 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:53.452142    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf"
Oct 30 13:33:53.452401 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:53.452174    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf" event={"ID":"05286a92-510f-4ddf-acdc-1dab447686f1","Type":"ContainerDied","Data":"df24c80262d782620e2a635604d43ea6d7e2caaeba877419a49af5f7534cb600"}
Oct 30 13:33:53.452401 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:53.452213    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf" event={"ID":"05286a92-510f-4ddf-acdc-1dab447686f1","Type":"ContainerDied","Data":"9d1b01142f3d96ae9c4543e4d345a96ff688ed597480fb8fc46cfe55155ed912"}
Oct 30 13:33:53.452401 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:53.452247    4544 scope.go:122] "RemoveContainer" containerID="df24c80262d782620e2a635604d43ea6d7e2caaeba877419a49af5f7534cb600"
Oct 30 13:33:53.468956 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:53.468933    4544 scope.go:122] "RemoveContainer" containerID="df24c80262d782620e2a635604d43ea6d7e2caaeba877419a49af5f7534cb600"
Oct 30 13:33:53.469331 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:33:53.469304    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df24c80262d782620e2a635604d43ea6d7e2caaeba877419a49af5f7534cb600\": container with ID starting with df24c80262d782620e2a635604d43ea6d7e2caaeba877419a49af5f7534cb600 not found: ID does not exist" containerID="df24c80262d782620e2a635604d43ea6d7e2caaeba877419a49af5f7534cb600"
Oct 30 13:33:53.469416 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:53.469337    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df24c80262d782620e2a635604d43ea6d7e2caaeba877419a49af5f7534cb600"} err="failed to get container status \"df24c80262d782620e2a635604d43ea6d7e2caaeba877419a49af5f7534cb600\": rpc error: code = NotFound desc = could not find container \"df24c80262d782620e2a635604d43ea6d7e2caaeba877419a49af5f7534cb600\": container with ID starting with df24c80262d782620e2a635604d43ea6d7e2caaeba877419a49af5f7534cb600 not found: ID does not exist"
Oct 30 13:33:53.542164 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:53.542115    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf"]
Oct 30 13:33:53.547066 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:53.547030    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-8a5d5e/auth-test-model-8a5d5e-kserve-6889c49bb9-sdqcf"]
Oct 30 13:33:53.840999 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:53.840953    4544 patch_prober.go:29] interesting pod/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body=
Oct 30 13:33:53.841204 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:53.841021    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz" podUID="4400fc6b-5d1c-4f3f-93be-7ef8540e8d79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused"
Oct 30 13:33:54.110591 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:54.110479    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:54.840962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:54.840919    4544 patch_prober.go:29] interesting pod/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body=
Oct 30 13:33:54.841212 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:54.840976    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz" podUID="4400fc6b-5d1c-4f3f-93be-7ef8540e8d79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused"
Oct 30 13:33:55.091338 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:55.091201    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05286a92-510f-4ddf-acdc-1dab447686f1" path="/var/lib/kubelet/pods/05286a92-510f-4ddf-acdc-1dab447686f1/volumes"
Oct 30 13:33:55.841026 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:55.840663    4544 patch_prober.go:29] interesting pod/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body=
Oct 30 13:33:55.841026 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:55.840732    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz" podUID="4400fc6b-5d1c-4f3f-93be-7ef8540e8d79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused"
Oct 30 13:33:56.243720 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:56.243624    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:56.569084 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:56.568995    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:56.708419 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:56.708385    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:56.846801 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:56.846763    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:33:56.847303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:56.847283    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:33:56.848346 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:56.848327    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:33:57.610509 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:57.610473    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:58.289848 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:58.289814    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4"
Oct 30 13:33:58.290284 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:58.289886    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl"
Oct 30 13:33:58.489047 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:58.488951    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:58.740840 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:58.740748    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:58.936780 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:33:58.936744    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:33:59.138244 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:33:59.080533    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:34:00.312380 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:00.312348    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg"
Oct 30 13:34:00.996135 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:00.996088    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:03.084398 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:03.084355    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:05.078303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:05.078263    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:05.885927 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:05.885856    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k" podUID="dbeeada2-dca6-47fb-8b83-559703fc24f4" containerName="main" containerID="cri-o://54ae21511e7f6b6c16b8d856dff9432420915b131fa2ce61f00ad3709828dd94" gracePeriod=45
Oct 30 13:34:06.202093 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:06.202064    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k"
Oct 30 13:34:06.265951 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:06.265909    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbeeada2-dca6-47fb-8b83-559703fc24f4-home\" (UniqueName: \"kubernetes.io/empty-dir/dbeeada2-dca6-47fb-8b83-559703fc24f4-home\") pod \"dbeeada2-dca6-47fb-8b83-559703fc24f4\" (UID: \"dbeeada2-dca6-47fb-8b83-559703fc24f4\") "
Oct 30 13:34:06.266168 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:06.266006    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbeeada2-dca6-47fb-8b83-559703fc24f4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbeeada2-dca6-47fb-8b83-559703fc24f4-model-cache\") pod \"dbeeada2-dca6-47fb-8b83-559703fc24f4\" (UID: \"dbeeada2-dca6-47fb-8b83-559703fc24f4\") "
Oct 30 13:34:06.266168 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:06.266048    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbeeada2-dca6-47fb-8b83-559703fc24f4-kube-api-access-qhxxq\" (UniqueName: \"kubernetes.io/projected/dbeeada2-dca6-47fb-8b83-559703fc24f4-kube-api-access-qhxxq\") pod \"dbeeada2-dca6-47fb-8b83-559703fc24f4\" (UID: \"dbeeada2-dca6-47fb-8b83-559703fc24f4\") "
Oct 30 13:34:06.266168 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:06.266137    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbeeada2-dca6-47fb-8b83-559703fc24f4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbeeada2-dca6-47fb-8b83-559703fc24f4-dshm\") pod \"dbeeada2-dca6-47fb-8b83-559703fc24f4\" (UID: \"dbeeada2-dca6-47fb-8b83-559703fc24f4\") "
Oct 30 13:34:06.266409 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:06.266202    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbeeada2-dca6-47fb-8b83-559703fc24f4-home" pod "dbeeada2-dca6-47fb-8b83-559703fc24f4" (UID: "dbeeada2-dca6-47fb-8b83-559703fc24f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:34:06.266409 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:06.266331    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbeeada2-dca6-47fb-8b83-559703fc24f4-model-cache" pod "dbeeada2-dca6-47fb-8b83-559703fc24f4" (UID: "dbeeada2-dca6-47fb-8b83-559703fc24f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:34:06.266409 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:06.266339    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbeeada2-dca6-47fb-8b83-559703fc24f4-tls-certs\" (UniqueName: \"kubernetes.io/secret/dbeeada2-dca6-47fb-8b83-559703fc24f4-tls-certs\") pod \"dbeeada2-dca6-47fb-8b83-559703fc24f4\" (UID: \"dbeeada2-dca6-47fb-8b83-559703fc24f4\") "
Oct 30 13:34:06.266600 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:06.266412    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbeeada2-dca6-47fb-8b83-559703fc24f4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbeeada2-dca6-47fb-8b83-559703fc24f4-tmp-dir\") pod \"dbeeada2-dca6-47fb-8b83-559703fc24f4\" (UID: \"dbeeada2-dca6-47fb-8b83-559703fc24f4\") "
Oct 30 13:34:06.266775 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:06.266752    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbeeada2-dca6-47fb-8b83-559703fc24f4-tmp-dir" pod "dbeeada2-dca6-47fb-8b83-559703fc24f4" (UID: "dbeeada2-dca6-47fb-8b83-559703fc24f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:34:06.269274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:06.267838    4544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbeeada2-dca6-47fb-8b83-559703fc24f4-home\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:34:06.269274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:06.267867    4544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbeeada2-dca6-47fb-8b83-559703fc24f4-model-cache\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:34:06.269274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:06.267884    4544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbeeada2-dca6-47fb-8b83-559703fc24f4-tmp-dir\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:34:06.269274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:06.268868    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbeeada2-dca6-47fb-8b83-559703fc24f4-dshm" pod "dbeeada2-dca6-47fb-8b83-559703fc24f4" (UID: "dbeeada2-dca6-47fb-8b83-559703fc24f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:34:06.269274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:06.268950    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbeeada2-dca6-47fb-8b83-559703fc24f4-kube-api-access-qhxxq" pod "dbeeada2-dca6-47fb-8b83-559703fc24f4" (UID: "dbeeada2-dca6-47fb-8b83-559703fc24f4"). InnerVolumeSpecName "kube-api-access-qhxxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:34:06.269274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:06.269271    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbeeada2-dca6-47fb-8b83-559703fc24f4-tls-certs" pod "dbeeada2-dca6-47fb-8b83-559703fc24f4" (UID: "dbeeada2-dca6-47fb-8b83-559703fc24f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:34:06.369274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:06.369216    4544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbeeada2-dca6-47fb-8b83-559703fc24f4-tls-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:34:06.369274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:06.369267    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhxxq\" (UniqueName: \"kubernetes.io/projected/dbeeada2-dca6-47fb-8b83-559703fc24f4-kube-api-access-qhxxq\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:34:06.369274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:06.369277    4544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbeeada2-dca6-47fb-8b83-559703fc24f4-dshm\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:34:06.649597 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:06.649556    4544 generic.go:356] "Generic (PLEG): container finished" podID="dbeeada2-dca6-47fb-8b83-559703fc24f4" containerID="54ae21511e7f6b6c16b8d856dff9432420915b131fa2ce61f00ad3709828dd94" exitCode=0
Oct 30 13:34:06.649833 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:06.649606    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k"
Oct 30 13:34:06.649833 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:06.649643    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k" event={"ID":"dbeeada2-dca6-47fb-8b83-559703fc24f4","Type":"ContainerDied","Data":"54ae21511e7f6b6c16b8d856dff9432420915b131fa2ce61f00ad3709828dd94"}
Oct 30 13:34:06.649833 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:06.649693    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k" event={"ID":"dbeeada2-dca6-47fb-8b83-559703fc24f4","Type":"ContainerDied","Data":"195816db74933cdd7f9e5cb47bfd2c253f23727b8f943a8c3cdf0c508e2463ac"}
Oct 30 13:34:06.649833 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:06.649715    4544 scope.go:122] "RemoveContainer" containerID="54ae21511e7f6b6c16b8d856dff9432420915b131fa2ce61f00ad3709828dd94"
Oct 30 13:34:06.661845 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:06.661822    4544 scope.go:122] "RemoveContainer" containerID="54ae21511e7f6b6c16b8d856dff9432420915b131fa2ce61f00ad3709828dd94"
Oct 30 13:34:06.662253 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:34:06.662210    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54ae21511e7f6b6c16b8d856dff9432420915b131fa2ce61f00ad3709828dd94\": container with ID starting with 54ae21511e7f6b6c16b8d856dff9432420915b131fa2ce61f00ad3709828dd94 not found: ID does not exist" containerID="54ae21511e7f6b6c16b8d856dff9432420915b131fa2ce61f00ad3709828dd94"
Oct 30 13:34:06.662361 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:06.662264    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54ae21511e7f6b6c16b8d856dff9432420915b131fa2ce61f00ad3709828dd94"} err="failed to get container status \"54ae21511e7f6b6c16b8d856dff9432420915b131fa2ce61f00ad3709828dd94\": rpc error: code = NotFound desc = could not find container \"54ae21511e7f6b6c16b8d856dff9432420915b131fa2ce61f00ad3709828dd94\": container with ID starting with 54ae21511e7f6b6c16b8d856dff9432420915b131fa2ce61f00ad3709828dd94 not found: ID does not exist"
Oct 30 13:34:06.712161 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:06.712126    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k"]
Oct 30 13:34:06.717880 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:06.717846    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-e9d641/rate-test-model-e9d641-kserve-7d4f8b4865-swg9k"]
Oct 30 13:34:06.928682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:06.928583    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:07.091617 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:07.091576    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbeeada2-dca6-47fb-8b83-559703fc24f4" path="/var/lib/kubelet/pods/dbeeada2-dca6-47fb-8b83-559703fc24f4/volumes"
Oct 30 13:34:08.354056 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:08.354009    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp"]
Oct 30 13:34:08.399146 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:08.399104    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d"]
Oct 30 13:34:08.581459 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:08.581402    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:08.718004 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:08.717917    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:08.947708 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:08.947677    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:10.764825 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:10.764754    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:13.017787 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:13.017738    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:13.366353 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:13.366298    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp" podUID="0a5e1569-11e1-4247-ae2e-4507f8826474" containerName="payload-processing" containerID="cri-o://c84e8c806c8b026494947f3bc392f06a8663dc03c4f73bfec7813ea841e41ab6" gracePeriod=25
Oct 30 13:34:13.402375 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:13.402304    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d" podUID="29c1d897-b96c-45dd-a6cd-341b8abc28fe" containerName="payload-pre-processing" containerID="cri-o://f714a4fe8eea47da1571266e6dd5919ae1d876af94d56d36f4588697a7cc7bae" gracePeriod=25
Oct 30 13:34:13.778831 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:13.778793    4544 generic.go:356] "Generic (PLEG): container finished" podID="0a5e1569-11e1-4247-ae2e-4507f8826474" containerID="c84e8c806c8b026494947f3bc392f06a8663dc03c4f73bfec7813ea841e41ab6" exitCode=0
Oct 30 13:34:13.779107 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:13.779084    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp" event={"ID":"0a5e1569-11e1-4247-ae2e-4507f8826474","Type":"ContainerDied","Data":"c84e8c806c8b026494947f3bc392f06a8663dc03c4f73bfec7813ea841e41ab6"}
Oct 30 13:34:14.112790 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.112758    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp"
Oct 30 13:34:14.152468 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.152438    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d"
Oct 30 13:34:14.192590 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.192551    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a5e1569-11e1-4247-ae2e-4507f8826474-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a5e1569-11e1-4247-ae2e-4507f8826474-plugins-config-volume\") pod \"0a5e1569-11e1-4247-ae2e-4507f8826474\" (UID: \"0a5e1569-11e1-4247-ae2e-4507f8826474\") "
Oct 30 13:34:14.192819 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.192797    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a5e1569-11e1-4247-ae2e-4507f8826474-kube-api-access-wqxrn\" (UniqueName: \"kubernetes.io/projected/0a5e1569-11e1-4247-ae2e-4507f8826474-kube-api-access-wqxrn\") pod \"0a5e1569-11e1-4247-ae2e-4507f8826474\" (UID: \"0a5e1569-11e1-4247-ae2e-4507f8826474\") "
Oct 30 13:34:14.193104 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.193073    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a5e1569-11e1-4247-ae2e-4507f8826474-plugins-config-volume" pod "0a5e1569-11e1-4247-ae2e-4507f8826474" (UID: "0a5e1569-11e1-4247-ae2e-4507f8826474"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:34:14.193878 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.193850    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a5e1569-11e1-4247-ae2e-4507f8826474-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:34:14.195843 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.195813    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a5e1569-11e1-4247-ae2e-4507f8826474-kube-api-access-wqxrn" pod "0a5e1569-11e1-4247-ae2e-4507f8826474" (UID: "0a5e1569-11e1-4247-ae2e-4507f8826474"). InnerVolumeSpecName "kube-api-access-wqxrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:34:14.230683 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.230641    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"]
Oct 30 13:34:14.232164 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.232132    4544 cpu_manager.go:414] "containerMap: removing container" podUID="05286a92-510f-4ddf-acdc-1dab447686f1" containerName="main"
Oct 30 13:34:14.232164 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.232164    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05286a92-510f-4ddf-acdc-1dab447686f1" containerName="main"
Oct 30 13:34:14.232443 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.232209    4544 cpu_manager.go:414] "containerMap: removing container" podUID="0a5e1569-11e1-4247-ae2e-4507f8826474" containerName="payload-processing"
Oct 30 13:34:14.232443 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.232234    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a5e1569-11e1-4247-ae2e-4507f8826474" containerName="payload-processing"
Oct 30 13:34:14.232443 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.232251    4544 cpu_manager.go:414] "containerMap: removing container" podUID="29c1d897-b96c-45dd-a6cd-341b8abc28fe" containerName="payload-pre-processing"
Oct 30 13:34:14.232443 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.232259    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29c1d897-b96c-45dd-a6cd-341b8abc28fe" containerName="payload-pre-processing"
Oct 30 13:34:14.232443 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.232335    4544 cpu_manager.go:414] "containerMap: removing container" podUID="dbeeada2-dca6-47fb-8b83-559703fc24f4" containerName="main"
Oct 30 13:34:14.232443 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.232343    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbeeada2-dca6-47fb-8b83-559703fc24f4" containerName="main"
Oct 30 13:34:14.232791 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.232606    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="05286a92-510f-4ddf-acdc-1dab447686f1" containerName="main"
Oct 30 13:34:14.232791 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.232631    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a5e1569-11e1-4247-ae2e-4507f8826474" containerName="payload-processing"
Oct 30 13:34:14.232791 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.232641    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="29c1d897-b96c-45dd-a6cd-341b8abc28fe" containerName="payload-pre-processing"
Oct 30 13:34:14.232791 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.232665    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbeeada2-dca6-47fb-8b83-559703fc24f4" containerName="main"
Oct 30 13:34:14.238366 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.238336    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:34:14.252433 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.251544    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"]
Oct 30 13:34:14.294981 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.294893    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29c1d897-b96c-45dd-a6cd-341b8abc28fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c1d897-b96c-45dd-a6cd-341b8abc28fe-plugins-config-volume\") pod \"29c1d897-b96c-45dd-a6cd-341b8abc28fe\" (UID: \"29c1d897-b96c-45dd-a6cd-341b8abc28fe\") "
Oct 30 13:34:14.295139 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.295099    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29c1d897-b96c-45dd-a6cd-341b8abc28fe-kube-api-access-bd9xm\" (UniqueName: \"kubernetes.io/projected/29c1d897-b96c-45dd-a6cd-341b8abc28fe-kube-api-access-bd9xm\") pod \"29c1d897-b96c-45dd-a6cd-341b8abc28fe\" (UID: \"29c1d897-b96c-45dd-a6cd-341b8abc28fe\") "
Oct 30 13:34:14.295305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.295285    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29c1d897-b96c-45dd-a6cd-341b8abc28fe-plugins-config-volume" pod "29c1d897-b96c-45dd-a6cd-341b8abc28fe" (UID: "29c1d897-b96c-45dd-a6cd-341b8abc28fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:34:14.296257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.296193    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqxrn\" (UniqueName: \"kubernetes.io/projected/0a5e1569-11e1-4247-ae2e-4507f8826474-kube-api-access-wqxrn\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:34:14.296257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.296244    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c1d897-b96c-45dd-a6cd-341b8abc28fe-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:34:14.297210 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.297190    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c1d897-b96c-45dd-a6cd-341b8abc28fe-kube-api-access-bd9xm" pod "29c1d897-b96c-45dd-a6cd-341b8abc28fe" (UID: "29c1d897-b96c-45dd-a6cd-341b8abc28fe"). InnerVolumeSpecName "kube-api-access-bd9xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:34:14.398108 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.398069    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-workload-socket\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:34:14.398372 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.398134    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-workload-certs\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:34:14.398372 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.398208    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istiod-ca-cert\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:34:14.398372 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.398349    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istio-envoy\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:34:14.398503 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.398384    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istio-podinfo\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:34:14.398588 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.398575    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-credential-socket\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:34:14.398628 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.398606    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istio-data\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:34:14.398798 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.398779    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istio-token\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:34:14.398853 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.398837    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjhhc\" (UniqueName: \"kubernetes.io/projected/80b08e28-171e-49cd-bafe-0fbf4efc09e3-kube-api-access-zjhhc\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:34:14.398992 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.398977    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd9xm\" (UniqueName: \"kubernetes.io/projected/29c1d897-b96c-45dd-a6cd-341b8abc28fe-kube-api-access-bd9xm\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:34:14.500813 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.500766    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjhhc\" (UniqueName: \"kubernetes.io/projected/80b08e28-171e-49cd-bafe-0fbf4efc09e3-kube-api-access-zjhhc\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:34:14.501034 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.500870    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-workload-socket\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:34:14.501034 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.500912    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-workload-certs\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:34:14.501034 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.500932    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istiod-ca-cert\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:34:14.501034 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.501005    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istio-envoy\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:34:14.501301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.501040    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istio-podinfo\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:34:14.501378 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.501297    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-credential-socket\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:34:14.501378 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.501342    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istio-data\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:34:14.501488 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.501428    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-workload-socket\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:34:14.501547 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.501492    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-workload-certs\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:34:14.501610 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.501559    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istio-token\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:34:14.501892 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.501841    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istio-data\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:34:14.501892 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.501867    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-credential-socket\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:34:14.502149 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.502127    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istiod-ca-cert\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:34:14.503806 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.503776    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istio-envoy\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:34:14.503941 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.503917    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istio-podinfo\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:34:14.525079 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.525041    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istio-token\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:34:14.525292 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.525176    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjhhc\" (UniqueName: \"kubernetes.io/projected/80b08e28-171e-49cd-bafe-0fbf4efc09e3-kube-api-access-zjhhc\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:34:14.563605 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.563502    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:34:14.789415 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.789379    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp"
Oct 30 13:34:14.789662 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.789388    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp" event={"ID":"0a5e1569-11e1-4247-ae2e-4507f8826474","Type":"ContainerDied","Data":"5dcc06e08d7c7fc923f896e4ed77efc679a8df0935c0a643cfaf06df7c444386"}
Oct 30 13:34:14.789662 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.789540    4544 scope.go:122] "RemoveContainer" containerID="c84e8c806c8b026494947f3bc392f06a8663dc03c4f73bfec7813ea841e41ab6"
Oct 30 13:34:14.794536 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.794504    4544 generic.go:356] "Generic (PLEG): container finished" podID="29c1d897-b96c-45dd-a6cd-341b8abc28fe" containerID="f714a4fe8eea47da1571266e6dd5919ae1d876af94d56d36f4588697a7cc7bae" exitCode=0
Oct 30 13:34:14.794672 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.794560    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d"
Oct 30 13:34:14.794672 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.794567    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d" event={"ID":"29c1d897-b96c-45dd-a6cd-341b8abc28fe","Type":"ContainerDied","Data":"f714a4fe8eea47da1571266e6dd5919ae1d876af94d56d36f4588697a7cc7bae"}
Oct 30 13:34:14.794672 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.794618    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d" event={"ID":"29c1d897-b96c-45dd-a6cd-341b8abc28fe","Type":"ContainerDied","Data":"eb80b18c555d873a9e6bb326e406ca01be6632d54334d5c86776a710a3aea730"}
Oct 30 13:34:14.810801 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.810774    4544 scope.go:122] "RemoveContainer" containerID="f714a4fe8eea47da1571266e6dd5919ae1d876af94d56d36f4588697a7cc7bae"
Oct 30 13:34:14.821597 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.821574    4544 scope.go:122] "RemoveContainer" containerID="f714a4fe8eea47da1571266e6dd5919ae1d876af94d56d36f4588697a7cc7bae"
Oct 30 13:34:14.821960 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:34:14.821928    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f714a4fe8eea47da1571266e6dd5919ae1d876af94d56d36f4588697a7cc7bae\": container with ID starting with f714a4fe8eea47da1571266e6dd5919ae1d876af94d56d36f4588697a7cc7bae not found: ID does not exist" containerID="f714a4fe8eea47da1571266e6dd5919ae1d876af94d56d36f4588697a7cc7bae"
Oct 30 13:34:14.822030 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.821964    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f714a4fe8eea47da1571266e6dd5919ae1d876af94d56d36f4588697a7cc7bae"} err="failed to get container status \"f714a4fe8eea47da1571266e6dd5919ae1d876af94d56d36f4588697a7cc7bae\": rpc error: code = NotFound desc = could not find container \"f714a4fe8eea47da1571266e6dd5919ae1d876af94d56d36f4588697a7cc7bae\": container with ID starting with f714a4fe8eea47da1571266e6dd5919ae1d876af94d56d36f4588697a7cc7bae not found: ID does not exist"
Oct 30 13:34:14.848953 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.848918    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d"]
Oct 30 13:34:14.859154 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.859119    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d"]
Oct 30 13:34:14.886176 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.886138    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp"]
Oct 30 13:34:14.891672 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.891630    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp"]
Oct 30 13:34:14.904550 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.904518    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"]
Oct 30 13:34:14.906475 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:34:14.906361    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b08e28_171e_49cd_bafe_0fbf4efc09e3.slice/crio-29d7dbca3c7840b3f6c0b7ab312f36243ec00073e678cdade51f291c902c0836 WatchSource:0}: Error finding container 29d7dbca3c7840b3f6c0b7ab312f36243ec00073e678cdade51f291c902c0836: Status 404 returned error can't find the container with id 29d7dbca3c7840b3f6c0b7ab312f36243ec00073e678cdade51f291c902c0836
Oct 30 13:34:14.910516 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.910471    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:34:14.910622 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.910557    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:34:14.910622 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:14.910593    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:34:15.015731 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:15.015704    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:15.092348 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:15.092311    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a5e1569-11e1-4247-ae2e-4507f8826474" path="/var/lib/kubelet/pods/0a5e1569-11e1-4247-ae2e-4507f8826474/volumes"
Oct 30 13:34:15.092774 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:15.092753    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29c1d897-b96c-45dd-a6cd-341b8abc28fe" path="/var/lib/kubelet/pods/29c1d897-b96c-45dd-a6cd-341b8abc28fe/volumes"
Oct 30 13:34:15.847570 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:15.847529    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g" event={"ID":"80b08e28-171e-49cd-bafe-0fbf4efc09e3","Type":"ContainerStarted","Data":"caa31e59de79b72b8cc8b40f021564188634ab8d33cb4ce65747986721379178"}
Oct 30 13:34:15.848009 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:15.847612    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g" event={"ID":"80b08e28-171e-49cd-bafe-0fbf4efc09e3","Type":"ContainerStarted","Data":"29d7dbca3c7840b3f6c0b7ab312f36243ec00073e678cdade51f291c902c0836"}
Oct 30 13:34:15.936797 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:15.936723    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g" podStartSLOduration=22.936702064 podStartE2EDuration="22.936702064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:34:15.931976414 +0000 UTC m=+2950.979110368" watchObservedRunningTime="2026-10-30 13:34:15.936702064 +0000 UTC m=+2950.983836016"
Oct 30 13:34:16.564642 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:16.564603    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:34:16.566737 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:16.566706    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body=
Oct 30 13:34:16.566917 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:16.566758    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g" podUID="80b08e28-171e-49cd-bafe-0fbf4efc09e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused"
Oct 30 13:34:17.053545 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:17.053506    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:17.564821 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:17.564783    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body=
Oct 30 13:34:17.565018 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:17.564853    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g" podUID="80b08e28-171e-49cd-bafe-0fbf4efc09e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused"
Oct 30 13:34:18.563921 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:18.563881    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body=
Oct 30 13:34:18.564483 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:18.563945    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g" podUID="80b08e28-171e-49cd-bafe-0fbf4efc09e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused"
Oct 30 13:34:19.011908 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:19.011872    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:19.563958 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:19.563916    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body=
Oct 30 13:34:19.564459 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:19.563997    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g" podUID="80b08e28-171e-49cd-bafe-0fbf4efc09e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused"
Oct 30 13:34:20.564483 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:20.564439    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body=
Oct 30 13:34:20.564908 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:20.564513    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g" podUID="80b08e28-171e-49cd-bafe-0fbf4efc09e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused"
Oct 30 13:34:20.911143 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:20.911108    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:21.570080 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:21.570041    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:34:21.570564 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:21.570488    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:34:21.571265 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:21.571201    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:34:23.342393 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:23.342351    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:25.110112 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:25.110079    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:27.495419 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:27.495363    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:29.010696 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:34:29.010660    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:34:30.040308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:30.040268    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:31.827556 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:31.827491    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:33.660506 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:33.660462    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:35.062974 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:35.062933    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:35.314309 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:35.313813    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:36.403154 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:36.403117    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:38.252868 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:38.252826    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:40.045817 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:40.045745    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:40.159723 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:40.159685    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:40.292953 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:40.292899    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:42.131631 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:42.131589    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:43.346550 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:43.346509    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:43.775077 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:43.774987    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:43.891542 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:43.891509    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:44.012263 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:44.012208    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:45.724546 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:45.724510    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:47.529998 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:47.529960    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:49.299186 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:49.299120    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:51.219117 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:51.219073    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:52.123292 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:52.123255    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b1a6c9a-jmx47"]
Oct 30 13:34:52.129839 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:52.129790    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b1a6c9a-jmx47"
Oct 30 13:34:52.150689 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:52.150662    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b1a6c9a-jmx47"]
Oct 30 13:34:52.155993 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:52.155967    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:34:52.156294 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:52.156271    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:34:52.323658 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:52.323617    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjmx9\" (UniqueName: \"kubernetes.io/projected/fef0affe-c298-4c61-9740-fe7218c39162-kube-api-access-mjmx9\") pod \"maas-api-revoke-sub-5b1a6c9a-jmx47\" (UID: \"fef0affe-c298-4c61-9740-fe7218c39162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b1a6c9a-jmx47"
Oct 30 13:34:52.324109 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:52.323873    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef0affe-c298-4c61-9740-fe7218c39162-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b1a6c9a-jmx47\" (UID: \"fef0affe-c298-4c61-9740-fe7218c39162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b1a6c9a-jmx47"
Oct 30 13:34:52.324162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:52.324112    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef0affe-c298-4c61-9740-fe7218c39162-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b1a6c9a-jmx47\" (UID: \"fef0affe-c298-4c61-9740-fe7218c39162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b1a6c9a-jmx47"
Oct 30 13:34:52.425565 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:52.425473    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef0affe-c298-4c61-9740-fe7218c39162-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b1a6c9a-jmx47\" (UID: \"fef0affe-c298-4c61-9740-fe7218c39162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b1a6c9a-jmx47"
Oct 30 13:34:52.425749 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:52.425708    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef0affe-c298-4c61-9740-fe7218c39162-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b1a6c9a-jmx47\" (UID: \"fef0affe-c298-4c61-9740-fe7218c39162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b1a6c9a-jmx47"
Oct 30 13:34:52.425824 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:52.425764    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjmx9\" (UniqueName: \"kubernetes.io/projected/fef0affe-c298-4c61-9740-fe7218c39162-kube-api-access-mjmx9\") pod \"maas-api-revoke-sub-5b1a6c9a-jmx47\" (UID: \"fef0affe-c298-4c61-9740-fe7218c39162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b1a6c9a-jmx47"
Oct 30 13:34:52.426412 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:52.426386    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef0affe-c298-4c61-9740-fe7218c39162-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b1a6c9a-jmx47\" (UID: \"fef0affe-c298-4c61-9740-fe7218c39162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b1a6c9a-jmx47"
Oct 30 13:34:52.428106 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:52.428086    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef0affe-c298-4c61-9740-fe7218c39162-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b1a6c9a-jmx47\" (UID: \"fef0affe-c298-4c61-9740-fe7218c39162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b1a6c9a-jmx47"
Oct 30 13:34:52.451731 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:52.451694    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjmx9\" (UniqueName: \"kubernetes.io/projected/fef0affe-c298-4c61-9740-fe7218c39162-kube-api-access-mjmx9\") pod \"maas-api-revoke-sub-5b1a6c9a-jmx47\" (UID: \"fef0affe-c298-4c61-9740-fe7218c39162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b1a6c9a-jmx47"
Oct 30 13:34:52.746072 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:52.745976    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b1a6c9a-jmx47"
Oct 30 13:34:53.095958 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:53.095909    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:53.127358 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:53.127329    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b1a6c9a-jmx47"]
Oct 30 13:34:53.128456 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:34:53.128426    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfef0affe_c298_4c61_9740_fe7218c39162.slice/crio-77e87df51f140e20e752bd4ae1797841f1a1d16cd31bbfb5de1d20341c031775 WatchSource:0}: Error finding container 77e87df51f140e20e752bd4ae1797841f1a1d16cd31bbfb5de1d20341c031775: Status 404 returned error can't find the container with id 77e87df51f140e20e752bd4ae1797841f1a1d16cd31bbfb5de1d20341c031775
Oct 30 13:34:53.177858 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:53.177821    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b1a6c9a-jmx47" event={"ID":"fef0affe-c298-4c61-9740-fe7218c39162","Type":"ContainerStarted","Data":"77e87df51f140e20e752bd4ae1797841f1a1d16cd31bbfb5de1d20341c031775"}
Oct 30 13:34:53.473040 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:53.472924    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b30a2b9b-hhhkp"]
Oct 30 13:34:53.482895 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:53.482863    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30a2b9b-hhhkp"
Oct 30 13:34:53.497270 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:53.497195    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b30a2b9b-hhhkp"]
Oct 30 13:34:53.552689 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:53.552648    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kld2d\" (UniqueName: \"kubernetes.io/projected/677b185b-bc7d-4d32-87e7-a5b47de80407-kube-api-access-kld2d\") pod \"maas-api-revoke-sub-b30a2b9b-hhhkp\" (UID: \"677b185b-bc7d-4d32-87e7-a5b47de80407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30a2b9b-hhhkp"
Oct 30 13:34:53.552928 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:53.552892    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677b185b-bc7d-4d32-87e7-a5b47de80407-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b30a2b9b-hhhkp\" (UID: \"677b185b-bc7d-4d32-87e7-a5b47de80407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30a2b9b-hhhkp"
Oct 30 13:34:53.553053 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:53.553023    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677b185b-bc7d-4d32-87e7-a5b47de80407-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b30a2b9b-hhhkp\" (UID: \"677b185b-bc7d-4d32-87e7-a5b47de80407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30a2b9b-hhhkp"
Oct 30 13:34:53.655319 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:53.655204    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kld2d\" (UniqueName: \"kubernetes.io/projected/677b185b-bc7d-4d32-87e7-a5b47de80407-kube-api-access-kld2d\") pod \"maas-api-revoke-sub-b30a2b9b-hhhkp\" (UID: \"677b185b-bc7d-4d32-87e7-a5b47de80407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30a2b9b-hhhkp"
Oct 30 13:34:53.655540 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:53.655438    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677b185b-bc7d-4d32-87e7-a5b47de80407-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b30a2b9b-hhhkp\" (UID: \"677b185b-bc7d-4d32-87e7-a5b47de80407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30a2b9b-hhhkp"
Oct 30 13:34:53.655540 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:53.655503    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677b185b-bc7d-4d32-87e7-a5b47de80407-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b30a2b9b-hhhkp\" (UID: \"677b185b-bc7d-4d32-87e7-a5b47de80407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30a2b9b-hhhkp"
Oct 30 13:34:53.656342 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:53.656212    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677b185b-bc7d-4d32-87e7-a5b47de80407-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b30a2b9b-hhhkp\" (UID: \"677b185b-bc7d-4d32-87e7-a5b47de80407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30a2b9b-hhhkp"
Oct 30 13:34:53.658130 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:53.658104    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677b185b-bc7d-4d32-87e7-a5b47de80407-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b30a2b9b-hhhkp\" (UID: \"677b185b-bc7d-4d32-87e7-a5b47de80407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30a2b9b-hhhkp"
Oct 30 13:34:53.679377 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:53.679336    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kld2d\" (UniqueName: \"kubernetes.io/projected/677b185b-bc7d-4d32-87e7-a5b47de80407-kube-api-access-kld2d\") pod \"maas-api-revoke-sub-b30a2b9b-hhhkp\" (UID: \"677b185b-bc7d-4d32-87e7-a5b47de80407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30a2b9b-hhhkp"
Oct 30 13:34:53.803035 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:53.802939    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30a2b9b-hhhkp"
Oct 30 13:34:54.119288 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:54.119250    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b30a2b9b-hhhkp"]
Oct 30 13:34:54.120526 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:34:54.120496    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod677b185b_bc7d_4d32_87e7_a5b47de80407.slice/crio-2fb2710e2c1712fb5ab815a0a78799e205677304e08677224a89a02b44415320 WatchSource:0}: Error finding container 2fb2710e2c1712fb5ab815a0a78799e205677304e08677224a89a02b44415320: Status 404 returned error can't find the container with id 2fb2710e2c1712fb5ab815a0a78799e205677304e08677224a89a02b44415320
Oct 30 13:34:54.188626 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:54.188586    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b1a6c9a-jmx47" event={"ID":"fef0affe-c298-4c61-9740-fe7218c39162","Type":"ContainerStarted","Data":"27911f49001a65d44895b932c5f5b68092f9fa23b3867f2c3e2260aeb48e8eda"}
Oct 30 13:34:54.190109 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:54.190079    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30a2b9b-hhhkp" event={"ID":"677b185b-bc7d-4d32-87e7-a5b47de80407","Type":"ContainerStarted","Data":"2fb2710e2c1712fb5ab815a0a78799e205677304e08677224a89a02b44415320"}
Oct 30 13:34:54.230179 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:54.230129    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b1a6c9a-jmx47" podStartSLOduration=2.23011634 podStartE2EDuration="2.23011634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:34:54.228819742 +0000 UTC m=+2989.275953696" watchObservedRunningTime="2026-10-30 13:34:54.23011634 +0000 UTC m=+2989.277250293"
Oct 30 13:34:54.911051 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:54.911018    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:55.054858 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:55.054797    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:55.201332 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:55.201206    4544 generic.go:356] "Generic (PLEG): container finished" podID="fef0affe-c298-4c61-9740-fe7218c39162" containerID="27911f49001a65d44895b932c5f5b68092f9fa23b3867f2c3e2260aeb48e8eda" exitCode=0
Oct 30 13:34:55.201332 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:55.201274    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b1a6c9a-jmx47" event={"ID":"fef0affe-c298-4c61-9740-fe7218c39162","Type":"ContainerDied","Data":"27911f49001a65d44895b932c5f5b68092f9fa23b3867f2c3e2260aeb48e8eda"}
Oct 30 13:34:55.203137 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:55.203112    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30a2b9b-hhhkp" event={"ID":"677b185b-bc7d-4d32-87e7-a5b47de80407","Type":"ContainerStarted","Data":"e233384f8c5723049834601be4eb32fc061ba41351b790b423f7e7a4bf01f7b7"}
Oct 30 13:34:55.282973 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:55.282918    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30a2b9b-hhhkp" podStartSLOduration=2.28290375 podStartE2EDuration="2.28290375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:34:55.282463816 +0000 UTC m=+2990.329597769" watchObservedRunningTime="2026-10-30 13:34:55.28290375 +0000 UTC m=+2990.330037702"
Oct 30 13:34:55.303142 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:55.303106    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:56.316799 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:56.316755    4544 generic.go:356] "Generic (PLEG): container finished" podID="677b185b-bc7d-4d32-87e7-a5b47de80407" containerID="e233384f8c5723049834601be4eb32fc061ba41351b790b423f7e7a4bf01f7b7" exitCode=0
Oct 30 13:34:56.317616 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:56.317579    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30a2b9b-hhhkp" event={"ID":"677b185b-bc7d-4d32-87e7-a5b47de80407","Type":"ContainerDied","Data":"e233384f8c5723049834601be4eb32fc061ba41351b790b423f7e7a4bf01f7b7"}
Oct 30 13:34:56.543713 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:56.543681    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b1a6c9a-jmx47"
Oct 30 13:34:56.635759 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:56.635716    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fef0affe-c298-4c61-9740-fe7218c39162-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef0affe-c298-4c61-9740-fe7218c39162-maas-api-xks-ca\") pod \"fef0affe-c298-4c61-9740-fe7218c39162\" (UID: \"fef0affe-c298-4c61-9740-fe7218c39162\") "
Oct 30 13:34:56.636014 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:56.635990    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fef0affe-c298-4c61-9740-fe7218c39162-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef0affe-c298-4c61-9740-fe7218c39162-maas-api-openshift-ca\") pod \"fef0affe-c298-4c61-9740-fe7218c39162\" (UID: \"fef0affe-c298-4c61-9740-fe7218c39162\") "
Oct 30 13:34:56.636327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:56.636304    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fef0affe-c298-4c61-9740-fe7218c39162-kube-api-access-mjmx9\" (UniqueName: \"kubernetes.io/projected/fef0affe-c298-4c61-9740-fe7218c39162-kube-api-access-mjmx9\") pod \"fef0affe-c298-4c61-9740-fe7218c39162\" (UID: \"fef0affe-c298-4c61-9740-fe7218c39162\") "
Oct 30 13:34:56.636456 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:56.636392    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fef0affe-c298-4c61-9740-fe7218c39162-maas-api-openshift-ca" pod "fef0affe-c298-4c61-9740-fe7218c39162" (UID: "fef0affe-c298-4c61-9740-fe7218c39162"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:34:56.637126 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:56.637098    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef0affe-c298-4c61-9740-fe7218c39162-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:34:56.638137 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:56.638101    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fef0affe-c298-4c61-9740-fe7218c39162-maas-api-xks-ca" pod "fef0affe-c298-4c61-9740-fe7218c39162" (UID: "fef0affe-c298-4c61-9740-fe7218c39162"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:34:56.638937 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:56.638911    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fef0affe-c298-4c61-9740-fe7218c39162-kube-api-access-mjmx9" pod "fef0affe-c298-4c61-9740-fe7218c39162" (UID: "fef0affe-c298-4c61-9740-fe7218c39162"). InnerVolumeSpecName "kube-api-access-mjmx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:34:56.739039 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:56.739003    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjmx9\" (UniqueName: \"kubernetes.io/projected/fef0affe-c298-4c61-9740-fe7218c39162-kube-api-access-mjmx9\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:34:56.739039 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:56.739039    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef0affe-c298-4c61-9740-fe7218c39162-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:34:56.788724 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:56.788693    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:57.327618 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:57.327580    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b1a6c9a-jmx47"
Oct 30 13:34:57.328036 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:57.327612    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b1a6c9a-jmx47" event={"ID":"fef0affe-c298-4c61-9740-fe7218c39162","Type":"ContainerDied","Data":"77e87df51f140e20e752bd4ae1797841f1a1d16cd31bbfb5de1d20341c031775"}
Oct 30 13:34:57.328036 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:57.327665    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77e87df51f140e20e752bd4ae1797841f1a1d16cd31bbfb5de1d20341c031775"
Oct 30 13:34:57.537817 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:57.537792    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30a2b9b-hhhkp"
Oct 30 13:34:57.695269 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:57.695143    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/677b185b-bc7d-4d32-87e7-a5b47de80407-kube-api-access-kld2d\" (UniqueName: \"kubernetes.io/projected/677b185b-bc7d-4d32-87e7-a5b47de80407-kube-api-access-kld2d\") pod \"677b185b-bc7d-4d32-87e7-a5b47de80407\" (UID: \"677b185b-bc7d-4d32-87e7-a5b47de80407\") "
Oct 30 13:34:57.695269 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:57.695205    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/677b185b-bc7d-4d32-87e7-a5b47de80407-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677b185b-bc7d-4d32-87e7-a5b47de80407-maas-api-openshift-ca\") pod \"677b185b-bc7d-4d32-87e7-a5b47de80407\" (UID: \"677b185b-bc7d-4d32-87e7-a5b47de80407\") "
Oct 30 13:34:57.695476 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:57.695333    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/677b185b-bc7d-4d32-87e7-a5b47de80407-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677b185b-bc7d-4d32-87e7-a5b47de80407-maas-api-xks-ca\") pod \"677b185b-bc7d-4d32-87e7-a5b47de80407\" (UID: \"677b185b-bc7d-4d32-87e7-a5b47de80407\") "
Oct 30 13:34:57.695702 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:57.695675    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/677b185b-bc7d-4d32-87e7-a5b47de80407-maas-api-openshift-ca" pod "677b185b-bc7d-4d32-87e7-a5b47de80407" (UID: "677b185b-bc7d-4d32-87e7-a5b47de80407"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:34:57.696310 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:57.696290    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677b185b-bc7d-4d32-87e7-a5b47de80407-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:34:57.697560 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:57.697538    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/677b185b-bc7d-4d32-87e7-a5b47de80407-maas-api-xks-ca" pod "677b185b-bc7d-4d32-87e7-a5b47de80407" (UID: "677b185b-bc7d-4d32-87e7-a5b47de80407"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:34:57.697655 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:57.697629    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/677b185b-bc7d-4d32-87e7-a5b47de80407-kube-api-access-kld2d" pod "677b185b-bc7d-4d32-87e7-a5b47de80407" (UID: "677b185b-bc7d-4d32-87e7-a5b47de80407"). InnerVolumeSpecName "kube-api-access-kld2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:34:57.737886 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:57.737849    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:57.798295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:57.798259    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kld2d\" (UniqueName: \"kubernetes.io/projected/677b185b-bc7d-4d32-87e7-a5b47de80407-kube-api-access-kld2d\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:34:57.798295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:57.798294    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677b185b-bc7d-4d32-87e7-a5b47de80407-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:34:58.337158 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:58.337130    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30a2b9b-hhhkp"
Oct 30 13:34:58.337607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:58.337166    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30a2b9b-hhhkp" event={"ID":"677b185b-bc7d-4d32-87e7-a5b47de80407","Type":"ContainerDied","Data":"2fb2710e2c1712fb5ab815a0a78799e205677304e08677224a89a02b44415320"}
Oct 30 13:34:58.337607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:58.337252    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fb2710e2c1712fb5ab815a0a78799e205677304e08677224a89a02b44415320"
Oct 30 13:34:58.511166 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:34:58.511075    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:34:59.009896 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:34:59.009863    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:35:00.445445 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:00.445411    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:01.804708 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:01.804672    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:02.023249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:02.010019    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:02.677600 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:02.677561    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f48014e-ffspr"]
Oct 30 13:35:02.678630 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:02.678612    4544 cpu_manager.go:414] "containerMap: removing container" podUID="677b185b-bc7d-4d32-87e7-a5b47de80407" containerName="revoke-keys"
Oct 30 13:35:02.678630 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:02.678632    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="677b185b-bc7d-4d32-87e7-a5b47de80407" containerName="revoke-keys"
Oct 30 13:35:02.678737 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:02.678664    4544 cpu_manager.go:414] "containerMap: removing container" podUID="fef0affe-c298-4c61-9740-fe7218c39162" containerName="revoke-keys"
Oct 30 13:35:02.678737 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:02.678670    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fef0affe-c298-4c61-9740-fe7218c39162" containerName="revoke-keys"
Oct 30 13:35:02.678856 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:02.678844    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="677b185b-bc7d-4d32-87e7-a5b47de80407" containerName="revoke-keys"
Oct 30 13:35:02.678892 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:02.678864    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="fef0affe-c298-4c61-9740-fe7218c39162" containerName="revoke-keys"
Oct 30 13:35:02.683110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:02.683092    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f48014e-ffspr"
Oct 30 13:35:02.704107 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:02.704067    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f48014e-ffspr"]
Oct 30 13:35:02.711294 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:02.709666    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:35:02.711294 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:02.709753    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:35:02.760432 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:02.760389    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40943ed5-3bb8-4e6d-830a-b15671e1c0aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f48014e-ffspr\" (UID: \"40943ed5-3bb8-4e6d-830a-b15671e1c0aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f48014e-ffspr"
Oct 30 13:35:02.760432 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:02.760435    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft5l2\" (UniqueName: \"kubernetes.io/projected/40943ed5-3bb8-4e6d-830a-b15671e1c0aa-kube-api-access-ft5l2\") pod \"maas-api-revoke-sub-5f48014e-ffspr\" (UID: \"40943ed5-3bb8-4e6d-830a-b15671e1c0aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f48014e-ffspr"
Oct 30 13:35:02.760903 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:02.760870    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40943ed5-3bb8-4e6d-830a-b15671e1c0aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f48014e-ffspr\" (UID: \"40943ed5-3bb8-4e6d-830a-b15671e1c0aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f48014e-ffspr"
Oct 30 13:35:02.862620 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:02.862573    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40943ed5-3bb8-4e6d-830a-b15671e1c0aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f48014e-ffspr\" (UID: \"40943ed5-3bb8-4e6d-830a-b15671e1c0aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f48014e-ffspr"
Oct 30 13:35:02.863175 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:02.862715    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40943ed5-3bb8-4e6d-830a-b15671e1c0aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f48014e-ffspr\" (UID: \"40943ed5-3bb8-4e6d-830a-b15671e1c0aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f48014e-ffspr"
Oct 30 13:35:02.863175 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:02.862742    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft5l2\" (UniqueName: \"kubernetes.io/projected/40943ed5-3bb8-4e6d-830a-b15671e1c0aa-kube-api-access-ft5l2\") pod \"maas-api-revoke-sub-5f48014e-ffspr\" (UID: \"40943ed5-3bb8-4e6d-830a-b15671e1c0aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f48014e-ffspr"
Oct 30 13:35:02.863502 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:02.863478    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40943ed5-3bb8-4e6d-830a-b15671e1c0aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f48014e-ffspr\" (UID: \"40943ed5-3bb8-4e6d-830a-b15671e1c0aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f48014e-ffspr"
Oct 30 13:35:02.865455 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:02.865430    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40943ed5-3bb8-4e6d-830a-b15671e1c0aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f48014e-ffspr\" (UID: \"40943ed5-3bb8-4e6d-830a-b15671e1c0aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f48014e-ffspr"
Oct 30 13:35:02.886688 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:02.886654    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft5l2\" (UniqueName: \"kubernetes.io/projected/40943ed5-3bb8-4e6d-830a-b15671e1c0aa-kube-api-access-ft5l2\") pod \"maas-api-revoke-sub-5f48014e-ffspr\" (UID: \"40943ed5-3bb8-4e6d-830a-b15671e1c0aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f48014e-ffspr"
Oct 30 13:35:02.998555 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:02.998473    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f48014e-ffspr"
Oct 30 13:35:03.175626 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:03.175590    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:03.318168 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:03.318142    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f48014e-ffspr"]
Oct 30 13:35:03.321197 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:35:03.321168    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40943ed5_3bb8_4e6d_830a_b15671e1c0aa.slice/crio-ab540800664bdeae319f28047b0ce74c9053628380553eec376e5f58072b04a9 WatchSource:0}: Error finding container ab540800664bdeae319f28047b0ce74c9053628380553eec376e5f58072b04a9: Status 404 returned error can't find the container with id ab540800664bdeae319f28047b0ce74c9053628380553eec376e5f58072b04a9
Oct 30 13:35:03.550057 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:03.550017    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f48014e-ffspr" event={"ID":"40943ed5-3bb8-4e6d-830a-b15671e1c0aa","Type":"ContainerStarted","Data":"ab540800664bdeae319f28047b0ce74c9053628380553eec376e5f58072b04a9"}
Oct 30 13:35:03.783482 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:03.783448    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4334d165-nlzsw"]
Oct 30 13:35:03.788608 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:03.788586    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4334d165-nlzsw"
Oct 30 13:35:03.809234 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:03.809199    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4334d165-nlzsw"]
Oct 30 13:35:03.880178 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:03.880076    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/606e1cf8-1e8b-49d9-bc90-6e2403bc59a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4334d165-nlzsw\" (UID: \"606e1cf8-1e8b-49d9-bc90-6e2403bc59a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4334d165-nlzsw"
Oct 30 13:35:03.880675 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:03.880525    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bw4v\" (UniqueName: \"kubernetes.io/projected/606e1cf8-1e8b-49d9-bc90-6e2403bc59a7-kube-api-access-6bw4v\") pod \"maas-api-revoke-sub-4334d165-nlzsw\" (UID: \"606e1cf8-1e8b-49d9-bc90-6e2403bc59a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4334d165-nlzsw"
Oct 30 13:35:03.880675 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:03.880570    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/606e1cf8-1e8b-49d9-bc90-6e2403bc59a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4334d165-nlzsw\" (UID: \"606e1cf8-1e8b-49d9-bc90-6e2403bc59a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4334d165-nlzsw"
Oct 30 13:35:03.982133 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:03.982091    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bw4v\" (UniqueName: \"kubernetes.io/projected/606e1cf8-1e8b-49d9-bc90-6e2403bc59a7-kube-api-access-6bw4v\") pod \"maas-api-revoke-sub-4334d165-nlzsw\" (UID: \"606e1cf8-1e8b-49d9-bc90-6e2403bc59a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4334d165-nlzsw"
Oct 30 13:35:03.982346 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:03.982137    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/606e1cf8-1e8b-49d9-bc90-6e2403bc59a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4334d165-nlzsw\" (UID: \"606e1cf8-1e8b-49d9-bc90-6e2403bc59a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4334d165-nlzsw"
Oct 30 13:35:03.982346 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:03.982283    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/606e1cf8-1e8b-49d9-bc90-6e2403bc59a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4334d165-nlzsw\" (UID: \"606e1cf8-1e8b-49d9-bc90-6e2403bc59a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4334d165-nlzsw"
Oct 30 13:35:03.983090 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:03.983067    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/606e1cf8-1e8b-49d9-bc90-6e2403bc59a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4334d165-nlzsw\" (UID: \"606e1cf8-1e8b-49d9-bc90-6e2403bc59a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4334d165-nlzsw"
Oct 30 13:35:03.984781 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:03.984750    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/606e1cf8-1e8b-49d9-bc90-6e2403bc59a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4334d165-nlzsw\" (UID: \"606e1cf8-1e8b-49d9-bc90-6e2403bc59a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4334d165-nlzsw"
Oct 30 13:35:04.006294 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:04.006259    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bw4v\" (UniqueName: \"kubernetes.io/projected/606e1cf8-1e8b-49d9-bc90-6e2403bc59a7-kube-api-access-6bw4v\") pod \"maas-api-revoke-sub-4334d165-nlzsw\" (UID: \"606e1cf8-1e8b-49d9-bc90-6e2403bc59a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4334d165-nlzsw"
Oct 30 13:35:04.104008 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:04.103977    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4334d165-nlzsw"
Oct 30 13:35:04.417608 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:04.417579    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4334d165-nlzsw"]
Oct 30 13:35:04.419095 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:35:04.419068    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod606e1cf8_1e8b_49d9_bc90_6e2403bc59a7.slice/crio-bd7e4b3b55edbc69b20bbf0ff581d6e9c6561cc6966dd2ca8dead294e0ab0d87 WatchSource:0}: Error finding container bd7e4b3b55edbc69b20bbf0ff581d6e9c6561cc6966dd2ca8dead294e0ab0d87: Status 404 returned error can't find the container with id bd7e4b3b55edbc69b20bbf0ff581d6e9c6561cc6966dd2ca8dead294e0ab0d87
Oct 30 13:35:04.567041 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:04.567003    4544 generic.go:356] "Generic (PLEG): container finished" podID="40943ed5-3bb8-4e6d-830a-b15671e1c0aa" containerID="5a6c7f38649a5178d037c95c7920401ea42dc0376a50a3df90a6ea2f48edacb4" exitCode=0
Oct 30 13:35:04.567205 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:04.567083    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f48014e-ffspr" event={"ID":"40943ed5-3bb8-4e6d-830a-b15671e1c0aa","Type":"ContainerDied","Data":"5a6c7f38649a5178d037c95c7920401ea42dc0376a50a3df90a6ea2f48edacb4"}
Oct 30 13:35:04.568852 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:04.568822    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4334d165-nlzsw" event={"ID":"606e1cf8-1e8b-49d9-bc90-6e2403bc59a7","Type":"ContainerStarted","Data":"bd7e4b3b55edbc69b20bbf0ff581d6e9c6561cc6966dd2ca8dead294e0ab0d87"}
Oct 30 13:35:04.901540 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:04.901507    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:05.578476 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:05.578443    4544 generic.go:356] "Generic (PLEG): container finished" podID="606e1cf8-1e8b-49d9-bc90-6e2403bc59a7" containerID="18be4a2509897282841202e22ae92eafe6b361bb348b776a16223ba6e0634ff7" exitCode=0
Oct 30 13:35:05.578661 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:05.578528    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4334d165-nlzsw" event={"ID":"606e1cf8-1e8b-49d9-bc90-6e2403bc59a7","Type":"ContainerDied","Data":"18be4a2509897282841202e22ae92eafe6b361bb348b776a16223ba6e0634ff7"}
Oct 30 13:35:05.756045 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:05.756015    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f48014e-ffspr"
Oct 30 13:35:05.920370 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:05.920339    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40943ed5-3bb8-4e6d-830a-b15671e1c0aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40943ed5-3bb8-4e6d-830a-b15671e1c0aa-maas-api-xks-ca\") pod \"40943ed5-3bb8-4e6d-830a-b15671e1c0aa\" (UID: \"40943ed5-3bb8-4e6d-830a-b15671e1c0aa\") "
Oct 30 13:35:05.920857 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:05.920387    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40943ed5-3bb8-4e6d-830a-b15671e1c0aa-kube-api-access-ft5l2\" (UniqueName: \"kubernetes.io/projected/40943ed5-3bb8-4e6d-830a-b15671e1c0aa-kube-api-access-ft5l2\") pod \"40943ed5-3bb8-4e6d-830a-b15671e1c0aa\" (UID: \"40943ed5-3bb8-4e6d-830a-b15671e1c0aa\") "
Oct 30 13:35:05.920857 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:05.920744    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40943ed5-3bb8-4e6d-830a-b15671e1c0aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40943ed5-3bb8-4e6d-830a-b15671e1c0aa-maas-api-openshift-ca\") pod \"40943ed5-3bb8-4e6d-830a-b15671e1c0aa\" (UID: \"40943ed5-3bb8-4e6d-830a-b15671e1c0aa\") "
Oct 30 13:35:05.921131 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:05.921104    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40943ed5-3bb8-4e6d-830a-b15671e1c0aa-maas-api-openshift-ca" pod "40943ed5-3bb8-4e6d-830a-b15671e1c0aa" (UID: "40943ed5-3bb8-4e6d-830a-b15671e1c0aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:35:05.921571 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:05.921547    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40943ed5-3bb8-4e6d-830a-b15671e1c0aa-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:35:05.922825 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:05.922783    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40943ed5-3bb8-4e6d-830a-b15671e1c0aa-maas-api-xks-ca" pod "40943ed5-3bb8-4e6d-830a-b15671e1c0aa" (UID: "40943ed5-3bb8-4e6d-830a-b15671e1c0aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:35:05.922825 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:05.922780    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40943ed5-3bb8-4e6d-830a-b15671e1c0aa-kube-api-access-ft5l2" pod "40943ed5-3bb8-4e6d-830a-b15671e1c0aa" (UID: "40943ed5-3bb8-4e6d-830a-b15671e1c0aa"). InnerVolumeSpecName "kube-api-access-ft5l2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:06.023653 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:06.023570    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40943ed5-3bb8-4e6d-830a-b15671e1c0aa-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:35:06.023653 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:06.023601    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft5l2\" (UniqueName: \"kubernetes.io/projected/40943ed5-3bb8-4e6d-830a-b15671e1c0aa-kube-api-access-ft5l2\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:35:06.590916 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:06.590878    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f48014e-ffspr" event={"ID":"40943ed5-3bb8-4e6d-830a-b15671e1c0aa","Type":"ContainerDied","Data":"ab540800664bdeae319f28047b0ce74c9053628380553eec376e5f58072b04a9"}
Oct 30 13:35:06.591146 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:06.590924    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f48014e-ffspr"
Oct 30 13:35:06.591146 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:06.590937    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab540800664bdeae319f28047b0ce74c9053628380553eec376e5f58072b04a9"
Oct 30 13:35:06.773824 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:06.773796    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4334d165-nlzsw"
Oct 30 13:35:06.793381 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:06.793340    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:06.940382 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:06.940275    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/606e1cf8-1e8b-49d9-bc90-6e2403bc59a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/606e1cf8-1e8b-49d9-bc90-6e2403bc59a7-maas-api-xks-ca\") pod \"606e1cf8-1e8b-49d9-bc90-6e2403bc59a7\" (UID: \"606e1cf8-1e8b-49d9-bc90-6e2403bc59a7\") "
Oct 30 13:35:06.940382 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:06.940346    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/606e1cf8-1e8b-49d9-bc90-6e2403bc59a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/606e1cf8-1e8b-49d9-bc90-6e2403bc59a7-maas-api-openshift-ca\") pod \"606e1cf8-1e8b-49d9-bc90-6e2403bc59a7\" (UID: \"606e1cf8-1e8b-49d9-bc90-6e2403bc59a7\") "
Oct 30 13:35:06.940902 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:06.940676    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/606e1cf8-1e8b-49d9-bc90-6e2403bc59a7-kube-api-access-6bw4v\" (UniqueName: \"kubernetes.io/projected/606e1cf8-1e8b-49d9-bc90-6e2403bc59a7-kube-api-access-6bw4v\") pod \"606e1cf8-1e8b-49d9-bc90-6e2403bc59a7\" (UID: \"606e1cf8-1e8b-49d9-bc90-6e2403bc59a7\") "
Oct 30 13:35:06.940902 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:06.940780    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/606e1cf8-1e8b-49d9-bc90-6e2403bc59a7-maas-api-openshift-ca" pod "606e1cf8-1e8b-49d9-bc90-6e2403bc59a7" (UID: "606e1cf8-1e8b-49d9-bc90-6e2403bc59a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:35:06.941630 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:06.941605    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/606e1cf8-1e8b-49d9-bc90-6e2403bc59a7-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:35:06.942823 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:06.942801    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/606e1cf8-1e8b-49d9-bc90-6e2403bc59a7-kube-api-access-6bw4v" pod "606e1cf8-1e8b-49d9-bc90-6e2403bc59a7" (UID: "606e1cf8-1e8b-49d9-bc90-6e2403bc59a7"). InnerVolumeSpecName "kube-api-access-6bw4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:06.942904 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:06.942821    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/606e1cf8-1e8b-49d9-bc90-6e2403bc59a7-maas-api-xks-ca" pod "606e1cf8-1e8b-49d9-bc90-6e2403bc59a7" (UID: "606e1cf8-1e8b-49d9-bc90-6e2403bc59a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:35:07.043153 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:07.043116    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/606e1cf8-1e8b-49d9-bc90-6e2403bc59a7-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:35:07.043153 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:07.043147    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bw4v\" (UniqueName: \"kubernetes.io/projected/606e1cf8-1e8b-49d9-bc90-6e2403bc59a7-kube-api-access-6bw4v\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:35:07.604860 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:07.604800    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4334d165-nlzsw" event={"ID":"606e1cf8-1e8b-49d9-bc90-6e2403bc59a7","Type":"ContainerDied","Data":"bd7e4b3b55edbc69b20bbf0ff581d6e9c6561cc6966dd2ca8dead294e0ab0d87"}
Oct 30 13:35:07.605081 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:07.604917    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd7e4b3b55edbc69b20bbf0ff581d6e9c6561cc6966dd2ca8dead294e0ab0d87"
Oct 30 13:35:07.605081 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:07.605028    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4334d165-nlzsw"
Oct 30 13:35:08.491016 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:08.490979    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:10.232461 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:10.232426    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:12.323518 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:12.323483    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:14.238146 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:14.238114    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:15.318656 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:15.318614    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:15.538954 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:15.538917    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:16.148024 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:16.147973    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:18.192791 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:18.192717    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:20.584288 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:20.577043    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:23.010413 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:23.010364    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:25.113307 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:25.113270    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:26.463368 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:26.463336    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:27.122496 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:27.122455    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:27.576987 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:27.576943    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:27.757244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:27.756783    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:27.901990 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:27.901904    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:29.010541 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:35:29.010492    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:35:29.969112 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:29.969076    4544 generic.go:356] "Generic (PLEG): container finished" podID="c713af1f-1695-4ad5-a564-25420ba533a6" containerID="89b11d4dfe81eb377fa0657c55a2f4f775b613b4b6f4a86b7e6a1f139cffb103" exitCode=0
Oct 30 13:35:29.969370 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:29.969119    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd" event={"ID":"c713af1f-1695-4ad5-a564-25420ba533a6","Type":"ContainerDied","Data":"89b11d4dfe81eb377fa0657c55a2f4f775b613b4b6f4a86b7e6a1f139cffb103"}
Oct 30 13:35:29.969638 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:29.969617    4544 scope.go:122] "RemoveContainer" containerID="89b11d4dfe81eb377fa0657c55a2f4f775b613b4b6f4a86b7e6a1f139cffb103"
Oct 30 13:35:29.971902 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:29.971866    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:35:29.971990 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:29.971948    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:35:29.971990 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:29.971982    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:35:29.997577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:29.997546    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:30.980238 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:30.980195    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd" event={"ID":"c713af1f-1695-4ad5-a564-25420ba533a6","Type":"ContainerStarted","Data":"b2bafbd277d63f97ad6ce60e32d6f9749ab6f4bf470148aa97716768d9033831"}
Oct 30 13:35:31.959492 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:31.959458    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:31.963311 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:31.963288    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:35:31.963519 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:31.963496    4544 patch_prober.go:29] interesting pod/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused" start-of-body=
Oct 30 13:35:31.963602 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:31.963581    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd" podUID="c713af1f-1695-4ad5-a564-25420ba533a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused"
Oct 30 13:35:32.962612 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:32.962579    4544 patch_prober.go:29] interesting pod/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused" start-of-body=
Oct 30 13:35:32.963004 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:32.962630    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd" podUID="c713af1f-1695-4ad5-a564-25420ba533a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused"
Oct 30 13:35:33.799141 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:33.799100    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:33.963359 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:33.963327    4544 patch_prober.go:29] interesting pod/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused" start-of-body=
Oct 30 13:35:33.963739 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:33.963387    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd" podUID="c713af1f-1695-4ad5-a564-25420ba533a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused"
Oct 30 13:35:34.566080 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:34.566050    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:34.963205 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:34.963158    4544 patch_prober.go:29] interesting pod/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused" start-of-body=
Oct 30 13:35:34.963427 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:34.963277    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd" podUID="c713af1f-1695-4ad5-a564-25420ba533a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused"
Oct 30 13:35:35.049165 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:35.049120    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ba901-mqfx9"]
Oct 30 13:35:35.050618 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:35.050590    4544 cpu_manager.go:414] "containerMap: removing container" podUID="40943ed5-3bb8-4e6d-830a-b15671e1c0aa" containerName="revoke-keys"
Oct 30 13:35:35.050820 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:35.050803    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40943ed5-3bb8-4e6d-830a-b15671e1c0aa" containerName="revoke-keys"
Oct 30 13:35:35.050996 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:35.050984    4544 cpu_manager.go:414] "containerMap: removing container" podUID="606e1cf8-1e8b-49d9-bc90-6e2403bc59a7" containerName="revoke-keys"
Oct 30 13:35:35.051103 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:35.051078    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="606e1cf8-1e8b-49d9-bc90-6e2403bc59a7" containerName="revoke-keys"
Oct 30 13:35:35.051507 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:35.051489    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="606e1cf8-1e8b-49d9-bc90-6e2403bc59a7" containerName="revoke-keys"
Oct 30 13:35:35.051672 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:35.051658    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="40943ed5-3bb8-4e6d-830a-b15671e1c0aa" containerName="revoke-keys"
Oct 30 13:35:35.056806 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:35.056789    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ba901-mqfx9"
Oct 30 13:35:35.074103 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:35.073823    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ba901-mqfx9"]
Oct 30 13:35:35.078849 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:35.078823    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:35:35.079035 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:35.079017    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:35:35.091700 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:35.091320    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aecee731-ca16-4787-9eca-ff4a8b419e4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1ba901-mqfx9\" (UID: \"aecee731-ca16-4787-9eca-ff4a8b419e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ba901-mqfx9"
Oct 30 13:35:35.091700 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:35.091415    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aecee731-ca16-4787-9eca-ff4a8b419e4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1ba901-mqfx9\" (UID: \"aecee731-ca16-4787-9eca-ff4a8b419e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ba901-mqfx9"
Oct 30 13:35:35.091700 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:35.091642    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np27q\" (UniqueName: \"kubernetes.io/projected/aecee731-ca16-4787-9eca-ff4a8b419e4f-kube-api-access-np27q\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1ba901-mqfx9\" (UID: \"aecee731-ca16-4787-9eca-ff4a8b419e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ba901-mqfx9"
Oct 30 13:35:35.192989 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:35.192953    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np27q\" (UniqueName: \"kubernetes.io/projected/aecee731-ca16-4787-9eca-ff4a8b419e4f-kube-api-access-np27q\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1ba901-mqfx9\" (UID: \"aecee731-ca16-4787-9eca-ff4a8b419e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ba901-mqfx9"
Oct 30 13:35:35.193182 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:35.193139    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aecee731-ca16-4787-9eca-ff4a8b419e4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1ba901-mqfx9\" (UID: \"aecee731-ca16-4787-9eca-ff4a8b419e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ba901-mqfx9"
Oct 30 13:35:35.193303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:35.193265    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aecee731-ca16-4787-9eca-ff4a8b419e4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1ba901-mqfx9\" (UID: \"aecee731-ca16-4787-9eca-ff4a8b419e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ba901-mqfx9"
Oct 30 13:35:35.194012 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:35.193986    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aecee731-ca16-4787-9eca-ff4a8b419e4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1ba901-mqfx9\" (UID: \"aecee731-ca16-4787-9eca-ff4a8b419e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ba901-mqfx9"
Oct 30 13:35:35.195669 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:35.195642    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aecee731-ca16-4787-9eca-ff4a8b419e4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1ba901-mqfx9\" (UID: \"aecee731-ca16-4787-9eca-ff4a8b419e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ba901-mqfx9"
Oct 30 13:35:35.244300 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:35.243402    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np27q\" (UniqueName: \"kubernetes.io/projected/aecee731-ca16-4787-9eca-ff4a8b419e4f-kube-api-access-np27q\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1ba901-mqfx9\" (UID: \"aecee731-ca16-4787-9eca-ff4a8b419e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ba901-mqfx9"
Oct 30 13:35:35.380114 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:35.380076    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ba901-mqfx9"
Oct 30 13:35:35.742622 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:35:35.742570    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaecee731_ca16_4787_9eca_ff4a8b419e4f.slice/crio-7c6ccaf6bbf4788d21fa8fd9893a49f9da051d8cb0457d3701cb4314e51fceeb WatchSource:0}: Error finding container 7c6ccaf6bbf4788d21fa8fd9893a49f9da051d8cb0457d3701cb4314e51fceeb: Status 404 returned error can't find the container with id 7c6ccaf6bbf4788d21fa8fd9893a49f9da051d8cb0457d3701cb4314e51fceeb
Oct 30 13:35:35.743563 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:35.743527    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ba901-mqfx9"]
Oct 30 13:35:35.962907 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:35.962872    4544 patch_prober.go:29] interesting pod/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused" start-of-body=
Oct 30 13:35:35.963074 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:35.962933    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd" podUID="c713af1f-1695-4ad5-a564-25420ba533a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.45:15021/healthz/ready\": dial tcp 10.133.1.45:15021: connect: connection refused"
Oct 30 13:35:36.078861 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:36.078830    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ba901-mqfx9" event={"ID":"aecee731-ca16-4787-9eca-ff4a8b419e4f","Type":"ContainerStarted","Data":"7c6ccaf6bbf4788d21fa8fd9893a49f9da051d8cb0457d3701cb4314e51fceeb"}
Oct 30 13:35:36.144874 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:36.144835    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:36.264855 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:36.264815    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:36.485260 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:36.485141    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:36.977444 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:36.977410    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:35:36.978332 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:36.978305    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:35:36.979108 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:36.979088    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:35:37.110704 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:37.110669    4544 generic.go:356] "Generic (PLEG): container finished" podID="aecee731-ca16-4787-9eca-ff4a8b419e4f" containerID="4a5222e162eba5ed413d154e65f07ce07be363296910adfffd4fa45657cedcab" exitCode=0
Oct 30 13:35:37.113144 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:37.113109    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ba901-mqfx9" event={"ID":"aecee731-ca16-4787-9eca-ff4a8b419e4f","Type":"ContainerDied","Data":"4a5222e162eba5ed413d154e65f07ce07be363296910adfffd4fa45657cedcab"}
Oct 30 13:35:38.324949 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:38.324919    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ba901-mqfx9"
Oct 30 13:35:38.403588 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:38.403494    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:38.521932 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:38.520930    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aecee731-ca16-4787-9eca-ff4a8b419e4f-kube-api-access-np27q\" (UniqueName: \"kubernetes.io/projected/aecee731-ca16-4787-9eca-ff4a8b419e4f-kube-api-access-np27q\") pod \"aecee731-ca16-4787-9eca-ff4a8b419e4f\" (UID: \"aecee731-ca16-4787-9eca-ff4a8b419e4f\") "
Oct 30 13:35:38.521932 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:38.521149    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aecee731-ca16-4787-9eca-ff4a8b419e4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aecee731-ca16-4787-9eca-ff4a8b419e4f-maas-api-xks-ca\") pod \"aecee731-ca16-4787-9eca-ff4a8b419e4f\" (UID: \"aecee731-ca16-4787-9eca-ff4a8b419e4f\") "
Oct 30 13:35:38.521932 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:38.521288    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aecee731-ca16-4787-9eca-ff4a8b419e4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aecee731-ca16-4787-9eca-ff4a8b419e4f-maas-api-openshift-ca\") pod \"aecee731-ca16-4787-9eca-ff4a8b419e4f\" (UID: \"aecee731-ca16-4787-9eca-ff4a8b419e4f\") "
Oct 30 13:35:38.522714 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:38.522679    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aecee731-ca16-4787-9eca-ff4a8b419e4f-maas-api-openshift-ca" pod "aecee731-ca16-4787-9eca-ff4a8b419e4f" (UID: "aecee731-ca16-4787-9eca-ff4a8b419e4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:35:38.528145 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:38.528096    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aecee731-ca16-4787-9eca-ff4a8b419e4f-maas-api-xks-ca" pod "aecee731-ca16-4787-9eca-ff4a8b419e4f" (UID: "aecee731-ca16-4787-9eca-ff4a8b419e4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:35:38.528382 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:38.528107    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aecee731-ca16-4787-9eca-ff4a8b419e4f-kube-api-access-np27q" pod "aecee731-ca16-4787-9eca-ff4a8b419e4f" (UID: "aecee731-ca16-4787-9eca-ff4a8b419e4f"). InnerVolumeSpecName "kube-api-access-np27q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:38.623608 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:38.623573    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aecee731-ca16-4787-9eca-ff4a8b419e4f-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:35:38.623608 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:38.623602    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aecee731-ca16-4787-9eca-ff4a8b419e4f-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:35:38.623608 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:38.623613    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np27q\" (UniqueName: \"kubernetes.io/projected/aecee731-ca16-4787-9eca-ff4a8b419e4f-kube-api-access-np27q\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:35:39.130017 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:39.129988    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ba901-mqfx9"
Oct 30 13:35:39.130205 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:39.130018    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ba901-mqfx9" event={"ID":"aecee731-ca16-4787-9eca-ff4a8b419e4f","Type":"ContainerDied","Data":"7c6ccaf6bbf4788d21fa8fd9893a49f9da051d8cb0457d3701cb4314e51fceeb"}
Oct 30 13:35:39.130205 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:39.130076    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c6ccaf6bbf4788d21fa8fd9893a49f9da051d8cb0457d3701cb4314e51fceeb"
Oct 30 13:35:40.251765 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:40.251721    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:42.198316 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:42.198058    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:43.935331 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:43.935296    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:45.528497 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:45.528453    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk"]
Oct 30 13:35:45.570930 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:45.570882    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl"]
Oct 30 13:35:45.619913 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:45.619875    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:45.728157 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:45.728123    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:47.545858 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:47.545815    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:49.415458 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:49.415414    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:50.529275 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:50.529187    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk" podUID="a0e6a87f-6868-4a77-b75f-c3e1b0b00018" containerName="payload-processing" containerID="cri-o://f22e25bd952e208ebf7fbfb99328b067de736c4827ba2eef4b5d3cc2922f7f67" gracePeriod=25
Oct 30 13:35:50.572181 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:50.572132    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl" podUID="63baae32-04f3-47e2-9b88-b04768e3f762" containerName="payload-pre-processing" containerID="cri-o://e205f35122b58e091e3688de4c77f65ab10f4234f5dca2d7e9341679a3682e9a" gracePeriod=25
Oct 30 13:35:51.378490 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:51.378457    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:53.186400 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:53.186362    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:54.940027 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:54.939990    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:56.776197 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:56.776159    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:57.521305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:57.521264    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl" podUID="63baae32-04f3-47e2-9b88-b04768e3f762" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:35:57.521305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:57.521280    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk" podUID="a0e6a87f-6868-4a77-b75f-c3e1b0b00018" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:35:57.874865 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:57.874781    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:58.144947 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:58.144834    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:58.461693 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:35:58.461576    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:35:59.010116 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:35:59.010076    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:36:00.275838 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:00.275800    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:02.885289 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:02.885249    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:03.747469 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:03.747399    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:36:04.976911 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:04.976849    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:06.855053 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:06.855002    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:07.521487 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:07.521441    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl" podUID="63baae32-04f3-47e2-9b88-b04768e3f762" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:36:07.521487 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:07.521476    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk" podUID="a0e6a87f-6868-4a77-b75f-c3e1b0b00018" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:36:08.773977 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:08.773944    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:09.794783 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:09.794743    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"]
Oct 30 13:36:09.795297 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:09.794956    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd" podUID="f559800f-587f-4681-8e8c-5694a16cff06" containerName="istio-proxy" containerID="cri-o://b0d00d6817f6ed909780497e2a109113004a2963504083cb982d9268a9040d99" gracePeriod=30
Oct 30 13:36:10.542768 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:10.542730    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:10.898791 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:10.898761    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk"
Oct 30 13:36:10.935136 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:10.935097    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl"
Oct 30 13:36:10.967866 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:10.967829    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck"]
Oct 30 13:36:10.968791 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:10.968769    4544 cpu_manager.go:414] "containerMap: removing container" podUID="a0e6a87f-6868-4a77-b75f-c3e1b0b00018" containerName="payload-processing"
Oct 30 13:36:10.968791 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:10.968789    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0e6a87f-6868-4a77-b75f-c3e1b0b00018" containerName="payload-processing"
Oct 30 13:36:10.968990 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:10.968817    4544 cpu_manager.go:414] "containerMap: removing container" podUID="63baae32-04f3-47e2-9b88-b04768e3f762" containerName="payload-pre-processing"
Oct 30 13:36:10.968990 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:10.968823    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63baae32-04f3-47e2-9b88-b04768e3f762" containerName="payload-pre-processing"
Oct 30 13:36:10.968990 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:10.968867    4544 cpu_manager.go:414] "containerMap: removing container" podUID="aecee731-ca16-4787-9eca-ff4a8b419e4f" containerName="revoke-keys"
Oct 30 13:36:10.968990 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:10.968873    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aecee731-ca16-4787-9eca-ff4a8b419e4f" containerName="revoke-keys"
Oct 30 13:36:10.969283 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:10.969016    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="63baae32-04f3-47e2-9b88-b04768e3f762" containerName="payload-pre-processing"
Oct 30 13:36:10.969283 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:10.969032    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="aecee731-ca16-4787-9eca-ff4a8b419e4f" containerName="revoke-keys"
Oct 30 13:36:10.969283 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:10.969043    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0e6a87f-6868-4a77-b75f-c3e1b0b00018" containerName="payload-processing"
Oct 30 13:36:10.973747 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:10.973718    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck"
Oct 30 13:36:10.984490 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:10.984416    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0e6a87f-6868-4a77-b75f-c3e1b0b00018-kube-api-access-zbv9s\" (UniqueName: \"kubernetes.io/projected/a0e6a87f-6868-4a77-b75f-c3e1b0b00018-kube-api-access-zbv9s\") pod \"a0e6a87f-6868-4a77-b75f-c3e1b0b00018\" (UID: \"a0e6a87f-6868-4a77-b75f-c3e1b0b00018\") "
Oct 30 13:36:10.984694 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:10.984676    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0e6a87f-6868-4a77-b75f-c3e1b0b00018-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e6a87f-6868-4a77-b75f-c3e1b0b00018-plugins-config-volume\") pod \"a0e6a87f-6868-4a77-b75f-c3e1b0b00018\" (UID: \"a0e6a87f-6868-4a77-b75f-c3e1b0b00018\") "
Oct 30 13:36:10.984840 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:10.984825    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63baae32-04f3-47e2-9b88-b04768e3f762-kube-api-access-b72rv\" (UniqueName: \"kubernetes.io/projected/63baae32-04f3-47e2-9b88-b04768e3f762-kube-api-access-b72rv\") pod \"63baae32-04f3-47e2-9b88-b04768e3f762\" (UID: \"63baae32-04f3-47e2-9b88-b04768e3f762\") "
Oct 30 13:36:10.988441 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:10.984923    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63baae32-04f3-47e2-9b88-b04768e3f762-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63baae32-04f3-47e2-9b88-b04768e3f762-plugins-config-volume\") pod \"63baae32-04f3-47e2-9b88-b04768e3f762\" (UID: \"63baae32-04f3-47e2-9b88-b04768e3f762\") "
Oct 30 13:36:10.988441 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:10.985169    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0e6a87f-6868-4a77-b75f-c3e1b0b00018-plugins-config-volume" pod "a0e6a87f-6868-4a77-b75f-c3e1b0b00018" (UID: "a0e6a87f-6868-4a77-b75f-c3e1b0b00018"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:36:10.988441 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:10.985547    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63baae32-04f3-47e2-9b88-b04768e3f762-plugins-config-volume" pod "63baae32-04f3-47e2-9b88-b04768e3f762" (UID: "63baae32-04f3-47e2-9b88-b04768e3f762"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:36:11.001263 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:10.989357    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e6a87f-6868-4a77-b75f-c3e1b0b00018-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:36:11.001263 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:10.989391    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63baae32-04f3-47e2-9b88-b04768e3f762-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:36:11.001263 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:10.993433    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1c8130-dockercfg-dz2mk\""
Oct 30 13:36:11.001263 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:10.993767    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1c8130\""
Oct 30 13:36:11.020306 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:11.020196    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63baae32-04f3-47e2-9b88-b04768e3f762-kube-api-access-b72rv" pod "63baae32-04f3-47e2-9b88-b04768e3f762" (UID: "63baae32-04f3-47e2-9b88-b04768e3f762"). InnerVolumeSpecName "kube-api-access-b72rv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:36:11.020707 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:11.020665    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e6a87f-6868-4a77-b75f-c3e1b0b00018-kube-api-access-zbv9s" pod "a0e6a87f-6868-4a77-b75f-c3e1b0b00018" (UID: "a0e6a87f-6868-4a77-b75f-c3e1b0b00018"). InnerVolumeSpecName "kube-api-access-zbv9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:36:11.026275 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:11.024303    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck"]
Oct 30 13:36:11.090984 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:11.090939    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w59wl\" (UniqueName: \"kubernetes.io/projected/7ab2ca89-081f-411d-8ec4-20a8d399010f-kube-api-access-w59wl\") pod \"payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck\" (UID: \"7ab2ca89-081f-411d-8ec4-20a8d399010f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck"
Oct 30 13:36:11.091212 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:11.091136    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab2ca89-081f-411d-8ec4-20a8d399010f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck\" (UID: \"7ab2ca89-081f-411d-8ec4-20a8d399010f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck"
Oct 30 13:36:11.091516 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:11.091488    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b72rv\" (UniqueName: \"kubernetes.io/projected/63baae32-04f3-47e2-9b88-b04768e3f762-kube-api-access-b72rv\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:36:11.091639 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:11.091522    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbv9s\" (UniqueName: \"kubernetes.io/projected/a0e6a87f-6868-4a77-b75f-c3e1b0b00018-kube-api-access-zbv9s\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:36:11.193072 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:11.193034    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w59wl\" (UniqueName: \"kubernetes.io/projected/7ab2ca89-081f-411d-8ec4-20a8d399010f-kube-api-access-w59wl\") pod \"payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck\" (UID: \"7ab2ca89-081f-411d-8ec4-20a8d399010f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck"
Oct 30 13:36:11.193313 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:11.193162    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab2ca89-081f-411d-8ec4-20a8d399010f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck\" (UID: \"7ab2ca89-081f-411d-8ec4-20a8d399010f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck"
Oct 30 13:36:11.194165 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:11.194141    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab2ca89-081f-411d-8ec4-20a8d399010f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck\" (UID: \"7ab2ca89-081f-411d-8ec4-20a8d399010f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck"
Oct 30 13:36:11.217390 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:11.217357    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w59wl\" (UniqueName: \"kubernetes.io/projected/7ab2ca89-081f-411d-8ec4-20a8d399010f-kube-api-access-w59wl\") pod \"payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck\" (UID: \"7ab2ca89-081f-411d-8ec4-20a8d399010f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck"
Oct 30 13:36:11.302362 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:11.302320    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck"
Oct 30 13:36:11.617416 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:11.617380    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck"]
Oct 30 13:36:11.618576 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:36:11.618548    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab2ca89_081f_411d_8ec4_20a8d399010f.slice/crio-6f2f1e069afaca79e82def92eea22a4da59231aced4f55c2b276579a4f12af63 WatchSource:0}: Error finding container 6f2f1e069afaca79e82def92eea22a4da59231aced4f55c2b276579a4f12af63: Status 404 returned error can't find the container with id 6f2f1e069afaca79e82def92eea22a4da59231aced4f55c2b276579a4f12af63
Oct 30 13:36:11.634371 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:11.634341    4544 generic.go:356] "Generic (PLEG): container finished" podID="63baae32-04f3-47e2-9b88-b04768e3f762" containerID="e205f35122b58e091e3688de4c77f65ab10f4234f5dca2d7e9341679a3682e9a" exitCode=0
Oct 30 13:36:11.634504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:11.634386    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl"
Oct 30 13:36:11.634504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:11.634423    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl" event={"ID":"63baae32-04f3-47e2-9b88-b04768e3f762","Type":"ContainerDied","Data":"e205f35122b58e091e3688de4c77f65ab10f4234f5dca2d7e9341679a3682e9a"}
Oct 30 13:36:11.634504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:11.634478    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl" event={"ID":"63baae32-04f3-47e2-9b88-b04768e3f762","Type":"ContainerDied","Data":"221dbbdb33f4ee4fb333425bd096240f6afd41d047bbf0f4d32eaf18b4fe5126"}
Oct 30 13:36:11.634504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:11.634501    4544 scope.go:122] "RemoveContainer" containerID="e205f35122b58e091e3688de4c77f65ab10f4234f5dca2d7e9341679a3682e9a"
Oct 30 13:36:11.635936 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:11.635914    4544 generic.go:356] "Generic (PLEG): container finished" podID="a0e6a87f-6868-4a77-b75f-c3e1b0b00018" containerID="f22e25bd952e208ebf7fbfb99328b067de736c4827ba2eef4b5d3cc2922f7f67" exitCode=0
Oct 30 13:36:11.636081 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:11.635957    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk"
Oct 30 13:36:11.636081 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:11.636000    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk" event={"ID":"a0e6a87f-6868-4a77-b75f-c3e1b0b00018","Type":"ContainerDied","Data":"f22e25bd952e208ebf7fbfb99328b067de736c4827ba2eef4b5d3cc2922f7f67"}
Oct 30 13:36:11.636081 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:11.636044    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk" event={"ID":"a0e6a87f-6868-4a77-b75f-c3e1b0b00018","Type":"ContainerDied","Data":"59e40b20a719722c01434a166be10294f6fbd8f94f7f29fe8fcab17c4891f87e"}
Oct 30 13:36:11.646697 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:11.646672    4544 scope.go:122] "RemoveContainer" containerID="e205f35122b58e091e3688de4c77f65ab10f4234f5dca2d7e9341679a3682e9a"
Oct 30 13:36:11.647035 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:36:11.647012    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e205f35122b58e091e3688de4c77f65ab10f4234f5dca2d7e9341679a3682e9a\": container with ID starting with e205f35122b58e091e3688de4c77f65ab10f4234f5dca2d7e9341679a3682e9a not found: ID does not exist" containerID="e205f35122b58e091e3688de4c77f65ab10f4234f5dca2d7e9341679a3682e9a"
Oct 30 13:36:11.647089 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:11.647044    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e205f35122b58e091e3688de4c77f65ab10f4234f5dca2d7e9341679a3682e9a"} err="failed to get container status \"e205f35122b58e091e3688de4c77f65ab10f4234f5dca2d7e9341679a3682e9a\": rpc error: code = NotFound desc = could not find container \"e205f35122b58e091e3688de4c77f65ab10f4234f5dca2d7e9341679a3682e9a\": container with ID starting with e205f35122b58e091e3688de4c77f65ab10f4234f5dca2d7e9341679a3682e9a not found: ID does not exist"
Oct 30 13:36:11.647089 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:11.647062    4544 scope.go:122] "RemoveContainer" containerID="f22e25bd952e208ebf7fbfb99328b067de736c4827ba2eef4b5d3cc2922f7f67"
Oct 30 13:36:11.658191 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:11.658167    4544 scope.go:122] "RemoveContainer" containerID="f22e25bd952e208ebf7fbfb99328b067de736c4827ba2eef4b5d3cc2922f7f67"
Oct 30 13:36:11.658533 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:36:11.658513    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f22e25bd952e208ebf7fbfb99328b067de736c4827ba2eef4b5d3cc2922f7f67\": container with ID starting with f22e25bd952e208ebf7fbfb99328b067de736c4827ba2eef4b5d3cc2922f7f67 not found: ID does not exist" containerID="f22e25bd952e208ebf7fbfb99328b067de736c4827ba2eef4b5d3cc2922f7f67"
Oct 30 13:36:11.658584 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:11.658542    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f22e25bd952e208ebf7fbfb99328b067de736c4827ba2eef4b5d3cc2922f7f67"} err="failed to get container status \"f22e25bd952e208ebf7fbfb99328b067de736c4827ba2eef4b5d3cc2922f7f67\": rpc error: code = NotFound desc = could not find container \"f22e25bd952e208ebf7fbfb99328b067de736c4827ba2eef4b5d3cc2922f7f67\": container with ID starting with f22e25bd952e208ebf7fbfb99328b067de736c4827ba2eef4b5d3cc2922f7f67 not found: ID does not exist"
Oct 30 13:36:11.689126 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:11.689081    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk"]
Oct 30 13:36:11.697197 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:11.697160    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk"]
Oct 30 13:36:11.732713 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:11.732671    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl"]
Oct 30 13:36:11.737519 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:11.737491    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl"]
Oct 30 13:36:12.309468 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:12.309429    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:12.650563 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:12.650497    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck" event={"ID":"7ab2ca89-081f-411d-8ec4-20a8d399010f","Type":"ContainerStarted","Data":"6f2f1e069afaca79e82def92eea22a4da59231aced4f55c2b276579a4f12af63"}
Oct 30 13:36:13.091721 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:13.091691    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63baae32-04f3-47e2-9b88-b04768e3f762" path="/var/lib/kubelet/pods/63baae32-04f3-47e2-9b88-b04768e3f762/volumes"
Oct 30 13:36:13.092297 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:13.092277    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0e6a87f-6868-4a77-b75f-c3e1b0b00018" path="/var/lib/kubelet/pods/a0e6a87f-6868-4a77-b75f-c3e1b0b00018/volumes"
Oct 30 13:36:13.664538 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:13.664430    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck" event={"ID":"7ab2ca89-081f-411d-8ec4-20a8d399010f","Type":"ContainerStarted","Data":"49fb526bc42cf0f223ef2e63d670e46bb2815851dea7a1a1ec4368e6188f8149"}
Oct 30 13:36:13.665027 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:13.664536    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck"
Oct 30 13:36:13.723769 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:13.723702    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck" podStartSLOduration=69.962594783 podStartE2EDuration="1m11.723684727s" totalImagesPullingTime="1.761089944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:13.722394644 +0000 UTC m=+3068.769528591" watchObservedRunningTime="2026-10-30 13:36:13.723684727 +0000 UTC m=+3068.770818685"
Oct 30 13:36:14.103683 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:14.103637    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:15.102581 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.102552    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:36:15.177676 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.177646    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-istio-envoy\") pod \"f559800f-587f-4681-8e8c-5694a16cff06\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") "
Oct 30 13:36:15.177865 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.177735    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-workload-socket\") pod \"f559800f-587f-4681-8e8c-5694a16cff06\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") "
Oct 30 13:36:15.177865 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.177777    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f559800f-587f-4681-8e8c-5694a16cff06-istio-token\" (UniqueName: \"kubernetes.io/projected/f559800f-587f-4681-8e8c-5694a16cff06-istio-token\") pod \"f559800f-587f-4681-8e8c-5694a16cff06\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") "
Oct 30 13:36:15.178007 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.177907    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-credential-socket\") pod \"f559800f-587f-4681-8e8c-5694a16cff06\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") "
Oct 30 13:36:15.178007 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.177974    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"]
Oct 30 13:36:15.178110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.178001    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-istio-data\") pod \"f559800f-587f-4681-8e8c-5694a16cff06\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") "
Oct 30 13:36:15.178110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.178041    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-workload-socket" pod "f559800f-587f-4681-8e8c-5694a16cff06" (UID: "f559800f-587f-4681-8e8c-5694a16cff06"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:15.178262 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.178180    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f559800f-587f-4681-8e8c-5694a16cff06-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f559800f-587f-4681-8e8c-5694a16cff06-istiod-ca-cert\") pod \"f559800f-587f-4681-8e8c-5694a16cff06\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") "
Oct 30 13:36:15.178262 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.178244    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-credential-socket" pod "f559800f-587f-4681-8e8c-5694a16cff06" (UID: "f559800f-587f-4681-8e8c-5694a16cff06"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:15.178406 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.178331    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-istio-data" pod "f559800f-587f-4681-8e8c-5694a16cff06" (UID: "f559800f-587f-4681-8e8c-5694a16cff06"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:15.178546 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.178527    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f559800f-587f-4681-8e8c-5694a16cff06-istiod-ca-cert" pod "f559800f-587f-4681-8e8c-5694a16cff06" (UID: "f559800f-587f-4681-8e8c-5694a16cff06"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:36:15.178640 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.178625    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f559800f-587f-4681-8e8c-5694a16cff06-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f559800f-587f-4681-8e8c-5694a16cff06-istio-podinfo\") pod \"f559800f-587f-4681-8e8c-5694a16cff06\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") "
Oct 30 13:36:15.178707 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.178659    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f559800f-587f-4681-8e8c-5694a16cff06-kube-api-access-d2v4m\" (UniqueName: \"kubernetes.io/projected/f559800f-587f-4681-8e8c-5694a16cff06-kube-api-access-d2v4m\") pod \"f559800f-587f-4681-8e8c-5694a16cff06\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") "
Oct 30 13:36:15.179075 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.179051    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-workload-certs\") pod \"f559800f-587f-4681-8e8c-5694a16cff06\" (UID: \"f559800f-587f-4681-8e8c-5694a16cff06\") "
Oct 30 13:36:15.179513 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.179489    4544 cpu_manager.go:414] "containerMap: removing container" podUID="f559800f-587f-4681-8e8c-5694a16cff06" containerName="istio-proxy"
Oct 30 13:36:15.179611 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.179517    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f559800f-587f-4681-8e8c-5694a16cff06" containerName="istio-proxy"
Oct 30 13:36:15.179787 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.179768    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="f559800f-587f-4681-8e8c-5694a16cff06" containerName="istio-proxy"
Oct 30 13:36:15.180449 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.180425    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:36:15.180553 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.180456    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:36:15.180553 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.180471    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:36:15.180553 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.180484    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f559800f-587f-4681-8e8c-5694a16cff06-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:36:15.180986 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.180958    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-workload-certs" pod "f559800f-587f-4681-8e8c-5694a16cff06" (UID: "f559800f-587f-4681-8e8c-5694a16cff06"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:15.181461 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.181439    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f559800f-587f-4681-8e8c-5694a16cff06-istio-token" pod "f559800f-587f-4681-8e8c-5694a16cff06" (UID: "f559800f-587f-4681-8e8c-5694a16cff06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:36:15.182050 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.182028    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-istio-envoy" pod "f559800f-587f-4681-8e8c-5694a16cff06" (UID: "f559800f-587f-4681-8e8c-5694a16cff06"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:15.182335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.182313    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f559800f-587f-4681-8e8c-5694a16cff06-kube-api-access-d2v4m" pod "f559800f-587f-4681-8e8c-5694a16cff06" (UID: "f559800f-587f-4681-8e8c-5694a16cff06"). InnerVolumeSpecName "kube-api-access-d2v4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:36:15.182446 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.182383    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f559800f-587f-4681-8e8c-5694a16cff06-istio-podinfo" pod "f559800f-587f-4681-8e8c-5694a16cff06" (UID: "f559800f-587f-4681-8e8c-5694a16cff06"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:36:15.184815 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.184797    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:36:15.202133 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.202091    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3b47e7-openshift-default-dockercfg-fkvcj\""
Oct 30 13:36:15.211257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.211214    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"]
Oct 30 13:36:15.285132 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.282775    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8rsc\" (UniqueName: \"kubernetes.io/projected/e3559421-6424-4480-8d49-5ed4a11deb3d-kube-api-access-w8rsc\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:36:15.285132 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.282869    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-workload-certs\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:36:15.285132 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.283021    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-credential-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:36:15.285132 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.283061    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-istio-data\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:36:15.285132 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.283131    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3559421-6424-4480-8d49-5ed4a11deb3d-istio-podinfo\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:36:15.285132 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.283157    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3559421-6424-4480-8d49-5ed4a11deb3d-istiod-ca-cert\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:36:15.285132 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.283249    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-workload-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:36:15.285132 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.283391    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-istio-envoy\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:36:15.285132 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.283522    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3559421-6424-4480-8d49-5ed4a11deb3d-istio-token\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:36:15.285132 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.283593    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f559800f-587f-4681-8e8c-5694a16cff06-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:36:15.285132 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.283612    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2v4m\" (UniqueName: \"kubernetes.io/projected/f559800f-587f-4681-8e8c-5694a16cff06-kube-api-access-d2v4m\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:36:15.285132 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.283627    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:36:15.285132 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.283643    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f559800f-587f-4681-8e8c-5694a16cff06-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:36:15.285132 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.283656    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f559800f-587f-4681-8e8c-5694a16cff06-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:36:15.385009 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.384975    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3559421-6424-4480-8d49-5ed4a11deb3d-istio-token\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:36:15.385209 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.385025    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8rsc\" (UniqueName: \"kubernetes.io/projected/e3559421-6424-4480-8d49-5ed4a11deb3d-kube-api-access-w8rsc\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:36:15.385209 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.385071    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-workload-certs\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:36:15.385209 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.385170    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-credential-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:36:15.385209 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.385192    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-istio-data\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:36:15.385209 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.385259    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3559421-6424-4480-8d49-5ed4a11deb3d-istio-podinfo\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:36:15.385564 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.385291    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3559421-6424-4480-8d49-5ed4a11deb3d-istiod-ca-cert\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:36:15.385564 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.385386    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-workload-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:36:15.385564 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.385541    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-istio-envoy\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:36:15.385748 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.385720    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-credential-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:36:15.385818 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.385733    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-istio-data\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:36:15.385900 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.385882    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-workload-certs\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:36:15.385988 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.385965    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-workload-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:36:15.386208 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.386189    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3559421-6424-4480-8d49-5ed4a11deb3d-istiod-ca-cert\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:36:15.387733 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.387712    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-istio-envoy\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:36:15.387999 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.387981    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3559421-6424-4480-8d49-5ed4a11deb3d-istio-podinfo\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:36:15.411365 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.411335    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3559421-6424-4480-8d49-5ed4a11deb3d-istio-token\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:36:15.411541 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.411511    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8rsc\" (UniqueName: \"kubernetes.io/projected/e3559421-6424-4480-8d49-5ed4a11deb3d-kube-api-access-w8rsc\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:36:15.537423 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.537333    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:36:15.685722 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.685679    4544 generic.go:356] "Generic (PLEG): container finished" podID="f559800f-587f-4681-8e8c-5694a16cff06" containerID="b0d00d6817f6ed909780497e2a109113004a2963504083cb982d9268a9040d99" exitCode=0
Oct 30 13:36:15.685918 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.685747    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"
Oct 30 13:36:15.685918 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.685771    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd" event={"ID":"f559800f-587f-4681-8e8c-5694a16cff06","Type":"ContainerDied","Data":"b0d00d6817f6ed909780497e2a109113004a2963504083cb982d9268a9040d99"}
Oct 30 13:36:15.685918 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.685857    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd" event={"ID":"f559800f-587f-4681-8e8c-5694a16cff06","Type":"ContainerDied","Data":"e8e46c27855b810d8767ae03c15f701e7257110e12f5af93fcac4066a351ab16"}
Oct 30 13:36:15.685918 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.685891    4544 scope.go:122] "RemoveContainer" containerID="b0d00d6817f6ed909780497e2a109113004a2963504083cb982d9268a9040d99"
Oct 30 13:36:15.709635 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.701472    4544 scope.go:122] "RemoveContainer" containerID="b0d00d6817f6ed909780497e2a109113004a2963504083cb982d9268a9040d99"
Oct 30 13:36:15.709635 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:36:15.701913    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0d00d6817f6ed909780497e2a109113004a2963504083cb982d9268a9040d99\": container with ID starting with b0d00d6817f6ed909780497e2a109113004a2963504083cb982d9268a9040d99 not found: ID does not exist" containerID="b0d00d6817f6ed909780497e2a109113004a2963504083cb982d9268a9040d99"
Oct 30 13:36:15.709635 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.701955    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0d00d6817f6ed909780497e2a109113004a2963504083cb982d9268a9040d99"} err="failed to get container status \"b0d00d6817f6ed909780497e2a109113004a2963504083cb982d9268a9040d99\": rpc error: code = NotFound desc = could not find container \"b0d00d6817f6ed909780497e2a109113004a2963504083cb982d9268a9040d99\": container with ID starting with b0d00d6817f6ed909780497e2a109113004a2963504083cb982d9268a9040d99 not found: ID does not exist"
Oct 30 13:36:15.756201 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.756162    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"]
Oct 30 13:36:15.761470 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.761438    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd"]
Oct 30 13:36:15.875107 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.875067    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"]
Oct 30 13:36:15.875907 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:36:15.875882    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3559421_6424_4480_8d49_5ed4a11deb3d.slice/crio-363c9b2c18c77f8395a6afaaed19b210f702a2c8d43ed7712f8c7fcff37fceb8 WatchSource:0}: Error finding container 363c9b2c18c77f8395a6afaaed19b210f702a2c8d43ed7712f8c7fcff37fceb8: Status 404 returned error can't find the container with id 363c9b2c18c77f8395a6afaaed19b210f702a2c8d43ed7712f8c7fcff37fceb8
Oct 30 13:36:15.879420 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.879383    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:36:15.879526 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.879460    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:36:15.879526 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:15.879491    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:36:16.087884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:16.087846    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:16.698196 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:16.698154    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j" event={"ID":"e3559421-6424-4480-8d49-5ed4a11deb3d","Type":"ContainerStarted","Data":"a9f7334495eef7f6dc76371fa6eee8c0a454b18686522a401d2a378e2514fa0c"}
Oct 30 13:36:16.698757 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:16.698261    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j" event={"ID":"e3559421-6424-4480-8d49-5ed4a11deb3d","Type":"ContainerStarted","Data":"363c9b2c18c77f8395a6afaaed19b210f702a2c8d43ed7712f8c7fcff37fceb8"}
Oct 30 13:36:16.744735 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:16.744678    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j" podStartSLOduration=39.744659773 podStartE2EDuration="39.744659773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:16.744022021 +0000 UTC m=+3071.791155974" watchObservedRunningTime="2026-10-30 13:36:16.744659773 +0000 UTC m=+3071.791793728"
Oct 30 13:36:17.093280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:17.093242    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f559800f-587f-4681-8e8c-5694a16cff06" path="/var/lib/kubelet/pods/f559800f-587f-4681-8e8c-5694a16cff06/volumes"
Oct 30 13:36:17.537580 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:17.537474    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:36:17.540095 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:17.540054    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Oct 30 13:36:17.540278 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:17.540130    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j" podUID="e3559421-6424-4480-8d49-5ed4a11deb3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Oct 30 13:36:17.995915 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:17.995877    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:18.538196 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:18.538154    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Oct 30 13:36:18.538434 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:18.538272    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j" podUID="e3559421-6424-4480-8d49-5ed4a11deb3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Oct 30 13:36:19.537893 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:19.537844    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Oct 30 13:36:19.538496 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:19.537921    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j" podUID="e3559421-6424-4480-8d49-5ed4a11deb3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Oct 30 13:36:19.840100 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:19.840061    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:19.953864 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:19.950373    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:20.537796 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:20.537744    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Oct 30 13:36:20.537999 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:20.537832    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j" podUID="e3559421-6424-4480-8d49-5ed4a11deb3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Oct 30 13:36:21.538069 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:21.537889    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Oct 30 13:36:21.538069 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:21.537961    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j" podUID="e3559421-6424-4480-8d49-5ed4a11deb3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Oct 30 13:36:22.034673 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:22.034631    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns"]
Oct 30 13:36:22.035166 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:22.035134    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns" podUID="501f27bf-f8e6-4e51-97be-9383291b04ff" containerName="maas-api" containerID="cri-o://3ee5798a02c755b6b56c847b265d54a61920bc6ad0a842540b12d90780fc40e0" gracePeriod=30
Oct 30 13:36:22.095330 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:22.095290    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:22.567244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:22.566367    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Oct 30 13:36:22.567244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:22.566440    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j" podUID="e3559421-6424-4480-8d49-5ed4a11deb3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Oct 30 13:36:22.880349 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:22.879845    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:22.930782 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:22.930535    4544 generic.go:356] "Generic (PLEG): container finished" podID="501f27bf-f8e6-4e51-97be-9383291b04ff" containerID="3ee5798a02c755b6b56c847b265d54a61920bc6ad0a842540b12d90780fc40e0" exitCode=0
Oct 30 13:36:22.930782 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:22.930680    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns" event={"ID":"501f27bf-f8e6-4e51-97be-9383291b04ff","Type":"ContainerDied","Data":"3ee5798a02c755b6b56c847b265d54a61920bc6ad0a842540b12d90780fc40e0"}
Oct 30 13:36:22.948466 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:22.948364    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns"
Oct 30 13:36:23.012633 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.012591    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"]
Oct 30 13:36:23.014253 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.014203    4544 cpu_manager.go:414] "containerMap: removing container" podUID="501f27bf-f8e6-4e51-97be-9383291b04ff" containerName="maas-api"
Oct 30 13:36:23.014389 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.014256    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="501f27bf-f8e6-4e51-97be-9383291b04ff" containerName="maas-api"
Oct 30 13:36:23.014616 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.014596    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="501f27bf-f8e6-4e51-97be-9383291b04ff" containerName="maas-api"
Oct 30 13:36:23.027257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.027193    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:36:23.042680 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.042644    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"]
Oct 30 13:36:23.049707 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.049675    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/501f27bf-f8e6-4e51-97be-9383291b04ff-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/501f27bf-f8e6-4e51-97be-9383291b04ff-maas-api-tls\") pod \"501f27bf-f8e6-4e51-97be-9383291b04ff\" (UID: \"501f27bf-f8e6-4e51-97be-9383291b04ff\") "
Oct 30 13:36:23.049897 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.049877    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/501f27bf-f8e6-4e51-97be-9383291b04ff-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/501f27bf-f8e6-4e51-97be-9383291b04ff-openshift-service-ca\") pod \"501f27bf-f8e6-4e51-97be-9383291b04ff\" (UID: \"501f27bf-f8e6-4e51-97be-9383291b04ff\") "
Oct 30 13:36:23.050014 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.049994    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/501f27bf-f8e6-4e51-97be-9383291b04ff-kube-api-access-x6cbt\" (UniqueName: \"kubernetes.io/projected/501f27bf-f8e6-4e51-97be-9383291b04ff-kube-api-access-x6cbt\") pod \"501f27bf-f8e6-4e51-97be-9383291b04ff\" (UID: \"501f27bf-f8e6-4e51-97be-9383291b04ff\") "
Oct 30 13:36:23.050211 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.050194    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/501f27bf-f8e6-4e51-97be-9383291b04ff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/501f27bf-f8e6-4e51-97be-9383291b04ff-metrics-certs\") pod \"501f27bf-f8e6-4e51-97be-9383291b04ff\" (UID: \"501f27bf-f8e6-4e51-97be-9383291b04ff\") "
Oct 30 13:36:23.050342 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.050315    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/501f27bf-f8e6-4e51-97be-9383291b04ff-openshift-service-ca" pod "501f27bf-f8e6-4e51-97be-9383291b04ff" (UID: "501f27bf-f8e6-4e51-97be-9383291b04ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:36:23.051146 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.051122    4544 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/501f27bf-f8e6-4e51-97be-9383291b04ff-openshift-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:36:23.052942 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.052914    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/501f27bf-f8e6-4e51-97be-9383291b04ff-metrics-certs" pod "501f27bf-f8e6-4e51-97be-9383291b04ff" (UID: "501f27bf-f8e6-4e51-97be-9383291b04ff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:36:23.053032 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.052962    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/501f27bf-f8e6-4e51-97be-9383291b04ff-kube-api-access-x6cbt" pod "501f27bf-f8e6-4e51-97be-9383291b04ff" (UID: "501f27bf-f8e6-4e51-97be-9383291b04ff"). InnerVolumeSpecName "kube-api-access-x6cbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:36:23.053204 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.053186    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/501f27bf-f8e6-4e51-97be-9383291b04ff-maas-api-tls" pod "501f27bf-f8e6-4e51-97be-9383291b04ff" (UID: "501f27bf-f8e6-4e51-97be-9383291b04ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:36:23.153048 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.153011    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-istio-envoy\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:36:23.153197 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.153073    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-workload-certs\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:36:23.153443 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.153411    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6aed883-80e6-44a5-ac18-855437d32862-istiod-ca-cert\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:36:23.153552 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.153480    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-credential-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:36:23.153733 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.153711    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8mmp\" (UniqueName: \"kubernetes.io/projected/e6aed883-80e6-44a5-ac18-855437d32862-kube-api-access-j8mmp\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:36:23.153829 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.153746    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6aed883-80e6-44a5-ac18-855437d32862-istio-token\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:36:23.153958 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.153931    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-workload-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:36:23.154117 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.154065    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-istio-data\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:36:23.154620 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.154593    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6aed883-80e6-44a5-ac18-855437d32862-istio-podinfo\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:36:23.154829 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.154810    4544 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/501f27bf-f8e6-4e51-97be-9383291b04ff-metrics-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:36:23.154924 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.154833    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/501f27bf-f8e6-4e51-97be-9383291b04ff-maas-api-tls\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:36:23.154924 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.154849    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6cbt\" (UniqueName: \"kubernetes.io/projected/501f27bf-f8e6-4e51-97be-9383291b04ff-kube-api-access-x6cbt\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:36:23.225906 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.225864    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:23.256388 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.256338    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8mmp\" (UniqueName: \"kubernetes.io/projected/e6aed883-80e6-44a5-ac18-855437d32862-kube-api-access-j8mmp\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:36:23.256388 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.256393    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6aed883-80e6-44a5-ac18-855437d32862-istio-token\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:36:23.256689 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.256541    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-workload-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:36:23.256689 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.256664    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-istio-data\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:36:23.256814 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.256727    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6aed883-80e6-44a5-ac18-855437d32862-istio-podinfo\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:36:23.256814 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.256807    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-istio-envoy\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:36:23.256950 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.256842    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-workload-certs\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:36:23.257019 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.257001    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6aed883-80e6-44a5-ac18-855437d32862-istiod-ca-cert\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:36:23.257081 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.257055    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-credential-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:36:23.257081 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.257000    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-workload-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:36:23.257197 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.257110    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-istio-data\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:36:23.257334 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.257312    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-workload-certs\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:36:23.257428 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.257406    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-credential-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:36:23.257626 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.257600    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6aed883-80e6-44a5-ac18-855437d32862-istiod-ca-cert\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:36:23.259545 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.259515    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-istio-envoy\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:36:23.259778 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.259759    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6aed883-80e6-44a5-ac18-855437d32862-istio-podinfo\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:36:23.320157 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.319694    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8mmp\" (UniqueName: \"kubernetes.io/projected/e6aed883-80e6-44a5-ac18-855437d32862-kube-api-access-j8mmp\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:36:23.320157 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.319747    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6aed883-80e6-44a5-ac18-855437d32862-istio-token\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:36:23.353529 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.353490    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:36:23.427840 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.427797    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ba901-mqfx9"]
Oct 30 13:36:23.456474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.456434    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1ba901-mqfx9"]
Oct 30 13:36:23.543684 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.543651    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:36:23.544137 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.544107    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:36:23.544771 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.544750    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:36:23.602468 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.602386    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:23.717380 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:36:23.717332    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6aed883_80e6_44a5_ac18_855437d32862.slice/crio-6e90f0ff4567adf97083426bab796e21d1eb35113c8f40afcc1915c523a84521 WatchSource:0}: Error finding container 6e90f0ff4567adf97083426bab796e21d1eb35113c8f40afcc1915c523a84521: Status 404 returned error can't find the container with id 6e90f0ff4567adf97083426bab796e21d1eb35113c8f40afcc1915c523a84521
Oct 30 13:36:23.719026 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.719001    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"]
Oct 30 13:36:23.721283 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.721245    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:36:23.721380 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.721324    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:36:23.721380 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.721353    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:36:23.947381 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.947175    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs" event={"ID":"e6aed883-80e6-44a5-ac18-855437d32862","Type":"ContainerStarted","Data":"7dcecf6dfb4b226246e8f47b96862623b45f2a3b508bf93b475327ec6305dcf7"}
Oct 30 13:36:23.947381 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.947280    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs" event={"ID":"e6aed883-80e6-44a5-ac18-855437d32862","Type":"ContainerStarted","Data":"6e90f0ff4567adf97083426bab796e21d1eb35113c8f40afcc1915c523a84521"}
Oct 30 13:36:23.952089 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.952041    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns"
Oct 30 13:36:23.952833 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.952809    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns" event={"ID":"501f27bf-f8e6-4e51-97be-9383291b04ff","Type":"ContainerDied","Data":"43055af08234a41a1d0d776fcdfe413626f6c2cffe663345a7cace37afc7a44d"}
Oct 30 13:36:23.953010 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:23.952869    4544 scope.go:122] "RemoveContainer" containerID="3ee5798a02c755b6b56c847b265d54a61920bc6ad0a842540b12d90780fc40e0"
Oct 30 13:36:24.040358 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:24.008778    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs" podStartSLOduration=2.008755298 podStartE2EDuration="2.008755298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:23.999117294 +0000 UTC m=+3079.046251250" watchObservedRunningTime="2026-10-30 13:36:24.008755298 +0000 UTC m=+3079.055889252"
Oct 30 13:36:24.040358 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:24.021914    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:24.218246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:24.214053    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns"]
Oct 30 13:36:24.263000 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:24.262933    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ba901-7ccbbd66cb-26bns"]
Oct 30 13:36:24.356505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:24.356446    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused" start-of-body=
Oct 30 13:36:24.356740 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:24.356555    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs" podUID="e6aed883-80e6-44a5-ac18-855437d32862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused"
Oct 30 13:36:24.356740 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:24.356458    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:36:24.675343 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:24.675249    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck"
Oct 30 13:36:25.090475 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:25.090441    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="501f27bf-f8e6-4e51-97be-9383291b04ff" path="/var/lib/kubelet/pods/501f27bf-f8e6-4e51-97be-9383291b04ff/volumes"
Oct 30 13:36:25.090996 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:25.090980    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aecee731-ca16-4787-9eca-ff4a8b419e4f" path="/var/lib/kubelet/pods/aecee731-ca16-4787-9eca-ff4a8b419e4f/volumes"
Oct 30 13:36:25.354325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:25.354210    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused" start-of-body=
Oct 30 13:36:25.354325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:25.354289    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs" podUID="e6aed883-80e6-44a5-ac18-855437d32862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused"
Oct 30 13:36:26.354366 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:26.354326    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused" start-of-body=
Oct 30 13:36:26.354785 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:26.354382    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs" podUID="e6aed883-80e6-44a5-ac18-855437d32862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused"
Oct 30 13:36:26.538575 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:26.538531    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:27.354997 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:27.354955    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused" start-of-body=
Oct 30 13:36:27.355472 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:27.355031    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs" podUID="e6aed883-80e6-44a5-ac18-855437d32862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused"
Oct 30 13:36:28.353949 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:28.353913    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused" start-of-body=
Oct 30 13:36:28.354154 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:28.353973    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs" podUID="e6aed883-80e6-44a5-ac18-855437d32862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused"
Oct 30 13:36:29.011469 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:36:29.011434    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:36:29.354166 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:29.354124    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused" start-of-body=
Oct 30 13:36:29.354421 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:29.354210    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs" podUID="e6aed883-80e6-44a5-ac18-855437d32862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused"
Oct 30 13:36:29.391311 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:29.391266    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:30.354873 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:30.354831    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused" start-of-body=
Oct 30 13:36:30.355397 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:30.354897    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs" podUID="e6aed883-80e6-44a5-ac18-855437d32862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused"
Oct 30 13:36:30.444879 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:30.444837    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:30.564652 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:30.564614    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:30.809851 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:30.809744    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"]
Oct 30 13:36:30.810036 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:30.809987    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc" podUID="d85faa5b-0e4f-477e-a7ac-784f37404bb2" containerName="istio-proxy" containerID="cri-o://8cf8fadf6c3b8d0bdc27229bcdc77cb1d7e03a6e6972d7419ff800702a686a0c" gracePeriod=30
Oct 30 13:36:30.826733 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:30.826693    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:30.987139 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:30.987103    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:31.359334 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:31.359294    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:36:31.359853 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:31.359839    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:36:31.360741 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:31.360713    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:36:31.407575 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:31.407538    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-03b5b3-t44vs"]
Oct 30 13:36:31.414593 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:31.414561    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-03b5b3-t44vs"
Oct 30 13:36:31.430863 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:31.430827    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:36:31.430863 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:31.430845    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:36:31.455362 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:31.455328    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-03b5b3-t44vs"]
Oct 30 13:36:31.548136 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:31.548089    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b99rq\" (UniqueName: \"kubernetes.io/projected/ff5dc740-a6f8-4f2b-94cc-cade624c2bdf-kube-api-access-b99rq\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-03b5b3-t44vs\" (UID: \"ff5dc740-a6f8-4f2b-94cc-cade624c2bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-03b5b3-t44vs"
Oct 30 13:36:31.548397 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:31.548284    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff5dc740-a6f8-4f2b-94cc-cade624c2bdf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-03b5b3-t44vs\" (UID: \"ff5dc740-a6f8-4f2b-94cc-cade624c2bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-03b5b3-t44vs"
Oct 30 13:36:31.548479 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:31.548406    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff5dc740-a6f8-4f2b-94cc-cade624c2bdf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-03b5b3-t44vs\" (UID: \"ff5dc740-a6f8-4f2b-94cc-cade624c2bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-03b5b3-t44vs"
Oct 30 13:36:31.650286 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:31.650167    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff5dc740-a6f8-4f2b-94cc-cade624c2bdf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-03b5b3-t44vs\" (UID: \"ff5dc740-a6f8-4f2b-94cc-cade624c2bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-03b5b3-t44vs"
Oct 30 13:36:31.650495 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:31.650479    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b99rq\" (UniqueName: \"kubernetes.io/projected/ff5dc740-a6f8-4f2b-94cc-cade624c2bdf-kube-api-access-b99rq\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-03b5b3-t44vs\" (UID: \"ff5dc740-a6f8-4f2b-94cc-cade624c2bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-03b5b3-t44vs"
Oct 30 13:36:31.650572 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:31.650514    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff5dc740-a6f8-4f2b-94cc-cade624c2bdf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-03b5b3-t44vs\" (UID: \"ff5dc740-a6f8-4f2b-94cc-cade624c2bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-03b5b3-t44vs"
Oct 30 13:36:31.651190 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:31.651169    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff5dc740-a6f8-4f2b-94cc-cade624c2bdf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-03b5b3-t44vs\" (UID: \"ff5dc740-a6f8-4f2b-94cc-cade624c2bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-03b5b3-t44vs"
Oct 30 13:36:31.652709 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:31.652689    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff5dc740-a6f8-4f2b-94cc-cade624c2bdf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-03b5b3-t44vs\" (UID: \"ff5dc740-a6f8-4f2b-94cc-cade624c2bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-03b5b3-t44vs"
Oct 30 13:36:31.655113 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:31.655086    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:31.674237 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:31.674182    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b99rq\" (UniqueName: \"kubernetes.io/projected/ff5dc740-a6f8-4f2b-94cc-cade624c2bdf-kube-api-access-b99rq\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-03b5b3-t44vs\" (UID: \"ff5dc740-a6f8-4f2b-94cc-cade624c2bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-03b5b3-t44vs"
Oct 30 13:36:31.740252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:31.733549    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-03b5b3-t44vs"
Oct 30 13:36:32.071819 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:32.071773    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-03b5b3-t44vs"]
Oct 30 13:36:33.090783 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:33.090743    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-03b5b3-t44vs" event={"ID":"ff5dc740-a6f8-4f2b-94cc-cade624c2bdf","Type":"ContainerStarted","Data":"2757fbe46f11f9f98bce167c7440ac4dc930f8af11ea09b30798b3fd608a3470"}
Oct 30 13:36:33.091189 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:33.090802    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-03b5b3-t44vs" event={"ID":"ff5dc740-a6f8-4f2b-94cc-cade624c2bdf","Type":"ContainerStarted","Data":"b09ddebc169788ba8588a4a4b761d83adf98f9a1ea4c0a1a0325b40ff0aafe53"}
Oct 30 13:36:33.130734 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:33.130679    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-03b5b3-t44vs" podStartSLOduration=2.130662346 podStartE2EDuration="2.130662346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:33.126884353 +0000 UTC m=+3088.174018305" watchObservedRunningTime="2026-10-30 13:36:33.130662346 +0000 UTC m=+3088.177796301"
Oct 30 13:36:33.550864 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:33.550762    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:36:33.550864 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:33.550832    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc" podUID="d85faa5b-0e4f-477e-a7ac-784f37404bb2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:36:33.623716 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:33.623666    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:34.098093 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:34.098060    4544 generic.go:356] "Generic (PLEG): container finished" podID="ff5dc740-a6f8-4f2b-94cc-cade624c2bdf" containerID="2757fbe46f11f9f98bce167c7440ac4dc930f8af11ea09b30798b3fd608a3470" exitCode=0
Oct 30 13:36:34.098663 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:34.098140    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-03b5b3-t44vs" event={"ID":"ff5dc740-a6f8-4f2b-94cc-cade624c2bdf","Type":"ContainerDied","Data":"2757fbe46f11f9f98bce167c7440ac4dc930f8af11ea09b30798b3fd608a3470"}
Oct 30 13:36:35.294444 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:35.294415    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-03b5b3-t44vs"
Oct 30 13:36:35.459162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:35.459075    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff5dc740-a6f8-4f2b-94cc-cade624c2bdf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff5dc740-a6f8-4f2b-94cc-cade624c2bdf-maas-api-openshift-ca\") pod \"ff5dc740-a6f8-4f2b-94cc-cade624c2bdf\" (UID: \"ff5dc740-a6f8-4f2b-94cc-cade624c2bdf\") "
Oct 30 13:36:35.459162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:35.459138    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff5dc740-a6f8-4f2b-94cc-cade624c2bdf-kube-api-access-b99rq\" (UniqueName: \"kubernetes.io/projected/ff5dc740-a6f8-4f2b-94cc-cade624c2bdf-kube-api-access-b99rq\") pod \"ff5dc740-a6f8-4f2b-94cc-cade624c2bdf\" (UID: \"ff5dc740-a6f8-4f2b-94cc-cade624c2bdf\") "
Oct 30 13:36:35.459502 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:35.459483    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff5dc740-a6f8-4f2b-94cc-cade624c2bdf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff5dc740-a6f8-4f2b-94cc-cade624c2bdf-maas-api-xks-ca\") pod \"ff5dc740-a6f8-4f2b-94cc-cade624c2bdf\" (UID: \"ff5dc740-a6f8-4f2b-94cc-cade624c2bdf\") "
Oct 30 13:36:35.459575 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:35.459506    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff5dc740-a6f8-4f2b-94cc-cade624c2bdf-maas-api-openshift-ca" pod "ff5dc740-a6f8-4f2b-94cc-cade624c2bdf" (UID: "ff5dc740-a6f8-4f2b-94cc-cade624c2bdf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:36:35.460402 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:35.460374    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff5dc740-a6f8-4f2b-94cc-cade624c2bdf-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:36:35.461599 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:35.461569    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff5dc740-a6f8-4f2b-94cc-cade624c2bdf-kube-api-access-b99rq" pod "ff5dc740-a6f8-4f2b-94cc-cade624c2bdf" (UID: "ff5dc740-a6f8-4f2b-94cc-cade624c2bdf"). InnerVolumeSpecName "kube-api-access-b99rq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:36:35.461701 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:35.461616    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff5dc740-a6f8-4f2b-94cc-cade624c2bdf-maas-api-xks-ca" pod "ff5dc740-a6f8-4f2b-94cc-cade624c2bdf" (UID: "ff5dc740-a6f8-4f2b-94cc-cade624c2bdf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:36:35.562366 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:35.562317    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b99rq\" (UniqueName: \"kubernetes.io/projected/ff5dc740-a6f8-4f2b-94cc-cade624c2bdf-kube-api-access-b99rq\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:36:35.562366 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:35.562361    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff5dc740-a6f8-4f2b-94cc-cade624c2bdf-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.087714 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.087677    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:36:36.118031 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.117997    4544 generic.go:356] "Generic (PLEG): container finished" podID="d85faa5b-0e4f-477e-a7ac-784f37404bb2" containerID="8cf8fadf6c3b8d0bdc27229bcdc77cb1d7e03a6e6972d7419ff800702a686a0c" exitCode=0
Oct 30 13:36:36.118268 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.118049    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"
Oct 30 13:36:36.118268 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.118086    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc" event={"ID":"d85faa5b-0e4f-477e-a7ac-784f37404bb2","Type":"ContainerDied","Data":"8cf8fadf6c3b8d0bdc27229bcdc77cb1d7e03a6e6972d7419ff800702a686a0c"}
Oct 30 13:36:36.118268 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.118141    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc" event={"ID":"d85faa5b-0e4f-477e-a7ac-784f37404bb2","Type":"ContainerDied","Data":"83cb52e05bed32b14f2e41f250ea5366ca2c9c7e1151e5294143fcb867614914"}
Oct 30 13:36:36.118268 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.118200    4544 scope.go:122] "RemoveContainer" containerID="8cf8fadf6c3b8d0bdc27229bcdc77cb1d7e03a6e6972d7419ff800702a686a0c"
Oct 30 13:36:36.119970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.119713    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-03b5b3-t44vs"
Oct 30 13:36:36.119970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.119738    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-03b5b3-t44vs" event={"ID":"ff5dc740-a6f8-4f2b-94cc-cade624c2bdf","Type":"ContainerDied","Data":"b09ddebc169788ba8588a4a4b761d83adf98f9a1ea4c0a1a0325b40ff0aafe53"}
Oct 30 13:36:36.119970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.119777    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b09ddebc169788ba8588a4a4b761d83adf98f9a1ea4c0a1a0325b40ff0aafe53"
Oct 30 13:36:36.132795 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.132768    4544 scope.go:122] "RemoveContainer" containerID="8cf8fadf6c3b8d0bdc27229bcdc77cb1d7e03a6e6972d7419ff800702a686a0c"
Oct 30 13:36:36.133147 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:36:36.133123    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cf8fadf6c3b8d0bdc27229bcdc77cb1d7e03a6e6972d7419ff800702a686a0c\": container with ID starting with 8cf8fadf6c3b8d0bdc27229bcdc77cb1d7e03a6e6972d7419ff800702a686a0c not found: ID does not exist" containerID="8cf8fadf6c3b8d0bdc27229bcdc77cb1d7e03a6e6972d7419ff800702a686a0c"
Oct 30 13:36:36.133199 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.133161    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cf8fadf6c3b8d0bdc27229bcdc77cb1d7e03a6e6972d7419ff800702a686a0c"} err="failed to get container status \"8cf8fadf6c3b8d0bdc27229bcdc77cb1d7e03a6e6972d7419ff800702a686a0c\": rpc error: code = NotFound desc = could not find container \"8cf8fadf6c3b8d0bdc27229bcdc77cb1d7e03a6e6972d7419ff800702a686a0c\": container with ID starting with 8cf8fadf6c3b8d0bdc27229bcdc77cb1d7e03a6e6972d7419ff800702a686a0c not found: ID does not exist"
Oct 30 13:36:36.165049 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.165000    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"]
Oct 30 13:36:36.165993 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.165972    4544 cpu_manager.go:414] "containerMap: removing container" podUID="d85faa5b-0e4f-477e-a7ac-784f37404bb2" containerName="istio-proxy"
Oct 30 13:36:36.166060 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.165994    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d85faa5b-0e4f-477e-a7ac-784f37404bb2" containerName="istio-proxy"
Oct 30 13:36:36.166060 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.166054    4544 cpu_manager.go:414] "containerMap: removing container" podUID="ff5dc740-a6f8-4f2b-94cc-cade624c2bdf" containerName="revoke-keys"
Oct 30 13:36:36.166156 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.166060    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff5dc740-a6f8-4f2b-94cc-cade624c2bdf" containerName="revoke-keys"
Oct 30 13:36:36.166215 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.166204    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff5dc740-a6f8-4f2b-94cc-cade624c2bdf" containerName="revoke-keys"
Oct 30 13:36:36.166215 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.166236    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="d85faa5b-0e4f-477e-a7ac-784f37404bb2" containerName="istio-proxy"
Oct 30 13:36:36.181305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.173299    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-credential-socket\") pod \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") "
Oct 30 13:36:36.181305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.173356    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istio-podinfo\") pod \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") "
Oct 30 13:36:36.181305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.173385    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-workload-certs\") pod \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") "
Oct 30 13:36:36.181305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.173499    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istio-data\") pod \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") "
Oct 30 13:36:36.181305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.173570    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d85faa5b-0e4f-477e-a7ac-784f37404bb2-kube-api-access-smh6p\" (UniqueName: \"kubernetes.io/projected/d85faa5b-0e4f-477e-a7ac-784f37404bb2-kube-api-access-smh6p\") pod \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") "
Oct 30 13:36:36.181305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.173654    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istio-envoy\") pod \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") "
Oct 30 13:36:36.181305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.173722    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-workload-socket\") pod \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") "
Oct 30 13:36:36.181305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.174019    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istiod-ca-cert\") pod \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") "
Oct 30 13:36:36.181305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.174091    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istio-token\" (UniqueName: \"kubernetes.io/projected/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istio-token\") pod \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\" (UID: \"d85faa5b-0e4f-477e-a7ac-784f37404bb2\") "
Oct 30 13:36:36.181305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.175414    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istio-data" pod "d85faa5b-0e4f-477e-a7ac-784f37404bb2" (UID: "d85faa5b-0e4f-477e-a7ac-784f37404bb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:36.181305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.175629    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-credential-socket" pod "d85faa5b-0e4f-477e-a7ac-784f37404bb2" (UID: "d85faa5b-0e4f-477e-a7ac-784f37404bb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:36.181305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.176448    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istiod-ca-cert" pod "d85faa5b-0e4f-477e-a7ac-784f37404bb2" (UID: "d85faa5b-0e4f-477e-a7ac-784f37404bb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:36:36.181305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.176692    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-workload-certs" pod "d85faa5b-0e4f-477e-a7ac-784f37404bb2" (UID: "d85faa5b-0e4f-477e-a7ac-784f37404bb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:36.181305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.177035    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:36:36.181305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.178317    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-workload-socket" pod "d85faa5b-0e4f-477e-a7ac-784f37404bb2" (UID: "d85faa5b-0e4f-477e-a7ac-784f37404bb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:36.181305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.179617    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.181305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.179641    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.181305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.179655    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.181305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.179670    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.181305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.179682    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.181305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.181133    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d85faa5b-0e4f-477e-a7ac-784f37404bb2-kube-api-access-smh6p" pod "d85faa5b-0e4f-477e-a7ac-784f37404bb2" (UID: "d85faa5b-0e4f-477e-a7ac-784f37404bb2"). InnerVolumeSpecName "kube-api-access-smh6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:36:36.187373 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.184598    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:36.188146 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.188118    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istio-envoy" pod "d85faa5b-0e4f-477e-a7ac-784f37404bb2" (UID: "d85faa5b-0e4f-477e-a7ac-784f37404bb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:36.188400 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.188311    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istio-podinfo" pod "d85faa5b-0e4f-477e-a7ac-784f37404bb2" (UID: "d85faa5b-0e4f-477e-a7ac-784f37404bb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:36:36.188505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.188484    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istio-token" pod "d85faa5b-0e4f-477e-a7ac-784f37404bb2" (UID: "d85faa5b-0e4f-477e-a7ac-784f37404bb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:36:36.201199 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.201160    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"]
Oct 30 13:36:36.326032 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.325432    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istio-token\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:36:36.326032 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.325536    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istio-envoy\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:36:36.326032 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.325566    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istiod-ca-cert\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:36:36.326032 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.325623    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istio-data\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:36:36.326032 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.325683    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istio-podinfo\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:36:36.326032 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.325750    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-credential-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:36:36.326032 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.325826    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-workload-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:36:36.327190 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.327163    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-workload-certs\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:36:36.327449 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.327281    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js2cm\" (UniqueName: \"kubernetes.io/projected/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-kube-api-access-js2cm\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:36:36.327527 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.327512    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smh6p\" (UniqueName: \"kubernetes.io/projected/d85faa5b-0e4f-477e-a7ac-784f37404bb2-kube-api-access-smh6p\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.327642 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.327534    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.327755 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.327655    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.327755 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.327670    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85faa5b-0e4f-477e-a7ac-784f37404bb2-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.428955 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.428921    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istio-token\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:36:36.429118 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.428973    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istio-envoy\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:36:36.429118 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.429009    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istiod-ca-cert\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:36:36.429118 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.429051    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istio-data\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:36:36.429313 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.429117    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istio-podinfo\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:36:36.429313 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.429197    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-credential-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:36:36.429313 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.429299    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-workload-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:36:36.429869 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.429508    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-workload-certs\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:36:36.429869 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.429555    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istio-data\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:36:36.429869 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.429623    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-credential-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:36:36.429869 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.429635    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js2cm\" (UniqueName: \"kubernetes.io/projected/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-kube-api-access-js2cm\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:36:36.429869 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.429811    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istiod-ca-cert\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:36:36.429869 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.429831    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-workload-certs\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:36:36.429869 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.429873    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-workload-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:36:36.431766 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.431742    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istio-envoy\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:36:36.432368 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.432349    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istio-podinfo\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:36:36.452185 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.452153    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istio-token\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:36:36.453883 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.453850    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js2cm\" (UniqueName: \"kubernetes.io/projected/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-kube-api-access-js2cm\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:36:36.469346 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.469309    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"]
Oct 30 13:36:36.474780 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.474749    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc"]
Oct 30 13:36:36.515300 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.515206    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:36:36.864196 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.864165    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"]
Oct 30 13:36:36.864371 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:36:36.864349    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff60abd5_8f12_4e67_99f4_e6baeaee47ce.slice/crio-70d3f83ae1e5d8899ec11db36cb4d5e5db4847eeace788d2f2940d2e01523582 WatchSource:0}: Error finding container 70d3f83ae1e5d8899ec11db36cb4d5e5db4847eeace788d2f2940d2e01523582: Status 404 returned error can't find the container with id 70d3f83ae1e5d8899ec11db36cb4d5e5db4847eeace788d2f2940d2e01523582
Oct 30 13:36:36.867599 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.867559    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:36:36.867720 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.867648    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:36:36.867720 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:36.867697    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:36:37.091058 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:37.091009    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d85faa5b-0e4f-477e-a7ac-784f37404bb2" path="/var/lib/kubelet/pods/d85faa5b-0e4f-477e-a7ac-784f37404bb2/volumes"
Oct 30 13:36:37.132290 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:37.132183    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn" event={"ID":"ff60abd5-8f12-4e67-99f4-e6baeaee47ce","Type":"ContainerStarted","Data":"b859c49495aa286dbbb49625bd0e28cdb6f01b56f18f257120889e82a3b6279d"}
Oct 30 13:36:37.132290 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:37.132259    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn" event={"ID":"ff60abd5-8f12-4e67-99f4-e6baeaee47ce","Type":"ContainerStarted","Data":"70d3f83ae1e5d8899ec11db36cb4d5e5db4847eeace788d2f2940d2e01523582"}
Oct 30 13:36:37.185396 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:37.185342    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn" podStartSLOduration=15.185325607 podStartE2EDuration="15.185325607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:37.180262054 +0000 UTC m=+3092.227396006" watchObservedRunningTime="2026-10-30 13:36:37.185325607 +0000 UTC m=+3092.232459560"
Oct 30 13:36:37.515715 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:37.515610    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:36:37.517329 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:37.517298    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body=
Oct 30 13:36:37.517476 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:37.517350    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn" podUID="ff60abd5-8f12-4e67-99f4-e6baeaee47ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused"
Oct 30 13:36:38.134246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:38.128458    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:38.518858 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:38.518759    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body=
Oct 30 13:36:38.519509 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:38.519474    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn" podUID="ff60abd5-8f12-4e67-99f4-e6baeaee47ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused"
Oct 30 13:36:39.522246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:39.518600    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body=
Oct 30 13:36:39.522246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:39.518675    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn" podUID="ff60abd5-8f12-4e67-99f4-e6baeaee47ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused"
Oct 30 13:36:40.075255 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:40.074798    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:40.532834 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:40.532727    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body=
Oct 30 13:36:40.533510 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:40.533472    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn" podUID="ff60abd5-8f12-4e67-99f4-e6baeaee47ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused"
Oct 30 13:36:41.156867 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:41.156834    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:41.391758 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:41.391674    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c1b28-gptll"]
Oct 30 13:36:41.398340 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:41.398313    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c1b28-gptll"
Oct 30 13:36:41.440662 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:41.440616    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:36:41.441094 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:41.441067    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:36:41.449278 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:41.449239    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c1b28-gptll"]
Oct 30 13:36:41.457091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:41.457055    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwqqn\" (UniqueName: \"kubernetes.io/projected/9cc129d7-3824-476b-b201-48f5b6aa2106-kube-api-access-pwqqn\") pod \"maas-api-revoke-keys-e2e-ipp-a-3c1b28-gptll\" (UID: \"9cc129d7-3824-476b-b201-48f5b6aa2106\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c1b28-gptll"
Oct 30 13:36:41.457404 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:41.457383    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cc129d7-3824-476b-b201-48f5b6aa2106-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3c1b28-gptll\" (UID: \"9cc129d7-3824-476b-b201-48f5b6aa2106\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c1b28-gptll"
Oct 30 13:36:41.457621 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:41.457597    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cc129d7-3824-476b-b201-48f5b6aa2106-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3c1b28-gptll\" (UID: \"9cc129d7-3824-476b-b201-48f5b6aa2106\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c1b28-gptll"
Oct 30 13:36:41.540409 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:41.540367    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body=
Oct 30 13:36:41.541446 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:41.541406    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn" podUID="ff60abd5-8f12-4e67-99f4-e6baeaee47ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused"
Oct 30 13:36:41.559915 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:41.559864    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cc129d7-3824-476b-b201-48f5b6aa2106-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3c1b28-gptll\" (UID: \"9cc129d7-3824-476b-b201-48f5b6aa2106\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c1b28-gptll"
Oct 30 13:36:41.560336 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:41.560307    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cc129d7-3824-476b-b201-48f5b6aa2106-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3c1b28-gptll\" (UID: \"9cc129d7-3824-476b-b201-48f5b6aa2106\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c1b28-gptll"
Oct 30 13:36:41.561350 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:41.561327    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwqqn\" (UniqueName: \"kubernetes.io/projected/9cc129d7-3824-476b-b201-48f5b6aa2106-kube-api-access-pwqqn\") pod \"maas-api-revoke-keys-e2e-ipp-a-3c1b28-gptll\" (UID: \"9cc129d7-3824-476b-b201-48f5b6aa2106\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c1b28-gptll"
Oct 30 13:36:41.561459 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:41.560655    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cc129d7-3824-476b-b201-48f5b6aa2106-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3c1b28-gptll\" (UID: \"9cc129d7-3824-476b-b201-48f5b6aa2106\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c1b28-gptll"
Oct 30 13:36:41.563349 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:41.563318    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cc129d7-3824-476b-b201-48f5b6aa2106-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3c1b28-gptll\" (UID: \"9cc129d7-3824-476b-b201-48f5b6aa2106\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c1b28-gptll"
Oct 30 13:36:41.584572 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:41.584551    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwqqn\" (UniqueName: \"kubernetes.io/projected/9cc129d7-3824-476b-b201-48f5b6aa2106-kube-api-access-pwqqn\") pod \"maas-api-revoke-keys-e2e-ipp-a-3c1b28-gptll\" (UID: \"9cc129d7-3824-476b-b201-48f5b6aa2106\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c1b28-gptll"
Oct 30 13:36:41.734972 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:41.734623    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c1b28-gptll"
Oct 30 13:36:42.065040 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:42.064930    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c1b28-gptll"]
Oct 30 13:36:42.072979 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:36:42.072941    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cc129d7_3824_476b_b201_48f5b6aa2106.slice/crio-2de6321efbf63f4d199ad359752503f95dadf04fbf40486156c9107dc5106959 WatchSource:0}: Error finding container 2de6321efbf63f4d199ad359752503f95dadf04fbf40486156c9107dc5106959: Status 404 returned error can't find the container with id 2de6321efbf63f4d199ad359752503f95dadf04fbf40486156c9107dc5106959
Oct 30 13:36:42.081972 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:42.081926    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:42.203939 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:42.203885    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:42.355177 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:42.355144    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c1b28-gptll" event={"ID":"9cc129d7-3824-476b-b201-48f5b6aa2106","Type":"ContainerStarted","Data":"2de6321efbf63f4d199ad359752503f95dadf04fbf40486156c9107dc5106959"}
Oct 30 13:36:42.538435 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:42.538385    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body=
Oct 30 13:36:42.538653 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:42.538467    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn" podUID="ff60abd5-8f12-4e67-99f4-e6baeaee47ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused"
Oct 30 13:36:42.740252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:42.726464    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn"]
Oct 30 13:36:42.776400 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:42.776366    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn"]
Oct 30 13:36:42.776756 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:42.776693    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn"
Oct 30 13:36:42.811614 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:42.811577    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd7mh\" (UniqueName: \"kubernetes.io/projected/33b21dec-41c6-42c7-a0ea-729d5bd768a9-kube-api-access-wd7mh\") pod \"maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn\" (UID: \"33b21dec-41c6-42c7-a0ea-729d5bd768a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn"
Oct 30 13:36:42.812094 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:42.812069    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b21dec-41c6-42c7-a0ea-729d5bd768a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn\" (UID: \"33b21dec-41c6-42c7-a0ea-729d5bd768a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn"
Oct 30 13:36:42.812298 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:42.812280    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b21dec-41c6-42c7-a0ea-729d5bd768a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn\" (UID: \"33b21dec-41c6-42c7-a0ea-729d5bd768a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn"
Oct 30 13:36:42.990301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:42.927919    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd7mh\" (UniqueName: \"kubernetes.io/projected/33b21dec-41c6-42c7-a0ea-729d5bd768a9-kube-api-access-wd7mh\") pod \"maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn\" (UID: \"33b21dec-41c6-42c7-a0ea-729d5bd768a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn"
Oct 30 13:36:42.990301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:42.928175    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b21dec-41c6-42c7-a0ea-729d5bd768a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn\" (UID: \"33b21dec-41c6-42c7-a0ea-729d5bd768a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn"
Oct 30 13:36:42.990301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:42.928258    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b21dec-41c6-42c7-a0ea-729d5bd768a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn\" (UID: \"33b21dec-41c6-42c7-a0ea-729d5bd768a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn"
Oct 30 13:36:42.990301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:42.929328    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b21dec-41c6-42c7-a0ea-729d5bd768a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn\" (UID: \"33b21dec-41c6-42c7-a0ea-729d5bd768a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn"
Oct 30 13:36:43.076178 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:43.076144    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b21dec-41c6-42c7-a0ea-729d5bd768a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn\" (UID: \"33b21dec-41c6-42c7-a0ea-729d5bd768a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn"
Oct 30 13:36:43.083320 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:43.083062    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd7mh\" (UniqueName: \"kubernetes.io/projected/33b21dec-41c6-42c7-a0ea-729d5bd768a9-kube-api-access-wd7mh\") pod \"maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn\" (UID: \"33b21dec-41c6-42c7-a0ea-729d5bd768a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn"
Oct 30 13:36:43.357765 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:43.357727    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn"
Oct 30 13:36:43.366546 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:43.366508    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c1b28-gptll" event={"ID":"9cc129d7-3824-476b-b201-48f5b6aa2106","Type":"ContainerStarted","Data":"26de7e75310929543da0a8bb10e9357e2fb9a70ead5c13c47387ca3ed4aefbab"}
Oct 30 13:36:43.411048 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:43.409674    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c1b28-gptll" podStartSLOduration=2.409657567 podStartE2EDuration="2.409657567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:43.408306486 +0000 UTC m=+3098.455440442" watchObservedRunningTime="2026-10-30 13:36:43.409657567 +0000 UTC m=+3098.456791522"
Oct 30 13:36:43.617834 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:43.617712    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:36:43.618795 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:43.618745    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:36:43.620543 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:43.620510    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:36:43.683529 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:43.683493    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn"]
Oct 30 13:36:43.688467 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:36:43.688431    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b21dec_41c6_42c7_a0ea_729d5bd768a9.slice/crio-c7045354ddb7283770f58001d34a9092b670863195a735c93e631c62a4946d71 WatchSource:0}: Error finding container c7045354ddb7283770f58001d34a9092b670863195a735c93e631c62a4946d71: Status 404 returned error can't find the container with id c7045354ddb7283770f58001d34a9092b670863195a735c93e631c62a4946d71
Oct 30 13:36:44.377744 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:44.377702    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn" event={"ID":"33b21dec-41c6-42c7-a0ea-729d5bd768a9","Type":"ContainerStarted","Data":"0c77b90a9471e7b28ae9b381a6cf6a25addf9267fc43f44e711f06acf5cd293b"}
Oct 30 13:36:44.378199 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:44.377773    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn" event={"ID":"33b21dec-41c6-42c7-a0ea-729d5bd768a9","Type":"ContainerStarted","Data":"c7045354ddb7283770f58001d34a9092b670863195a735c93e631c62a4946d71"}
Oct 30 13:36:44.379615 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:44.379586    4544 generic.go:356] "Generic (PLEG): container finished" podID="9cc129d7-3824-476b-b201-48f5b6aa2106" containerID="26de7e75310929543da0a8bb10e9357e2fb9a70ead5c13c47387ca3ed4aefbab" exitCode=0
Oct 30 13:36:44.379769 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:44.379667    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c1b28-gptll" event={"ID":"9cc129d7-3824-476b-b201-48f5b6aa2106","Type":"ContainerDied","Data":"26de7e75310929543da0a8bb10e9357e2fb9a70ead5c13c47387ca3ed4aefbab"}
Oct 30 13:36:44.424510 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:44.424441    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn" podStartSLOduration=2.424422267 podStartE2EDuration="2.424422267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:44.422431346 +0000 UTC m=+3099.469565299" watchObservedRunningTime="2026-10-30 13:36:44.424422267 +0000 UTC m=+3099.471556261"
Oct 30 13:36:44.662301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:44.660903    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:45.390072 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:45.390037    4544 generic.go:356] "Generic (PLEG): container finished" podID="33b21dec-41c6-42c7-a0ea-729d5bd768a9" containerID="0c77b90a9471e7b28ae9b381a6cf6a25addf9267fc43f44e711f06acf5cd293b" exitCode=7
Oct 30 13:36:45.390536 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:45.390123    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn" event={"ID":"33b21dec-41c6-42c7-a0ea-729d5bd768a9","Type":"ContainerDied","Data":"0c77b90a9471e7b28ae9b381a6cf6a25addf9267fc43f44e711f06acf5cd293b"}
Oct 30 13:36:45.390604 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:45.390571    4544 scope.go:122] "RemoveContainer" containerID="0c77b90a9471e7b28ae9b381a6cf6a25addf9267fc43f44e711f06acf5cd293b"
Oct 30 13:36:45.672645 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:45.672614    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c1b28-gptll"
Oct 30 13:36:45.720370 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:45.720330    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cc129d7-3824-476b-b201-48f5b6aa2106-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cc129d7-3824-476b-b201-48f5b6aa2106-maas-api-openshift-ca\") pod \"9cc129d7-3824-476b-b201-48f5b6aa2106\" (UID: \"9cc129d7-3824-476b-b201-48f5b6aa2106\") "
Oct 30 13:36:45.720695 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:45.720557    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cc129d7-3824-476b-b201-48f5b6aa2106-kube-api-access-pwqqn\" (UniqueName: \"kubernetes.io/projected/9cc129d7-3824-476b-b201-48f5b6aa2106-kube-api-access-pwqqn\") pod \"9cc129d7-3824-476b-b201-48f5b6aa2106\" (UID: \"9cc129d7-3824-476b-b201-48f5b6aa2106\") "
Oct 30 13:36:45.720695 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:45.720604    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cc129d7-3824-476b-b201-48f5b6aa2106-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cc129d7-3824-476b-b201-48f5b6aa2106-maas-api-xks-ca\") pod \"9cc129d7-3824-476b-b201-48f5b6aa2106\" (UID: \"9cc129d7-3824-476b-b201-48f5b6aa2106\") "
Oct 30 13:36:45.720845 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:45.720760    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cc129d7-3824-476b-b201-48f5b6aa2106-maas-api-openshift-ca" pod "9cc129d7-3824-476b-b201-48f5b6aa2106" (UID: "9cc129d7-3824-476b-b201-48f5b6aa2106"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:36:45.721840 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:45.721783    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cc129d7-3824-476b-b201-48f5b6aa2106-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:36:45.726547 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:45.726373    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cc129d7-3824-476b-b201-48f5b6aa2106-maas-api-xks-ca" pod "9cc129d7-3824-476b-b201-48f5b6aa2106" (UID: "9cc129d7-3824-476b-b201-48f5b6aa2106"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:36:45.727087 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:45.727049    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cc129d7-3824-476b-b201-48f5b6aa2106-kube-api-access-pwqqn" pod "9cc129d7-3824-476b-b201-48f5b6aa2106" (UID: "9cc129d7-3824-476b-b201-48f5b6aa2106"). InnerVolumeSpecName "kube-api-access-pwqqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:36:45.823103 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:45.823073    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwqqn\" (UniqueName: \"kubernetes.io/projected/9cc129d7-3824-476b-b201-48f5b6aa2106-kube-api-access-pwqqn\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:36:45.823103 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:45.823103    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cc129d7-3824-476b-b201-48f5b6aa2106-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:36:46.401871 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:46.401833    4544 generic.go:356] "Generic (PLEG): container finished" podID="33b21dec-41c6-42c7-a0ea-729d5bd768a9" containerID="08d559782601d48bd6a4df9982613081aff34bf96da07a3cb884eecb6afd3335" exitCode=7
Oct 30 13:36:46.402413 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:46.401915    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn" event={"ID":"33b21dec-41c6-42c7-a0ea-729d5bd768a9","Type":"ContainerDied","Data":"08d559782601d48bd6a4df9982613081aff34bf96da07a3cb884eecb6afd3335"}
Oct 30 13:36:46.402413 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:46.401979    4544 scope.go:122] "RemoveContainer" containerID="0c77b90a9471e7b28ae9b381a6cf6a25addf9267fc43f44e711f06acf5cd293b"
Oct 30 13:36:46.402413 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:46.402401    4544 scope.go:122] "RemoveContainer" containerID="08d559782601d48bd6a4df9982613081aff34bf96da07a3cb884eecb6afd3335"
Oct 30 13:36:46.403275 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:36:46.402907    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn_odh-ai-gateway-infra(33b21dec-41c6-42c7-a0ea-729d5bd768a9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn" podUID="33b21dec-41c6-42c7-a0ea-729d5bd768a9"
Oct 30 13:36:46.403569 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:46.403548    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c1b28-gptll" event={"ID":"9cc129d7-3824-476b-b201-48f5b6aa2106","Type":"ContainerDied","Data":"2de6321efbf63f4d199ad359752503f95dadf04fbf40486156c9107dc5106959"}
Oct 30 13:36:46.403637 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:46.403590    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2de6321efbf63f4d199ad359752503f95dadf04fbf40486156c9107dc5106959"
Oct 30 13:36:46.403637 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:46.403607    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c1b28-gptll"
Oct 30 13:36:46.423994 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:46.423963    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:47.417620 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:47.417533    4544 scope.go:122] "RemoveContainer" containerID="08d559782601d48bd6a4df9982613081aff34bf96da07a3cb884eecb6afd3335"
Oct 30 13:36:47.418073 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:36:47.418021    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn_odh-ai-gateway-infra(33b21dec-41c6-42c7-a0ea-729d5bd768a9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn" podUID="33b21dec-41c6-42c7-a0ea-729d5bd768a9"
Oct 30 13:36:48.469255 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:48.469198    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:50.212134 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:50.212101    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:52.190357 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:52.190311    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:54.074886 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:54.074851    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:55.809972 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:55.809937    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:55.988527 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:55.988487    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6"]
Oct 30 13:36:55.989517 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:55.989493    4544 cpu_manager.go:414] "containerMap: removing container" podUID="9cc129d7-3824-476b-b201-48f5b6aa2106" containerName="revoke-keys"
Oct 30 13:36:55.989517 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:55.989514    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cc129d7-3824-476b-b201-48f5b6aa2106" containerName="revoke-keys"
Oct 30 13:36:55.989760 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:55.989696    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cc129d7-3824-476b-b201-48f5b6aa2106" containerName="revoke-keys"
Oct 30 13:36:55.995296 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:55.995267    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6"
Oct 30 13:36:56.016283 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:56.016247    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6"]
Oct 30 13:36:56.018432 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:56.018390    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4vt4\" (UniqueName: \"kubernetes.io/projected/7224cc91-c4bd-4c4e-99c2-9f721c9f3a71-kube-api-access-p4vt4\") pod \"maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6\" (UID: \"7224cc91-c4bd-4c4e-99c2-9f721c9f3a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6"
Oct 30 13:36:56.018910 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:56.018870    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7224cc91-c4bd-4c4e-99c2-9f721c9f3a71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6\" (UID: \"7224cc91-c4bd-4c4e-99c2-9f721c9f3a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6"
Oct 30 13:36:56.019081 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:56.018988    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7224cc91-c4bd-4c4e-99c2-9f721c9f3a71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6\" (UID: \"7224cc91-c4bd-4c4e-99c2-9f721c9f3a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6"
Oct 30 13:36:56.121611 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:56.121568    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4vt4\" (UniqueName: \"kubernetes.io/projected/7224cc91-c4bd-4c4e-99c2-9f721c9f3a71-kube-api-access-p4vt4\") pod \"maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6\" (UID: \"7224cc91-c4bd-4c4e-99c2-9f721c9f3a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6"
Oct 30 13:36:56.121878 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:56.121758    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7224cc91-c4bd-4c4e-99c2-9f721c9f3a71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6\" (UID: \"7224cc91-c4bd-4c4e-99c2-9f721c9f3a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6"
Oct 30 13:36:56.121878 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:56.121821    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7224cc91-c4bd-4c4e-99c2-9f721c9f3a71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6\" (UID: \"7224cc91-c4bd-4c4e-99c2-9f721c9f3a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6"
Oct 30 13:36:56.122779 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:56.122750    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7224cc91-c4bd-4c4e-99c2-9f721c9f3a71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6\" (UID: \"7224cc91-c4bd-4c4e-99c2-9f721c9f3a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6"
Oct 30 13:36:56.124472 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:56.124450    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7224cc91-c4bd-4c4e-99c2-9f721c9f3a71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6\" (UID: \"7224cc91-c4bd-4c4e-99c2-9f721c9f3a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6"
Oct 30 13:36:56.145516 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:56.145486    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4vt4\" (UniqueName: \"kubernetes.io/projected/7224cc91-c4bd-4c4e-99c2-9f721c9f3a71-kube-api-access-p4vt4\") pod \"maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6\" (UID: \"7224cc91-c4bd-4c4e-99c2-9f721c9f3a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6"
Oct 30 13:36:56.313073 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:56.313029    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6"
Oct 30 13:36:56.634925 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:56.634885    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6"]
Oct 30 13:36:56.635886 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:36:56.635851    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7224cc91_c4bd_4c4e_99c2_9f721c9f3a71.slice/crio-3a7e92d033359f62355cda6ea38233c930ae0382c808a3bb1abd79944f8de667 WatchSource:0}: Error finding container 3a7e92d033359f62355cda6ea38233c930ae0382c808a3bb1abd79944f8de667: Status 404 returned error can't find the container with id 3a7e92d033359f62355cda6ea38233c930ae0382c808a3bb1abd79944f8de667
Oct 30 13:36:57.519522 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:57.519479    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6" event={"ID":"7224cc91-c4bd-4c4e-99c2-9f721c9f3a71","Type":"ContainerStarted","Data":"f40d7c935823dac6cffc38255a0de284b13b2e55d8e292aea965163997d2e4ab"}
Oct 30 13:36:57.519996 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:57.519554    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6" event={"ID":"7224cc91-c4bd-4c4e-99c2-9f721c9f3a71","Type":"ContainerStarted","Data":"3a7e92d033359f62355cda6ea38233c930ae0382c808a3bb1abd79944f8de667"}
Oct 30 13:36:57.564209 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:57.564145    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6" podStartSLOduration=2.564129072 podStartE2EDuration="2.564129072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:57.562759386 +0000 UTC m=+3112.609893338" watchObservedRunningTime="2026-10-30 13:36:57.564129072 +0000 UTC m=+3112.611263077"
Oct 30 13:36:57.624827 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:57.624794    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:57.994326 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:57.994283    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:58.078773 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:58.078738    4544 scope.go:122] "RemoveContainer" containerID="08d559782601d48bd6a4df9982613081aff34bf96da07a3cb884eecb6afd3335"
Oct 30 13:36:58.280880 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:58.280749    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:58.529814 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:58.529779    4544 generic.go:356] "Generic (PLEG): container finished" podID="7224cc91-c4bd-4c4e-99c2-9f721c9f3a71" containerID="f40d7c935823dac6cffc38255a0de284b13b2e55d8e292aea965163997d2e4ab" exitCode=7
Oct 30 13:36:58.530257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:58.529866    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6" event={"ID":"7224cc91-c4bd-4c4e-99c2-9f721c9f3a71","Type":"ContainerDied","Data":"f40d7c935823dac6cffc38255a0de284b13b2e55d8e292aea965163997d2e4ab"}
Oct 30 13:36:58.530341 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:58.530323    4544 scope.go:122] "RemoveContainer" containerID="f40d7c935823dac6cffc38255a0de284b13b2e55d8e292aea965163997d2e4ab"
Oct 30 13:36:59.008848 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:36:59.008809    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:36:59.402329 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:59.402290    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:36:59.541506 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:59.541471    4544 generic.go:356] "Generic (PLEG): container finished" podID="7224cc91-c4bd-4c4e-99c2-9f721c9f3a71" containerID="1f6b8693723fc47bf2d7a978572f06b3a534b951ed349b1c33bd3e0b8b5ac9e4" exitCode=7
Oct 30 13:36:59.542025 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:59.541562    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6" event={"ID":"7224cc91-c4bd-4c4e-99c2-9f721c9f3a71","Type":"ContainerDied","Data":"1f6b8693723fc47bf2d7a978572f06b3a534b951ed349b1c33bd3e0b8b5ac9e4"}
Oct 30 13:36:59.542025 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:59.541635    4544 scope.go:122] "RemoveContainer" containerID="f40d7c935823dac6cffc38255a0de284b13b2e55d8e292aea965163997d2e4ab"
Oct 30 13:36:59.542025 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:59.541953    4544 scope.go:122] "RemoveContainer" containerID="1f6b8693723fc47bf2d7a978572f06b3a534b951ed349b1c33bd3e0b8b5ac9e4"
Oct 30 13:36:59.542559 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:36:59.542536    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6_odh-ai-gateway-infra(7224cc91-c4bd-4c4e-99c2-9f721c9f3a71)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6" podUID="7224cc91-c4bd-4c4e-99c2-9f721c9f3a71"
Oct 30 13:36:59.543899 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:59.543878    4544 generic.go:356] "Generic (PLEG): container finished" podID="33b21dec-41c6-42c7-a0ea-729d5bd768a9" containerID="90a2f61d086be4bf544be770d8ac2041564aff4e62b7567d07b1bf77614ce984" exitCode=7
Oct 30 13:36:59.544008 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:59.543955    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn" event={"ID":"33b21dec-41c6-42c7-a0ea-729d5bd768a9","Type":"ContainerDied","Data":"90a2f61d086be4bf544be770d8ac2041564aff4e62b7567d07b1bf77614ce984"}
Oct 30 13:36:59.544447 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:59.544430    4544 scope.go:122] "RemoveContainer" containerID="90a2f61d086be4bf544be770d8ac2041564aff4e62b7567d07b1bf77614ce984"
Oct 30 13:36:59.544856 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:36:59.544838    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn_odh-ai-gateway-infra(33b21dec-41c6-42c7-a0ea-729d5bd768a9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn" podUID="33b21dec-41c6-42c7-a0ea-729d5bd768a9"
Oct 30 13:36:59.556746 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:36:59.556724    4544 scope.go:122] "RemoveContainer" containerID="08d559782601d48bd6a4df9982613081aff34bf96da07a3cb884eecb6afd3335"
Oct 30 13:37:00.557372 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:00.557331    4544 scope.go:122] "RemoveContainer" containerID="1f6b8693723fc47bf2d7a978572f06b3a534b951ed349b1c33bd3e0b8b5ac9e4"
Oct 30 13:37:00.557818 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:37:00.557799    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6_odh-ai-gateway-infra(7224cc91-c4bd-4c4e-99c2-9f721c9f3a71)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6" podUID="7224cc91-c4bd-4c4e-99c2-9f721c9f3a71"
Oct 30 13:37:00.645132 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:00.645081    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn"]
Oct 30 13:37:00.850011 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:00.849975    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn"
Oct 30 13:37:00.907616 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:00.907575    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33b21dec-41c6-42c7-a0ea-729d5bd768a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b21dec-41c6-42c7-a0ea-729d5bd768a9-maas-api-xks-ca\") pod \"33b21dec-41c6-42c7-a0ea-729d5bd768a9\" (UID: \"33b21dec-41c6-42c7-a0ea-729d5bd768a9\") "
Oct 30 13:37:00.907904 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:00.907884    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33b21dec-41c6-42c7-a0ea-729d5bd768a9-kube-api-access-wd7mh\" (UniqueName: \"kubernetes.io/projected/33b21dec-41c6-42c7-a0ea-729d5bd768a9-kube-api-access-wd7mh\") pod \"33b21dec-41c6-42c7-a0ea-729d5bd768a9\" (UID: \"33b21dec-41c6-42c7-a0ea-729d5bd768a9\") "
Oct 30 13:37:00.908285 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:00.907986    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33b21dec-41c6-42c7-a0ea-729d5bd768a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b21dec-41c6-42c7-a0ea-729d5bd768a9-maas-api-openshift-ca\") pod \"33b21dec-41c6-42c7-a0ea-729d5bd768a9\" (UID: \"33b21dec-41c6-42c7-a0ea-729d5bd768a9\") "
Oct 30 13:37:00.909076 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:00.909031    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b21dec-41c6-42c7-a0ea-729d5bd768a9-maas-api-openshift-ca" pod "33b21dec-41c6-42c7-a0ea-729d5bd768a9" (UID: "33b21dec-41c6-42c7-a0ea-729d5bd768a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:37:00.916287 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:00.911245    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33b21dec-41c6-42c7-a0ea-729d5bd768a9-maas-api-xks-ca" pod "33b21dec-41c6-42c7-a0ea-729d5bd768a9" (UID: "33b21dec-41c6-42c7-a0ea-729d5bd768a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:37:00.916287 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:00.911769    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b21dec-41c6-42c7-a0ea-729d5bd768a9-kube-api-access-wd7mh" pod "33b21dec-41c6-42c7-a0ea-729d5bd768a9" (UID: "33b21dec-41c6-42c7-a0ea-729d5bd768a9"). InnerVolumeSpecName "kube-api-access-wd7mh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:01.010154 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:01.010100    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd7mh\" (UniqueName: \"kubernetes.io/projected/33b21dec-41c6-42c7-a0ea-729d5bd768a9-kube-api-access-wd7mh\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:01.010154 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:01.010141    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b21dec-41c6-42c7-a0ea-729d5bd768a9-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:01.010154 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:01.010155    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b21dec-41c6-42c7-a0ea-729d5bd768a9-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:01.147413 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:01.147368    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:01.568945 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:01.568852    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn"
Oct 30 13:37:01.569500 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:01.568851    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn" event={"ID":"33b21dec-41c6-42c7-a0ea-729d5bd768a9","Type":"ContainerDied","Data":"c7045354ddb7283770f58001d34a9092b670863195a735c93e631c62a4946d71"}
Oct 30 13:37:01.569500 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:01.569001    4544 scope.go:122] "RemoveContainer" containerID="90a2f61d086be4bf544be770d8ac2041564aff4e62b7567d07b1bf77614ce984"
Oct 30 13:37:01.625966 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:01.625934    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn"]
Oct 30 13:37:01.631387 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:01.631356    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-2xzzn"]
Oct 30 13:37:03.089742 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:03.089702    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33b21dec-41c6-42c7-a0ea-729d5bd768a9" path="/var/lib/kubelet/pods/33b21dec-41c6-42c7-a0ea-729d5bd768a9/volumes"
Oct 30 13:37:03.578309 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:03.578272    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:05.362660 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:05.362626    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:07.256246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:07.250477    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:09.003611 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:09.003570    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:11.059484 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:11.059446    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:11.079064 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:11.079027    4544 scope.go:122] "RemoveContainer" containerID="1f6b8693723fc47bf2d7a978572f06b3a534b951ed349b1c33bd3e0b8b5ac9e4"
Oct 30 13:37:11.665179 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:11.665144    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6" event={"ID":"7224cc91-c4bd-4c4e-99c2-9f721c9f3a71","Type":"ContainerStarted","Data":"ef1446e01fe557a53ac1d05c4119a0d3e2b18748eec176c0c596c38fd2379f49"}
Oct 30 13:37:12.338516 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:12.338475    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"]
Oct 30 13:37:12.338940 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:12.338680    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x" podUID="44a8f645-e783-438c-9efe-4196f9e97dc7" containerName="istio-proxy" containerID="cri-o://f3ac95f6bc180f39f26ead4d3d19b71a830374a693482cf2e833f7112fccbfd8" gracePeriod=30
Oct 30 13:37:12.350624 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:12.350591    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"]
Oct 30 13:37:12.350825 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:12.350803    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s" podUID="00a9deb6-b7b1-4870-adc9-8025aa6b49e2" containerName="istio-proxy" containerID="cri-o://206b025f477102937bf636d950a52c8e69f3541117ad2e48489780ad47d50f96" gracePeriod=30
Oct 30 13:37:12.676186 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:12.676150    4544 generic.go:356] "Generic (PLEG): container finished" podID="7224cc91-c4bd-4c4e-99c2-9f721c9f3a71" containerID="ef1446e01fe557a53ac1d05c4119a0d3e2b18748eec176c0c596c38fd2379f49" exitCode=7
Oct 30 13:37:12.676428 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:12.676267    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6" event={"ID":"7224cc91-c4bd-4c4e-99c2-9f721c9f3a71","Type":"ContainerDied","Data":"ef1446e01fe557a53ac1d05c4119a0d3e2b18748eec176c0c596c38fd2379f49"}
Oct 30 13:37:12.676428 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:12.676339    4544 scope.go:122] "RemoveContainer" containerID="1f6b8693723fc47bf2d7a978572f06b3a534b951ed349b1c33bd3e0b8b5ac9e4"
Oct 30 13:37:12.676908 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:12.676885    4544 scope.go:122] "RemoveContainer" containerID="ef1446e01fe557a53ac1d05c4119a0d3e2b18748eec176c0c596c38fd2379f49"
Oct 30 13:37:12.677483 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:37:12.677376    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6_odh-ai-gateway-infra(7224cc91-c4bd-4c4e-99c2-9f721c9f3a71)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6" podUID="7224cc91-c4bd-4c4e-99c2-9f721c9f3a71"
Oct 30 13:37:12.728145 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:12.728108    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6"]
Oct 30 13:37:12.885005 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:12.884974    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:13.869327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:13.869301    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6"
Oct 30 13:37:13.903523 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:13.903489    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7224cc91-c4bd-4c4e-99c2-9f721c9f3a71-kube-api-access-p4vt4\" (UniqueName: \"kubernetes.io/projected/7224cc91-c4bd-4c4e-99c2-9f721c9f3a71-kube-api-access-p4vt4\") pod \"7224cc91-c4bd-4c4e-99c2-9f721c9f3a71\" (UID: \"7224cc91-c4bd-4c4e-99c2-9f721c9f3a71\") "
Oct 30 13:37:13.903740 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:13.903558    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7224cc91-c4bd-4c4e-99c2-9f721c9f3a71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7224cc91-c4bd-4c4e-99c2-9f721c9f3a71-maas-api-xks-ca\") pod \"7224cc91-c4bd-4c4e-99c2-9f721c9f3a71\" (UID: \"7224cc91-c4bd-4c4e-99c2-9f721c9f3a71\") "
Oct 30 13:37:13.903740 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:13.903593    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7224cc91-c4bd-4c4e-99c2-9f721c9f3a71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7224cc91-c4bd-4c4e-99c2-9f721c9f3a71-maas-api-openshift-ca\") pod \"7224cc91-c4bd-4c4e-99c2-9f721c9f3a71\" (UID: \"7224cc91-c4bd-4c4e-99c2-9f721c9f3a71\") "
Oct 30 13:37:13.904009 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:13.903987    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7224cc91-c4bd-4c4e-99c2-9f721c9f3a71-maas-api-openshift-ca" pod "7224cc91-c4bd-4c4e-99c2-9f721c9f3a71" (UID: "7224cc91-c4bd-4c4e-99c2-9f721c9f3a71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:37:13.904212 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:13.904195    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7224cc91-c4bd-4c4e-99c2-9f721c9f3a71-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:13.905810 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:13.905784    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7224cc91-c4bd-4c4e-99c2-9f721c9f3a71-maas-api-xks-ca" pod "7224cc91-c4bd-4c4e-99c2-9f721c9f3a71" (UID: "7224cc91-c4bd-4c4e-99c2-9f721c9f3a71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:37:13.906189 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:13.906164    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7224cc91-c4bd-4c4e-99c2-9f721c9f3a71-kube-api-access-p4vt4" pod "7224cc91-c4bd-4c4e-99c2-9f721c9f3a71" (UID: "7224cc91-c4bd-4c4e-99c2-9f721c9f3a71"). InnerVolumeSpecName "kube-api-access-p4vt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:14.006326 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:14.006194    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4vt4\" (UniqueName: \"kubernetes.io/projected/7224cc91-c4bd-4c4e-99c2-9f721c9f3a71-kube-api-access-p4vt4\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:14.006326 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:14.006278    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7224cc91-c4bd-4c4e-99c2-9f721c9f3a71-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:14.611474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:14.611442    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:14.710685 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:14.710640    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6" event={"ID":"7224cc91-c4bd-4c4e-99c2-9f721c9f3a71","Type":"ContainerDied","Data":"3a7e92d033359f62355cda6ea38233c930ae0382c808a3bb1abd79944f8de667"}
Oct 30 13:37:14.710908 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:14.710725    4544 scope.go:122] "RemoveContainer" containerID="ef1446e01fe557a53ac1d05c4119a0d3e2b18748eec176c0c596c38fd2379f49"
Oct 30 13:37:14.710908 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:14.710795    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6"
Oct 30 13:37:14.779313 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:14.779277    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6"]
Oct 30 13:37:14.786165 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:14.786131    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3b47e7-gfhn6"]
Oct 30 13:37:15.091477 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:15.091438    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7224cc91-c4bd-4c4e-99c2-9f721c9f3a71" path="/var/lib/kubelet/pods/7224cc91-c4bd-4c4e-99c2-9f721c9f3a71/volumes"
Oct 30 13:37:16.324192 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:16.324158    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:17.697793 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.697733    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:37:17.701699 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.701672    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:37:17.741020 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.740988    4544 generic.go:356] "Generic (PLEG): container finished" podID="00a9deb6-b7b1-4870-adc9-8025aa6b49e2" containerID="206b025f477102937bf636d950a52c8e69f3541117ad2e48489780ad47d50f96" exitCode=0
Oct 30 13:37:17.741246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.741055    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"
Oct 30 13:37:17.741246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.741068    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s" event={"ID":"00a9deb6-b7b1-4870-adc9-8025aa6b49e2","Type":"ContainerDied","Data":"206b025f477102937bf636d950a52c8e69f3541117ad2e48489780ad47d50f96"}
Oct 30 13:37:17.741246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.741140    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s" event={"ID":"00a9deb6-b7b1-4870-adc9-8025aa6b49e2","Type":"ContainerDied","Data":"5f4c670861820d5e6e404d494832f970a54502ffe037f7b5e2974c1c161181db"}
Oct 30 13:37:17.741246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.741179    4544 scope.go:122] "RemoveContainer" containerID="206b025f477102937bf636d950a52c8e69f3541117ad2e48489780ad47d50f96"
Oct 30 13:37:17.743059 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.743036    4544 generic.go:356] "Generic (PLEG): container finished" podID="44a8f645-e783-438c-9efe-4196f9e97dc7" containerID="f3ac95f6bc180f39f26ead4d3d19b71a830374a693482cf2e833f7112fccbfd8" exitCode=0
Oct 30 13:37:17.743238 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.743083    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x" event={"ID":"44a8f645-e783-438c-9efe-4196f9e97dc7","Type":"ContainerDied","Data":"f3ac95f6bc180f39f26ead4d3d19b71a830374a693482cf2e833f7112fccbfd8"}
Oct 30 13:37:17.743238 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.743129    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"
Oct 30 13:37:17.743238 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.743128    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x" event={"ID":"44a8f645-e783-438c-9efe-4196f9e97dc7","Type":"ContainerDied","Data":"65acde9178e2bfecb53e4300830dc5d50bb5dbe76ee2261b8e0c5053ff8e430c"}
Oct 30 13:37:17.754511 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.754489    4544 scope.go:122] "RemoveContainer" containerID="206b025f477102937bf636d950a52c8e69f3541117ad2e48489780ad47d50f96"
Oct 30 13:37:17.754853 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:37:17.754828    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"206b025f477102937bf636d950a52c8e69f3541117ad2e48489780ad47d50f96\": container with ID starting with 206b025f477102937bf636d950a52c8e69f3541117ad2e48489780ad47d50f96 not found: ID does not exist" containerID="206b025f477102937bf636d950a52c8e69f3541117ad2e48489780ad47d50f96"
Oct 30 13:37:17.754903 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.754865    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"206b025f477102937bf636d950a52c8e69f3541117ad2e48489780ad47d50f96"} err="failed to get container status \"206b025f477102937bf636d950a52c8e69f3541117ad2e48489780ad47d50f96\": rpc error: code = NotFound desc = could not find container \"206b025f477102937bf636d950a52c8e69f3541117ad2e48489780ad47d50f96\": container with ID starting with 206b025f477102937bf636d950a52c8e69f3541117ad2e48489780ad47d50f96 not found: ID does not exist"
Oct 30 13:37:17.754903 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.754884    4544 scope.go:122] "RemoveContainer" containerID="f3ac95f6bc180f39f26ead4d3d19b71a830374a693482cf2e833f7112fccbfd8"
Oct 30 13:37:17.767139 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.767025    4544 scope.go:122] "RemoveContainer" containerID="f3ac95f6bc180f39f26ead4d3d19b71a830374a693482cf2e833f7112fccbfd8"
Oct 30 13:37:17.767413 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:37:17.767387    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3ac95f6bc180f39f26ead4d3d19b71a830374a693482cf2e833f7112fccbfd8\": container with ID starting with f3ac95f6bc180f39f26ead4d3d19b71a830374a693482cf2e833f7112fccbfd8 not found: ID does not exist" containerID="f3ac95f6bc180f39f26ead4d3d19b71a830374a693482cf2e833f7112fccbfd8"
Oct 30 13:37:17.767514 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.767422    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3ac95f6bc180f39f26ead4d3d19b71a830374a693482cf2e833f7112fccbfd8"} err="failed to get container status \"f3ac95f6bc180f39f26ead4d3d19b71a830374a693482cf2e833f7112fccbfd8\": rpc error: code = NotFound desc = could not find container \"f3ac95f6bc180f39f26ead4d3d19b71a830374a693482cf2e833f7112fccbfd8\": container with ID starting with f3ac95f6bc180f39f26ead4d3d19b71a830374a693482cf2e833f7112fccbfd8 not found: ID does not exist"
Oct 30 13:37:17.774031 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.774003    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44a8f645-e783-438c-9efe-4196f9e97dc7-kube-api-access-47dxc\" (UniqueName: \"kubernetes.io/projected/44a8f645-e783-438c-9efe-4196f9e97dc7-kube-api-access-47dxc\") pod \"44a8f645-e783-438c-9efe-4196f9e97dc7\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") "
Oct 30 13:37:17.774167 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.774083    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-credential-socket\") pod \"44a8f645-e783-438c-9efe-4196f9e97dc7\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") "
Oct 30 13:37:17.774281 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.774168    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-workload-certs\") pod \"44a8f645-e783-438c-9efe-4196f9e97dc7\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") "
Oct 30 13:37:17.774281 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.774203    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istio-envoy\") pod \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") "
Oct 30 13:37:17.774281 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.774247    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-workload-certs\") pod \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") "
Oct 30 13:37:17.774463 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.774335    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istiod-ca-cert\") pod \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") "
Oct 30 13:37:17.774463 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.774370    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istio-token\" (UniqueName: \"kubernetes.io/projected/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istio-token\") pod \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") "
Oct 30 13:37:17.774463 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.774415    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44a8f645-e783-438c-9efe-4196f9e97dc7-istio-token\" (UniqueName: \"kubernetes.io/projected/44a8f645-e783-438c-9efe-4196f9e97dc7-istio-token\") pod \"44a8f645-e783-438c-9efe-4196f9e97dc7\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") "
Oct 30 13:37:17.774463 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.774443    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-credential-socket" pod "44a8f645-e783-438c-9efe-4196f9e97dc7" (UID: "44a8f645-e783-438c-9efe-4196f9e97dc7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:17.774666 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.774462    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-istio-data\") pod \"44a8f645-e783-438c-9efe-4196f9e97dc7\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") "
Oct 30 13:37:17.774666 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.774534    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-kube-api-access-zxs68\" (UniqueName: \"kubernetes.io/projected/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-kube-api-access-zxs68\") pod \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") "
Oct 30 13:37:17.774666 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.774581    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44a8f645-e783-438c-9efe-4196f9e97dc7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a8f645-e783-438c-9efe-4196f9e97dc7-istio-podinfo\") pod \"44a8f645-e783-438c-9efe-4196f9e97dc7\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") "
Oct 30 13:37:17.774666 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.774627    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-workload-certs" pod "00a9deb6-b7b1-4870-adc9-8025aa6b49e2" (UID: "00a9deb6-b7b1-4870-adc9-8025aa6b49e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:17.774666 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.774635    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44a8f645-e783-438c-9efe-4196f9e97dc7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a8f645-e783-438c-9efe-4196f9e97dc7-istiod-ca-cert\") pod \"44a8f645-e783-438c-9efe-4196f9e97dc7\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") "
Oct 30 13:37:17.774946 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.774803    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istio-podinfo\") pod \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") "
Oct 30 13:37:17.774946 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.774848    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-istio-envoy\") pod \"44a8f645-e783-438c-9efe-4196f9e97dc7\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") "
Oct 30 13:37:17.774946 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.774864    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istiod-ca-cert" pod "00a9deb6-b7b1-4870-adc9-8025aa6b49e2" (UID: "00a9deb6-b7b1-4870-adc9-8025aa6b49e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:37:17.774946 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.774904    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istio-data\") pod \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") "
Oct 30 13:37:17.775160 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.775053    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-workload-socket\") pod \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") "
Oct 30 13:37:17.775235 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.775190    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-credential-socket\") pod \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\" (UID: \"00a9deb6-b7b1-4870-adc9-8025aa6b49e2\") "
Oct 30 13:37:17.775303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.775246    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-workload-socket\") pod \"44a8f645-e783-438c-9efe-4196f9e97dc7\" (UID: \"44a8f645-e783-438c-9efe-4196f9e97dc7\") "
Oct 30 13:37:17.778193 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.776362    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:17.778193 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.776403    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:17.778193 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.776419    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:17.778193 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.776565    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-workload-certs" pod "44a8f645-e783-438c-9efe-4196f9e97dc7" (UID: "44a8f645-e783-438c-9efe-4196f9e97dc7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:17.778193 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.777067    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a8f645-e783-438c-9efe-4196f9e97dc7-kube-api-access-47dxc" pod "44a8f645-e783-438c-9efe-4196f9e97dc7" (UID: "44a8f645-e783-438c-9efe-4196f9e97dc7"). InnerVolumeSpecName "kube-api-access-47dxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:17.778193 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.777079    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44a8f645-e783-438c-9efe-4196f9e97dc7-istiod-ca-cert" pod "44a8f645-e783-438c-9efe-4196f9e97dc7" (UID: "44a8f645-e783-438c-9efe-4196f9e97dc7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:37:17.778193 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.777072    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-workload-socket" pod "44a8f645-e783-438c-9efe-4196f9e97dc7" (UID: "44a8f645-e783-438c-9efe-4196f9e97dc7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:17.778193 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.777264    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-istio-data" pod "44a8f645-e783-438c-9efe-4196f9e97dc7" (UID: "44a8f645-e783-438c-9efe-4196f9e97dc7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:17.778193 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.777918    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istio-envoy" pod "00a9deb6-b7b1-4870-adc9-8025aa6b49e2" (UID: "00a9deb6-b7b1-4870-adc9-8025aa6b49e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:17.778854 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.778294    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istio-data" pod "00a9deb6-b7b1-4870-adc9-8025aa6b49e2" (UID: "00a9deb6-b7b1-4870-adc9-8025aa6b49e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:17.778854 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.778378    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istio-token" pod "00a9deb6-b7b1-4870-adc9-8025aa6b49e2" (UID: "00a9deb6-b7b1-4870-adc9-8025aa6b49e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:17.778854 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.778672    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-credential-socket" pod "00a9deb6-b7b1-4870-adc9-8025aa6b49e2" (UID: "00a9deb6-b7b1-4870-adc9-8025aa6b49e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:17.778854 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.778730    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-workload-socket" pod "00a9deb6-b7b1-4870-adc9-8025aa6b49e2" (UID: "00a9deb6-b7b1-4870-adc9-8025aa6b49e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:17.779541 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.779513    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a8f645-e783-438c-9efe-4196f9e97dc7-istio-token" pod "44a8f645-e783-438c-9efe-4196f9e97dc7" (UID: "44a8f645-e783-438c-9efe-4196f9e97dc7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:17.779635 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.779554    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-kube-api-access-zxs68" pod "00a9deb6-b7b1-4870-adc9-8025aa6b49e2" (UID: "00a9deb6-b7b1-4870-adc9-8025aa6b49e2"). InnerVolumeSpecName "kube-api-access-zxs68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:17.779635 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.779405    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istio-podinfo" pod "00a9deb6-b7b1-4870-adc9-8025aa6b49e2" (UID: "00a9deb6-b7b1-4870-adc9-8025aa6b49e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:37:17.780590 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.780560    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44a8f645-e783-438c-9efe-4196f9e97dc7-istio-podinfo" pod "44a8f645-e783-438c-9efe-4196f9e97dc7" (UID: "44a8f645-e783-438c-9efe-4196f9e97dc7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:37:17.780877 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.780850    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-istio-envoy" pod "44a8f645-e783-438c-9efe-4196f9e97dc7" (UID: "44a8f645-e783-438c-9efe-4196f9e97dc7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:17.784333 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.784311    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"]
Oct 30 13:37:17.785345 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.785321    4544 cpu_manager.go:414] "containerMap: removing container" podUID="33b21dec-41c6-42c7-a0ea-729d5bd768a9" containerName="revoke-keys"
Oct 30 13:37:17.785345 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.785341    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b21dec-41c6-42c7-a0ea-729d5bd768a9" containerName="revoke-keys"
Oct 30 13:37:17.785498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.785355    4544 cpu_manager.go:414] "containerMap: removing container" podUID="33b21dec-41c6-42c7-a0ea-729d5bd768a9" containerName="revoke-keys"
Oct 30 13:37:17.785498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.785361    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b21dec-41c6-42c7-a0ea-729d5bd768a9" containerName="revoke-keys"
Oct 30 13:37:17.785498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.785374    4544 cpu_manager.go:414] "containerMap: removing container" podUID="7224cc91-c4bd-4c4e-99c2-9f721c9f3a71" containerName="revoke-keys"
Oct 30 13:37:17.785498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.785379    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7224cc91-c4bd-4c4e-99c2-9f721c9f3a71" containerName="revoke-keys"
Oct 30 13:37:17.785498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.785408    4544 cpu_manager.go:414] "containerMap: removing container" podUID="00a9deb6-b7b1-4870-adc9-8025aa6b49e2" containerName="istio-proxy"
Oct 30 13:37:17.785498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.785413    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00a9deb6-b7b1-4870-adc9-8025aa6b49e2" containerName="istio-proxy"
Oct 30 13:37:17.785498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.785432    4544 cpu_manager.go:414] "containerMap: removing container" podUID="44a8f645-e783-438c-9efe-4196f9e97dc7" containerName="istio-proxy"
Oct 30 13:37:17.785498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.785438    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44a8f645-e783-438c-9efe-4196f9e97dc7" containerName="istio-proxy"
Oct 30 13:37:17.785498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.785451    4544 cpu_manager.go:414] "containerMap: removing container" podUID="7224cc91-c4bd-4c4e-99c2-9f721c9f3a71" containerName="revoke-keys"
Oct 30 13:37:17.785498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.785456    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7224cc91-c4bd-4c4e-99c2-9f721c9f3a71" containerName="revoke-keys"
Oct 30 13:37:17.786144 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.785587    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="00a9deb6-b7b1-4870-adc9-8025aa6b49e2" containerName="istio-proxy"
Oct 30 13:37:17.786144 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.785596    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="7224cc91-c4bd-4c4e-99c2-9f721c9f3a71" containerName="revoke-keys"
Oct 30 13:37:17.786144 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.785604    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="44a8f645-e783-438c-9efe-4196f9e97dc7" containerName="istio-proxy"
Oct 30 13:37:17.786144 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.785611    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b21dec-41c6-42c7-a0ea-729d5bd768a9" containerName="revoke-keys"
Oct 30 13:37:17.786144 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.785616    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="7224cc91-c4bd-4c4e-99c2-9f721c9f3a71" containerName="revoke-keys"
Oct 30 13:37:17.786144 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.785626    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b21dec-41c6-42c7-a0ea-729d5bd768a9" containerName="revoke-keys"
Oct 30 13:37:17.791127 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.791106    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:37:17.837399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.834152    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"]
Oct 30 13:37:17.846182 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.846142    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"]
Oct 30 13:37:17.849057 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.848314    4544 cpu_manager.go:414] "containerMap: removing container" podUID="7224cc91-c4bd-4c4e-99c2-9f721c9f3a71" containerName="revoke-keys"
Oct 30 13:37:17.849057 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.848337    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7224cc91-c4bd-4c4e-99c2-9f721c9f3a71" containerName="revoke-keys"
Oct 30 13:37:17.849057 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.848546    4544 cpu_manager.go:414] "containerMap: removing container" podUID="33b21dec-41c6-42c7-a0ea-729d5bd768a9" containerName="revoke-keys"
Oct 30 13:37:17.849057 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.848557    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b21dec-41c6-42c7-a0ea-729d5bd768a9" containerName="revoke-keys"
Oct 30 13:37:17.849057 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.848998    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="7224cc91-c4bd-4c4e-99c2-9f721c9f3a71" containerName="revoke-keys"
Oct 30 13:37:17.849057 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.849058    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b21dec-41c6-42c7-a0ea-729d5bd768a9" containerName="revoke-keys"
Oct 30 13:37:17.869358 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.869278    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:37:17.878974 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.878941    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b5gf\" (UniqueName: \"kubernetes.io/projected/128497c2-9f64-4902-a913-5acb1489b88b-kube-api-access-6b5gf\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:37:17.879283 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.879255    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-workload-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:37:17.879423 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.879405    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-workload-certs\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:37:17.879497 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.879459    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-istio-envoy\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:37:17.879559 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.879511    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/128497c2-9f64-4902-a913-5acb1489b88b-istio-podinfo\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:37:17.879723 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.879705    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/128497c2-9f64-4902-a913-5acb1489b88b-istio-token\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:37:17.879791 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.879772    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/128497c2-9f64-4902-a913-5acb1489b88b-istiod-ca-cert\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:37:17.879886 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.879871    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-credential-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:37:17.879946 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.879905    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-istio-data\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:37:17.880117 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.880099    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:17.880264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.880247    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:17.880383 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.880368    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:17.880483 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.880470    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:17.880577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.880563    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47dxc\" (UniqueName: \"kubernetes.io/projected/44a8f645-e783-438c-9efe-4196f9e97dc7-kube-api-access-47dxc\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:17.880677 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.880661    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:17.880749 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.880681    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:17.880749 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.880693    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:17.880749 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.880707    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a8f645-e783-438c-9efe-4196f9e97dc7-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:17.880749 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.880725    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:17.880749 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.880739    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxs68\" (UniqueName: \"kubernetes.io/projected/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-kube-api-access-zxs68\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:17.881019 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.880757    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a8f645-e783-438c-9efe-4196f9e97dc7-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:17.881019 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.880773    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a8f645-e783-438c-9efe-4196f9e97dc7-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:17.881019 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.880796    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00a9deb6-b7b1-4870-adc9-8025aa6b49e2-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:17.881019 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.880810    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a8f645-e783-438c-9efe-4196f9e97dc7-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:17.883991 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.883958    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"]
Oct 30 13:37:17.982712 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.982656    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxx8g\" (UniqueName: \"kubernetes.io/projected/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-kube-api-access-mxx8g\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:37:17.982899 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.982859    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/128497c2-9f64-4902-a913-5acb1489b88b-istio-token\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:37:17.982954 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.982909    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/128497c2-9f64-4902-a913-5acb1489b88b-istiod-ca-cert\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:37:17.982992 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.982976    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-credential-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:37:17.983127 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.983100    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-istio-data\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:37:17.983180 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.983150    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-workload-certs\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:37:17.983256 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.983178    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istio-envoy\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:37:17.983320 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.983267    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istio-token\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:37:17.983366 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.983353    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-credential-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:37:17.983424 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.983404    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istio-podinfo\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:37:17.983491 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.983431    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-credential-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:37:17.983531 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.983503    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-istio-data\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:37:17.983586 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.983557    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b5gf\" (UniqueName: \"kubernetes.io/projected/128497c2-9f64-4902-a913-5acb1489b88b-kube-api-access-6b5gf\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:37:17.983653 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.983623    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istiod-ca-cert\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:37:17.983729 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.983711    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/128497c2-9f64-4902-a913-5acb1489b88b-istiod-ca-cert\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:37:17.983797 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.983745    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-workload-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:37:17.983797 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.983785    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-workload-certs\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:37:17.983914 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.983822    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-workload-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:37:17.983914 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.983863    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-istio-envoy\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:37:17.983914 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.983904    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istio-data\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:37:17.984085 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.983975    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/128497c2-9f64-4902-a913-5acb1489b88b-istio-podinfo\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:37:17.984085 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.983990    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-workload-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:37:17.984189 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.984109    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-workload-certs\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:37:17.986197 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.986174    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-istio-envoy\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:37:17.986698 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:17.986673    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/128497c2-9f64-4902-a913-5acb1489b88b-istio-podinfo\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:37:18.007583 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.007551    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/128497c2-9f64-4902-a913-5acb1489b88b-istio-token\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:37:18.007727 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.007670    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b5gf\" (UniqueName: \"kubernetes.io/projected/128497c2-9f64-4902-a913-5acb1489b88b-kube-api-access-6b5gf\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:37:18.086075 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.086042    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-workload-certs\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:37:18.086075 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.086080    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istio-envoy\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:37:18.086379 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.086104    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istio-token\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:37:18.086379 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.086134    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-credential-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:37:18.086379 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.086262    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istio-podinfo\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:37:18.086379 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.086363    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istiod-ca-cert\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:37:18.086618 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.086482    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-workload-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:37:18.086618 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.086492    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-workload-certs\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:37:18.086618 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.086564    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-credential-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:37:18.086618 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.086597    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istio-data\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:37:18.086853 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.086745    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxx8g\" (UniqueName: \"kubernetes.io/projected/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-kube-api-access-mxx8g\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:37:18.086853 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.086805    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-workload-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:37:18.086963 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.086922    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istio-data\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:37:18.087180 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.087160    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istiod-ca-cert\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:37:18.088704 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.088684    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istio-envoy\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:37:18.089195 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.089174    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istio-podinfo\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:37:18.099453 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.099420    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"]
Oct 30 13:37:18.104833 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.104800    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x"]
Oct 30 13:37:18.109830 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.109803    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:37:18.109967 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.109845    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istio-token\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:37:18.110921 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.110900    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxx8g\" (UniqueName: \"kubernetes.io/projected/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-kube-api-access-mxx8g\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:37:18.139537 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.139501    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"]
Oct 30 13:37:18.145908 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.145869    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s"]
Oct 30 13:37:18.192282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.191055    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:37:18.235027 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.234992    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:18.448295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.448267    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"]
Oct 30 13:37:18.450647 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:37:18.450617    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod128497c2_9f64_4902_a913_5acb1489b88b.slice/crio-4b8c6d5ec853ae3fab61383f583f9be75bae9d1f9544578643665b38b6c48071 WatchSource:0}: Error finding container 4b8c6d5ec853ae3fab61383f583f9be75bae9d1f9544578643665b38b6c48071: Status 404 returned error can't find the container with id 4b8c6d5ec853ae3fab61383f583f9be75bae9d1f9544578643665b38b6c48071
Oct 30 13:37:18.454465 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.454421    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:37:18.454576 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.454519    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:37:18.454576 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.454565    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:37:18.544247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.544199    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"]
Oct 30 13:37:18.545013 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:37:18.544983    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62db3dba_ac93_4845_9e85_6d83a6fb0cb1.slice/crio-5ca4caf10c1bc0512254af24b5835c36eaf56ffb06d3b56ba15da593e3c7aee1 WatchSource:0}: Error finding container 5ca4caf10c1bc0512254af24b5835c36eaf56ffb06d3b56ba15da593e3c7aee1: Status 404 returned error can't find the container with id 5ca4caf10c1bc0512254af24b5835c36eaf56ffb06d3b56ba15da593e3c7aee1
Oct 30 13:37:18.548387 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.548345    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:37:18.548485 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.548430    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:37:18.548485 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.548460    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:37:18.757392 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.757350    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww" event={"ID":"62db3dba-ac93-4845-9e85-6d83a6fb0cb1","Type":"ContainerStarted","Data":"93aecdd1f121efade27a99382f811c8b68b4aa47009bbcb61c09ecd3ba3a3096"}
Oct 30 13:37:18.757784 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.757409    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww" event={"ID":"62db3dba-ac93-4845-9e85-6d83a6fb0cb1","Type":"ContainerStarted","Data":"5ca4caf10c1bc0512254af24b5835c36eaf56ffb06d3b56ba15da593e3c7aee1"}
Oct 30 13:37:18.762410 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.762373    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k" event={"ID":"128497c2-9f64-4902-a913-5acb1489b88b","Type":"ContainerStarted","Data":"ca0c9e30de327cc6d9419739768d5a50e4af39540c170332e3e4222dae75619e"}
Oct 30 13:37:18.762578 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.762438    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k" event={"ID":"128497c2-9f64-4902-a913-5acb1489b88b","Type":"ContainerStarted","Data":"4b8c6d5ec853ae3fab61383f583f9be75bae9d1f9544578643665b38b6c48071"}
Oct 30 13:37:18.812588 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.812527    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww" podStartSLOduration=41.812507916 podStartE2EDuration="41.812507916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:18.805616473 +0000 UTC m=+3133.852750428" watchObservedRunningTime="2026-10-30 13:37:18.812507916 +0000 UTC m=+3133.859641944"
Oct 30 13:37:18.864991 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:18.864919    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k" podStartSLOduration=41.86490521 podStartE2EDuration="41.86490521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:18.863661551 +0000 UTC m=+3133.910795514" watchObservedRunningTime="2026-10-30 13:37:18.86490521 +0000 UTC m=+3133.912039162"
Oct 30 13:37:19.092152 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:19.092116    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00a9deb6-b7b1-4870-adc9-8025aa6b49e2" path="/var/lib/kubelet/pods/00a9deb6-b7b1-4870-adc9-8025aa6b49e2/volumes"
Oct 30 13:37:19.092911 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:19.092890    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44a8f645-e783-438c-9efe-4196f9e97dc7" path="/var/lib/kubelet/pods/44a8f645-e783-438c-9efe-4196f9e97dc7/volumes"
Oct 30 13:37:19.110549 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:19.110517    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:37:19.112077 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:19.112046    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" start-of-body=
Oct 30 13:37:19.112280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:19.112100    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k" podUID="128497c2-9f64-4902-a913-5acb1489b88b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused"
Oct 30 13:37:19.191968 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:19.191926    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:37:19.199820 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:19.197598    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" start-of-body=
Oct 30 13:37:19.199820 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:19.197675    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww" podUID="62db3dba-ac93-4845-9e85-6d83a6fb0cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused"
Oct 30 13:37:19.883113 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:19.883049    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:20.081243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:20.080952    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:20.111381 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:20.111341    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" start-of-body=
Oct 30 13:37:20.111592 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:20.111419    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k" podUID="128497c2-9f64-4902-a913-5acb1489b88b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused"
Oct 30 13:37:20.192366 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:20.192280    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" start-of-body=
Oct 30 13:37:20.192366 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:20.192334    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww" podUID="62db3dba-ac93-4845-9e85-6d83a6fb0cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused"
Oct 30 13:37:20.202112 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:20.202077    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:20.807246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:20.802857    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:20.966255 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:20.957184    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:21.099700 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:21.099667    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c1b28-gptll"]
Oct 30 13:37:21.099866 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:21.099713    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c1b28-gptll"]
Oct 30 13:37:21.111199 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:21.111157    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" start-of-body=
Oct 30 13:37:21.111384 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:21.111264    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k" podUID="128497c2-9f64-4902-a913-5acb1489b88b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused"
Oct 30 13:37:21.166247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:21.166157    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:21.201262 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:21.198657    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" start-of-body=
Oct 30 13:37:21.201262 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:21.198724    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww" podUID="62db3dba-ac93-4845-9e85-6d83a6fb0cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused"
Oct 30 13:37:21.611270 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:21.611239    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:22.120244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:22.110591    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" start-of-body=
Oct 30 13:37:22.120244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:22.110668    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k" podUID="128497c2-9f64-4902-a913-5acb1489b88b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused"
Oct 30 13:37:22.199307 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:22.192041    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" start-of-body=
Oct 30 13:37:22.199307 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:22.192138    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww" podUID="62db3dba-ac93-4845-9e85-6d83a6fb0cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused"
Oct 30 13:37:23.091724 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:23.091686    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cc129d7-3824-476b-b201-48f5b6aa2106" path="/var/lib/kubelet/pods/9cc129d7-3824-476b-b201-48f5b6aa2106/volumes"
Oct 30 13:37:23.111342 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:23.111295    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" start-of-body=
Oct 30 13:37:23.111520 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:23.111366    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k" podUID="128497c2-9f64-4902-a913-5acb1489b88b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused"
Oct 30 13:37:23.191759 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:23.191709    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" start-of-body=
Oct 30 13:37:23.192154 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:23.191768    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww" podUID="62db3dba-ac93-4845-9e85-6d83a6fb0cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused"
Oct 30 13:37:24.195292 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:24.179546    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" start-of-body=
Oct 30 13:37:24.195292 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:24.179609    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k" podUID="128497c2-9f64-4902-a913-5acb1489b88b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused"
Oct 30 13:37:24.208258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:24.204165    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" start-of-body=
Oct 30 13:37:24.208761 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:24.208536    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww" podUID="62db3dba-ac93-4845-9e85-6d83a6fb0cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused"
Oct 30 13:37:24.674411 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:24.674373    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:24.809843 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:24.809807    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:24.932309 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:24.927176    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:25.213314 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:25.211386    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" start-of-body=
Oct 30 13:37:25.213314 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:25.211455    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww" podUID="62db3dba-ac93-4845-9e85-6d83a6fb0cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused"
Oct 30 13:37:25.273249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:25.224625    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" start-of-body=
Oct 30 13:37:25.273249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:25.224713    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k" podUID="128497c2-9f64-4902-a913-5acb1489b88b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused"
Oct 30 13:37:26.117304 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:26.117248    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:37:26.117788 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:26.117754    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:37:26.118753 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:26.118725    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:37:26.202050 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:26.202015    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:37:26.202806 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:26.202779    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:37:26.203561 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:26.203537    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:37:26.432380 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:26.432295    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:26.551827 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:26.551792    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:27.922814 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:27.922774    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:28.349390 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:28.349350    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:28.514179 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:28.514147    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:28.696177 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:28.696084    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"]
Oct 30 13:37:28.696669 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:28.696629    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj" podUID="66d450d8-b5a4-406f-a427-015307e5a9f6" containerName="istio-proxy" containerID="cri-o://a5d7aabf9d2045725d76d8bf5c5bea75194491daa70b774fbc0d32e810bfb8fe" gracePeriod=30
Oct 30 13:37:28.712112 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:28.712079    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"]
Oct 30 13:37:28.712384 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:28.712349    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt" podUID="10885581-4aa8-4607-b81a-158066a7dd27" containerName="istio-proxy" containerID="cri-o://b1b6e8f55f2d229518c72cc3c354d25fbe7cf48ea872859b6dc54fecebbaf3ec" gracePeriod=30
Oct 30 13:37:28.745485 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:28.745443    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:29.127920 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:37:29.127886    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:37:29.296641 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:29.296606    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl"]
Oct 30 13:37:29.303609 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:29.303585    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl"
Oct 30 13:37:29.321934 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:29.321904    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl"]
Oct 30 13:37:29.375975 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:29.375941    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79rz4\" (UniqueName: \"kubernetes.io/projected/eb8eebdc-5933-463b-a511-264945e917e9-kube-api-access-79rz4\") pod \"maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl\" (UID: \"eb8eebdc-5933-463b-a511-264945e917e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl"
Oct 30 13:37:29.376160 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:29.376082    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb8eebdc-5933-463b-a511-264945e917e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl\" (UID: \"eb8eebdc-5933-463b-a511-264945e917e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl"
Oct 30 13:37:29.376205 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:29.376175    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb8eebdc-5933-463b-a511-264945e917e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl\" (UID: \"eb8eebdc-5933-463b-a511-264945e917e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl"
Oct 30 13:37:29.400298 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:29.400193    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:37:29.400463 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:29.400298    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:37:29.478302 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:29.478269    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79rz4\" (UniqueName: \"kubernetes.io/projected/eb8eebdc-5933-463b-a511-264945e917e9-kube-api-access-79rz4\") pod \"maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl\" (UID: \"eb8eebdc-5933-463b-a511-264945e917e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl"
Oct 30 13:37:29.478513 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:29.478389    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb8eebdc-5933-463b-a511-264945e917e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl\" (UID: \"eb8eebdc-5933-463b-a511-264945e917e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl"
Oct 30 13:37:29.478513 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:29.478479    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb8eebdc-5933-463b-a511-264945e917e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl\" (UID: \"eb8eebdc-5933-463b-a511-264945e917e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl"
Oct 30 13:37:29.479091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:29.479067    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb8eebdc-5933-463b-a511-264945e917e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl\" (UID: \"eb8eebdc-5933-463b-a511-264945e917e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl"
Oct 30 13:37:29.480967 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:29.480945    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb8eebdc-5933-463b-a511-264945e917e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl\" (UID: \"eb8eebdc-5933-463b-a511-264945e917e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl"
Oct 30 13:37:29.504658 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:29.504624    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79rz4\" (UniqueName: \"kubernetes.io/projected/eb8eebdc-5933-463b-a511-264945e917e9-kube-api-access-79rz4\") pod \"maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl\" (UID: \"eb8eebdc-5933-463b-a511-264945e917e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl"
Oct 30 13:37:29.571937 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:29.571895    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:29.618726 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:29.618693    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl"
Oct 30 13:37:30.333592 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:30.332348    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl"]
Oct 30 13:37:30.442411 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:30.442376    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:30.663554 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:30.663520    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:30.724052 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:30.724010    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-03b5b3-t44vs"]
Oct 30 13:37:30.729350 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:30.729321    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-03b5b3-t44vs"]
Oct 30 13:37:30.811485 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:30.811454    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:31.088993 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:31.088960    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff5dc740-a6f8-4f2b-94cc-cade624c2bdf" path="/var/lib/kubelet/pods/ff5dc740-a6f8-4f2b-94cc-cade624c2bdf/volumes"
Oct 30 13:37:31.236912 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:31.236882    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:31.351527 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:31.351450    4544 generic.go:356] "Generic (PLEG): container finished" podID="eb8eebdc-5933-463b-a511-264945e917e9" containerID="6713483c58665c9edbd6baa6f31c9fe6758b172daee73af642d52785aec898be" exitCode=7
Oct 30 13:37:31.351527 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:31.351511    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl" event={"ID":"eb8eebdc-5933-463b-a511-264945e917e9","Type":"ContainerDied","Data":"6713483c58665c9edbd6baa6f31c9fe6758b172daee73af642d52785aec898be"}
Oct 30 13:37:31.352016 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:31.351562    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl" event={"ID":"eb8eebdc-5933-463b-a511-264945e917e9","Type":"ContainerStarted","Data":"faa0f508ff6e00f6587b279c7b25d9564ca5de28d837846e4d1d3f509dc63f51"}
Oct 30 13:37:31.352016 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:31.351895    4544 scope.go:122] "RemoveContainer" containerID="6713483c58665c9edbd6baa6f31c9fe6758b172daee73af642d52785aec898be"
Oct 30 13:37:32.362339 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:32.362299    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl" event={"ID":"eb8eebdc-5933-463b-a511-264945e917e9","Type":"ContainerStarted","Data":"4c0b0f15d3a23f14b42ae835200a1924674449a5d5e370e3277514886209cb08"}
Oct 30 13:37:32.409109 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:32.409054    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl" podStartSLOduration=3.409039654 podStartE2EDuration="3.409039654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:32.403637531 +0000 UTC m=+3147.450771496" watchObservedRunningTime="2026-10-30 13:37:32.409039654 +0000 UTC m=+3147.456173607"
Oct 30 13:37:32.955608 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:32.955575    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5"]
Oct 30 13:37:32.962444 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:32.962417    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5"
Oct 30 13:37:33.001104 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:33.001066    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5"]
Oct 30 13:37:33.064116 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:33.064065    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01e24b49-04cb-4af7-a9cc-a0b6fa608451-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5\" (UID: \"01e24b49-04cb-4af7-a9cc-a0b6fa608451\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5"
Oct 30 13:37:33.064116 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:33.064121    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01e24b49-04cb-4af7-a9cc-a0b6fa608451-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5\" (UID: \"01e24b49-04cb-4af7-a9cc-a0b6fa608451\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5"
Oct 30 13:37:33.064460 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:33.064431    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vghrl\" (UniqueName: \"kubernetes.io/projected/01e24b49-04cb-4af7-a9cc-a0b6fa608451-kube-api-access-vghrl\") pod \"maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5\" (UID: \"01e24b49-04cb-4af7-a9cc-a0b6fa608451\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5"
Oct 30 13:37:33.166002 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:33.165962    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vghrl\" (UniqueName: \"kubernetes.io/projected/01e24b49-04cb-4af7-a9cc-a0b6fa608451-kube-api-access-vghrl\") pod \"maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5\" (UID: \"01e24b49-04cb-4af7-a9cc-a0b6fa608451\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5"
Oct 30 13:37:33.166359 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:33.166332    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01e24b49-04cb-4af7-a9cc-a0b6fa608451-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5\" (UID: \"01e24b49-04cb-4af7-a9cc-a0b6fa608451\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5"
Oct 30 13:37:33.166516 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:33.166400    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01e24b49-04cb-4af7-a9cc-a0b6fa608451-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5\" (UID: \"01e24b49-04cb-4af7-a9cc-a0b6fa608451\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5"
Oct 30 13:37:33.167149 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:33.167124    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01e24b49-04cb-4af7-a9cc-a0b6fa608451-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5\" (UID: \"01e24b49-04cb-4af7-a9cc-a0b6fa608451\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5"
Oct 30 13:37:33.168946 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:33.168918    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01e24b49-04cb-4af7-a9cc-a0b6fa608451-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5\" (UID: \"01e24b49-04cb-4af7-a9cc-a0b6fa608451\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5"
Oct 30 13:37:33.189372 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:33.189339    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vghrl\" (UniqueName: \"kubernetes.io/projected/01e24b49-04cb-4af7-a9cc-a0b6fa608451-kube-api-access-vghrl\") pod \"maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5\" (UID: \"01e24b49-04cb-4af7-a9cc-a0b6fa608451\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5"
Oct 30 13:37:33.277904 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:33.277814    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5"
Oct 30 13:37:33.381440 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:33.379979    4544 generic.go:356] "Generic (PLEG): container finished" podID="eb8eebdc-5933-463b-a511-264945e917e9" containerID="4c0b0f15d3a23f14b42ae835200a1924674449a5d5e370e3277514886209cb08" exitCode=7
Oct 30 13:37:33.381440 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:33.380074    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl" event={"ID":"eb8eebdc-5933-463b-a511-264945e917e9","Type":"ContainerDied","Data":"4c0b0f15d3a23f14b42ae835200a1924674449a5d5e370e3277514886209cb08"}
Oct 30 13:37:33.381440 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:33.380135    4544 scope.go:122] "RemoveContainer" containerID="6713483c58665c9edbd6baa6f31c9fe6758b172daee73af642d52785aec898be"
Oct 30 13:37:33.381440 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:33.380650    4544 scope.go:122] "RemoveContainer" containerID="4c0b0f15d3a23f14b42ae835200a1924674449a5d5e370e3277514886209cb08"
Oct 30 13:37:33.381440 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:37:33.381140    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl_odh-ai-gateway-infra(eb8eebdc-5933-463b-a511-264945e917e9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl" podUID="eb8eebdc-5933-463b-a511-264945e917e9"
Oct 30 13:37:33.511588 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:33.511553    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:33.647247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:33.643814    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5"]
Oct 30 13:37:34.026701 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.026666    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:37:34.040711 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.040683    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:37:34.091557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.091503    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt"]
Oct 30 13:37:34.093146 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.093111    4544 cpu_manager.go:414] "containerMap: removing container" podUID="10885581-4aa8-4607-b81a-158066a7dd27" containerName="istio-proxy"
Oct 30 13:37:34.093146 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.093145    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10885581-4aa8-4607-b81a-158066a7dd27" containerName="istio-proxy"
Oct 30 13:37:34.093442 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.093190    4544 cpu_manager.go:414] "containerMap: removing container" podUID="66d450d8-b5a4-406f-a427-015307e5a9f6" containerName="istio-proxy"
Oct 30 13:37:34.093442 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.093196    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66d450d8-b5a4-406f-a427-015307e5a9f6" containerName="istio-proxy"
Oct 30 13:37:34.093442 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.093439    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="10885581-4aa8-4607-b81a-158066a7dd27" containerName="istio-proxy"
Oct 30 13:37:34.093547 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.093458    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="66d450d8-b5a4-406f-a427-015307e5a9f6" containerName="istio-proxy"
Oct 30 13:37:34.099204 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.099159    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt"
Oct 30 13:37:34.126997 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.123638    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-78543b-serving-cert\""
Oct 30 13:37:34.143924 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.143721    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt"]
Oct 30 13:37:34.162542 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.162429    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"]
Oct 30 13:37:34.172471 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.169333    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:37:34.189985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.186830    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"]
Oct 30 13:37:34.189985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.188100    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10885581-4aa8-4607-b81a-158066a7dd27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10885581-4aa8-4607-b81a-158066a7dd27-istiod-ca-cert\") pod \"10885581-4aa8-4607-b81a-158066a7dd27\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") "
Oct 30 13:37:34.189985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.188157    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-workload-certs\") pod \"10885581-4aa8-4607-b81a-158066a7dd27\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") "
Oct 30 13:37:34.189985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.188254    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-istio-data\") pod \"10885581-4aa8-4607-b81a-158066a7dd27\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") "
Oct 30 13:37:34.189985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.188336    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-credential-socket\") pod \"10885581-4aa8-4607-b81a-158066a7dd27\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") "
Oct 30 13:37:34.189985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.188388    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-workload-certs\") pod \"66d450d8-b5a4-406f-a427-015307e5a9f6\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") "
Oct 30 13:37:34.189985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.188426    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10885581-4aa8-4607-b81a-158066a7dd27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10885581-4aa8-4607-b81a-158066a7dd27-istio-podinfo\") pod \"10885581-4aa8-4607-b81a-158066a7dd27\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") "
Oct 30 13:37:34.189985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.188451    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66d450d8-b5a4-406f-a427-015307e5a9f6-istio-token\" (UniqueName: \"kubernetes.io/projected/66d450d8-b5a4-406f-a427-015307e5a9f6-istio-token\") pod \"66d450d8-b5a4-406f-a427-015307e5a9f6\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") "
Oct 30 13:37:34.189985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.188480    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-istio-envoy\") pod \"66d450d8-b5a4-406f-a427-015307e5a9f6\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") "
Oct 30 13:37:34.189985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.188602    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66d450d8-b5a4-406f-a427-015307e5a9f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66d450d8-b5a4-406f-a427-015307e5a9f6-istio-podinfo\") pod \"66d450d8-b5a4-406f-a427-015307e5a9f6\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") "
Oct 30 13:37:34.189985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.188658    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-credential-socket\") pod \"66d450d8-b5a4-406f-a427-015307e5a9f6\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") "
Oct 30 13:37:34.189985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.188783    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-istio-data\") pod \"66d450d8-b5a4-406f-a427-015307e5a9f6\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") "
Oct 30 13:37:34.189985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.188859    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10885581-4aa8-4607-b81a-158066a7dd27-istio-token\" (UniqueName: \"kubernetes.io/projected/10885581-4aa8-4607-b81a-158066a7dd27-istio-token\") pod \"10885581-4aa8-4607-b81a-158066a7dd27\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") "
Oct 30 13:37:34.189985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.188951    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66d450d8-b5a4-406f-a427-015307e5a9f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66d450d8-b5a4-406f-a427-015307e5a9f6-istiod-ca-cert\") pod \"66d450d8-b5a4-406f-a427-015307e5a9f6\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") "
Oct 30 13:37:34.189985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.189019    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10885581-4aa8-4607-b81a-158066a7dd27-kube-api-access-vdr7h\" (UniqueName: \"kubernetes.io/projected/10885581-4aa8-4607-b81a-158066a7dd27-kube-api-access-vdr7h\") pod \"10885581-4aa8-4607-b81a-158066a7dd27\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") "
Oct 30 13:37:34.189985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.189080    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66d450d8-b5a4-406f-a427-015307e5a9f6-kube-api-access-tqm7d\" (UniqueName: \"kubernetes.io/projected/66d450d8-b5a4-406f-a427-015307e5a9f6-kube-api-access-tqm7d\") pod \"66d450d8-b5a4-406f-a427-015307e5a9f6\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") "
Oct 30 13:37:34.189985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.189154    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-istio-envoy\") pod \"10885581-4aa8-4607-b81a-158066a7dd27\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") "
Oct 30 13:37:34.189985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.189189    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-workload-socket\") pod \"66d450d8-b5a4-406f-a427-015307e5a9f6\" (UID: \"66d450d8-b5a4-406f-a427-015307e5a9f6\") "
Oct 30 13:37:34.189985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.189232    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-workload-socket\") pod \"10885581-4aa8-4607-b81a-158066a7dd27\" (UID: \"10885581-4aa8-4607-b81a-158066a7dd27\") "
Oct 30 13:37:34.191674 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.191238    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-istio-data" pod "66d450d8-b5a4-406f-a427-015307e5a9f6" (UID: "66d450d8-b5a4-406f-a427-015307e5a9f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:34.191674 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.191569    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10885581-4aa8-4607-b81a-158066a7dd27-istiod-ca-cert" pod "10885581-4aa8-4607-b81a-158066a7dd27" (UID: "10885581-4aa8-4607-b81a-158066a7dd27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:37:34.191674 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.191614    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.195617 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.193759    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-istio-data" pod "10885581-4aa8-4607-b81a-158066a7dd27" (UID: "10885581-4aa8-4607-b81a-158066a7dd27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:34.195617 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.194100    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-workload-certs" pod "10885581-4aa8-4607-b81a-158066a7dd27" (UID: "10885581-4aa8-4607-b81a-158066a7dd27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:34.195617 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.194412    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-credential-socket" pod "10885581-4aa8-4607-b81a-158066a7dd27" (UID: "10885581-4aa8-4607-b81a-158066a7dd27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:34.195617 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.194587    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-istio-envoy" pod "66d450d8-b5a4-406f-a427-015307e5a9f6" (UID: "66d450d8-b5a4-406f-a427-015307e5a9f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:34.195617 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.194884    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-workload-certs" pod "66d450d8-b5a4-406f-a427-015307e5a9f6" (UID: "66d450d8-b5a4-406f-a427-015307e5a9f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:34.195617 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.195141    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-credential-socket" pod "66d450d8-b5a4-406f-a427-015307e5a9f6" (UID: "66d450d8-b5a4-406f-a427-015307e5a9f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:34.197319 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.195923    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d450d8-b5a4-406f-a427-015307e5a9f6-istiod-ca-cert" pod "66d450d8-b5a4-406f-a427-015307e5a9f6" (UID: "66d450d8-b5a4-406f-a427-015307e5a9f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:37:34.197923 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.197896    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66d450d8-b5a4-406f-a427-015307e5a9f6-istio-token" pod "66d450d8-b5a4-406f-a427-015307e5a9f6" (UID: "66d450d8-b5a4-406f-a427-015307e5a9f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:34.199282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.198884    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10885581-4aa8-4607-b81a-158066a7dd27-istio-token" pod "10885581-4aa8-4607-b81a-158066a7dd27" (UID: "10885581-4aa8-4607-b81a-158066a7dd27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:34.199770 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.199360    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-workload-socket" pod "66d450d8-b5a4-406f-a427-015307e5a9f6" (UID: "66d450d8-b5a4-406f-a427-015307e5a9f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:34.199770 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.199589    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66d450d8-b5a4-406f-a427-015307e5a9f6-istio-podinfo" pod "66d450d8-b5a4-406f-a427-015307e5a9f6" (UID: "66d450d8-b5a4-406f-a427-015307e5a9f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:37:34.201264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.199987    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-workload-socket" pod "10885581-4aa8-4607-b81a-158066a7dd27" (UID: "10885581-4aa8-4607-b81a-158066a7dd27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:34.201894 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.201864    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10885581-4aa8-4607-b81a-158066a7dd27-istio-podinfo" pod "10885581-4aa8-4607-b81a-158066a7dd27" (UID: "10885581-4aa8-4607-b81a-158066a7dd27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:37:34.203067 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.203036    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10885581-4aa8-4607-b81a-158066a7dd27-kube-api-access-vdr7h" pod "10885581-4aa8-4607-b81a-158066a7dd27" (UID: "10885581-4aa8-4607-b81a-158066a7dd27"). InnerVolumeSpecName "kube-api-access-vdr7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:34.203497 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.203394    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-istio-envoy" pod "10885581-4aa8-4607-b81a-158066a7dd27" (UID: "10885581-4aa8-4607-b81a-158066a7dd27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:34.203497 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.203460    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66d450d8-b5a4-406f-a427-015307e5a9f6-kube-api-access-tqm7d" pod "66d450d8-b5a4-406f-a427-015307e5a9f6" (UID: "66d450d8-b5a4-406f-a427-015307e5a9f6"). InnerVolumeSpecName "kube-api-access-tqm7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:34.215683 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.215641    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-1c8130-openshift-default-dockercfg-drhsq\""
Oct 30 13:37:34.292684 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.292647    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ef74cc1-eb94-4550-af70-38d4f4149963-istiod-ca-cert\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:37:34.292684 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.292693    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9btb7\" (UniqueName: \"kubernetes.io/projected/7ef74cc1-eb94-4550-af70-38d4f4149963-kube-api-access-9btb7\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:37:34.292946 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.292726    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-workload-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:37:34.292995 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.292964    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ef74cc1-eb94-4550-af70-38d4f4149963-istio-token\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:37:34.293046 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.293011    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-istio-data\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:37:34.293234 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.293198    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ef74cc1-eb94-4550-af70-38d4f4149963-istio-podinfo\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:37:34.293338 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.293319    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8732b808-436c-4bf7-868e-1e934515e700-metrics-certs\") pod \"maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt\" (UID: \"8732b808-436c-4bf7-868e-1e934515e700\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt"
Oct 30 13:37:34.293399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.293382    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-credential-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:37:34.293455 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.293412    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-workload-certs\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:37:34.293508 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.293479    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-istio-envoy\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:37:34.293669 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.293642    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-547rh\" (UniqueName: \"kubernetes.io/projected/8732b808-436c-4bf7-868e-1e934515e700-kube-api-access-547rh\") pod \"maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt\" (UID: \"8732b808-436c-4bf7-868e-1e934515e700\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt"
Oct 30 13:37:34.293779 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.293761    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8732b808-436c-4bf7-868e-1e934515e700-maas-api-tls\") pod \"maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt\" (UID: \"8732b808-436c-4bf7-868e-1e934515e700\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt"
Oct 30 13:37:34.293922 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.293894    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8732b808-436c-4bf7-868e-1e934515e700-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt\" (UID: \"8732b808-436c-4bf7-868e-1e934515e700\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt"
Oct 30 13:37:34.294070 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.294054    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66d450d8-b5a4-406f-a427-015307e5a9f6-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.294244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.294076    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.294244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.294093    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10885581-4aa8-4607-b81a-158066a7dd27-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.294244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.294108    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66d450d8-b5a4-406f-a427-015307e5a9f6-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.294244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.294122    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdr7h\" (UniqueName: \"kubernetes.io/projected/10885581-4aa8-4607-b81a-158066a7dd27-kube-api-access-vdr7h\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.294244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.294138    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqm7d\" (UniqueName: \"kubernetes.io/projected/66d450d8-b5a4-406f-a427-015307e5a9f6-kube-api-access-tqm7d\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.294244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.294153    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.294244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.294165    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.294244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.294177    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.294244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.294191    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10885581-4aa8-4607-b81a-158066a7dd27-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.294244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.294204    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.294823 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.294270    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.294823 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.294288    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10885581-4aa8-4607-b81a-158066a7dd27-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.294823 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.294306    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.294823 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.294337    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10885581-4aa8-4607-b81a-158066a7dd27-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.294823 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.294349    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66d450d8-b5a4-406f-a427-015307e5a9f6-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.294823 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.294364    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66d450d8-b5a4-406f-a427-015307e5a9f6-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.393090 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.393040    4544 generic.go:356] "Generic (PLEG): container finished" podID="10885581-4aa8-4607-b81a-158066a7dd27" containerID="b1b6e8f55f2d229518c72cc3c354d25fbe7cf48ea872859b6dc54fecebbaf3ec" exitCode=0
Oct 30 13:37:34.393651 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.393174    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt" event={"ID":"10885581-4aa8-4607-b81a-158066a7dd27","Type":"ContainerDied","Data":"b1b6e8f55f2d229518c72cc3c354d25fbe7cf48ea872859b6dc54fecebbaf3ec"}
Oct 30 13:37:34.393651 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.393280    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt" event={"ID":"10885581-4aa8-4607-b81a-158066a7dd27","Type":"ContainerDied","Data":"25632e4e32d734b218454f88459d0b1cd6f98e0dacce0a0ee0b8d423c8872c0e"}
Oct 30 13:37:34.393651 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.393321    4544 scope.go:122] "RemoveContainer" containerID="b1b6e8f55f2d229518c72cc3c354d25fbe7cf48ea872859b6dc54fecebbaf3ec"
Oct 30 13:37:34.393651 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.393467    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"
Oct 30 13:37:34.396367 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.395975    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ef74cc1-eb94-4550-af70-38d4f4149963-istio-podinfo\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:37:34.396367 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.396053    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8732b808-436c-4bf7-868e-1e934515e700-metrics-certs\") pod \"maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt\" (UID: \"8732b808-436c-4bf7-868e-1e934515e700\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt"
Oct 30 13:37:34.396367 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.396103    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-credential-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:37:34.396367 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.396128    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-workload-certs\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:37:34.396367 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.396154    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-istio-envoy\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:37:34.396367 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.396323    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-547rh\" (UniqueName: \"kubernetes.io/projected/8732b808-436c-4bf7-868e-1e934515e700-kube-api-access-547rh\") pod \"maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt\" (UID: \"8732b808-436c-4bf7-868e-1e934515e700\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt"
Oct 30 13:37:34.400521 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.396424    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8732b808-436c-4bf7-868e-1e934515e700-maas-api-tls\") pod \"maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt\" (UID: \"8732b808-436c-4bf7-868e-1e934515e700\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt"
Oct 30 13:37:34.400521 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.396500    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8732b808-436c-4bf7-868e-1e934515e700-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt\" (UID: \"8732b808-436c-4bf7-868e-1e934515e700\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt"
Oct 30 13:37:34.400521 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.396552    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ef74cc1-eb94-4550-af70-38d4f4149963-istiod-ca-cert\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:37:34.400521 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.396579    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9btb7\" (UniqueName: \"kubernetes.io/projected/7ef74cc1-eb94-4550-af70-38d4f4149963-kube-api-access-9btb7\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:37:34.400521 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.396618    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-workload-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:37:34.400521 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.396727    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ef74cc1-eb94-4550-af70-38d4f4149963-istio-token\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:37:34.400521 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.396746    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-workload-certs\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:37:34.400521 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.396758    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-istio-data\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:37:34.400521 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.397004    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-istio-data\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:37:34.400521 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.397295    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-workload-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:37:34.400521 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.398202    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-credential-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:37:34.400521 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.398752    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ef74cc1-eb94-4550-af70-38d4f4149963-istiod-ca-cert\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:37:34.400521 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.399407    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8732b808-436c-4bf7-868e-1e934515e700-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt\" (UID: \"8732b808-436c-4bf7-868e-1e934515e700\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt"
Oct 30 13:37:34.403186 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.402830    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5" event={"ID":"01e24b49-04cb-4af7-a9cc-a0b6fa608451","Type":"ContainerStarted","Data":"f585dde75e1459c7410e3db22f94a388bda2c60845b3f09cbc2cb49fba4ec22b"}
Oct 30 13:37:34.403186 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.402913    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5" event={"ID":"01e24b49-04cb-4af7-a9cc-a0b6fa608451","Type":"ContainerStarted","Data":"8dae0828007976cddd46d7e847ffc1bf85ca40e83eabdfff99d1de17d343dcef"}
Oct 30 13:37:34.403532 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.403427    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8732b808-436c-4bf7-868e-1e934515e700-metrics-certs\") pod \"maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt\" (UID: \"8732b808-436c-4bf7-868e-1e934515e700\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt"
Oct 30 13:37:34.404099 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.404002    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ef74cc1-eb94-4550-af70-38d4f4149963-istio-podinfo\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:37:34.407960 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.405210    4544 generic.go:356] "Generic (PLEG): container finished" podID="66d450d8-b5a4-406f-a427-015307e5a9f6" containerID="a5d7aabf9d2045725d76d8bf5c5bea75194491daa70b774fbc0d32e810bfb8fe" exitCode=0
Oct 30 13:37:34.407960 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.405296    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"
Oct 30 13:37:34.407960 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.405350    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj" event={"ID":"66d450d8-b5a4-406f-a427-015307e5a9f6","Type":"ContainerDied","Data":"a5d7aabf9d2045725d76d8bf5c5bea75194491daa70b774fbc0d32e810bfb8fe"}
Oct 30 13:37:34.407960 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.405413    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj" event={"ID":"66d450d8-b5a4-406f-a427-015307e5a9f6","Type":"ContainerDied","Data":"35c90626575ec46d7fe770062f403f36264db7e0722af6033c128381531cfea4"}
Oct 30 13:37:34.407960 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.405650    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8732b808-436c-4bf7-868e-1e934515e700-maas-api-tls\") pod \"maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt\" (UID: \"8732b808-436c-4bf7-868e-1e934515e700\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt"
Oct 30 13:37:34.407960 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.405951    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-istio-envoy\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:37:34.409544 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.408337    4544 scope.go:122] "RemoveContainer" containerID="4c0b0f15d3a23f14b42ae835200a1924674449a5d5e370e3277514886209cb08"
Oct 30 13:37:34.409544 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:37:34.408639    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl_odh-ai-gateway-infra(eb8eebdc-5933-463b-a511-264945e917e9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl" podUID="eb8eebdc-5933-463b-a511-264945e917e9"
Oct 30 13:37:34.417719 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.417696    4544 scope.go:122] "RemoveContainer" containerID="b1b6e8f55f2d229518c72cc3c354d25fbe7cf48ea872859b6dc54fecebbaf3ec"
Oct 30 13:37:34.418038 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:37:34.418017    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1b6e8f55f2d229518c72cc3c354d25fbe7cf48ea872859b6dc54fecebbaf3ec\": container with ID starting with b1b6e8f55f2d229518c72cc3c354d25fbe7cf48ea872859b6dc54fecebbaf3ec not found: ID does not exist" containerID="b1b6e8f55f2d229518c72cc3c354d25fbe7cf48ea872859b6dc54fecebbaf3ec"
Oct 30 13:37:34.418118 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.418051    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1b6e8f55f2d229518c72cc3c354d25fbe7cf48ea872859b6dc54fecebbaf3ec"} err="failed to get container status \"b1b6e8f55f2d229518c72cc3c354d25fbe7cf48ea872859b6dc54fecebbaf3ec\": rpc error: code = NotFound desc = could not find container \"b1b6e8f55f2d229518c72cc3c354d25fbe7cf48ea872859b6dc54fecebbaf3ec\": container with ID starting with b1b6e8f55f2d229518c72cc3c354d25fbe7cf48ea872859b6dc54fecebbaf3ec not found: ID does not exist"
Oct 30 13:37:34.418118 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.418075    4544 scope.go:122] "RemoveContainer" containerID="a5d7aabf9d2045725d76d8bf5c5bea75194491daa70b774fbc0d32e810bfb8fe"
Oct 30 13:37:34.420956 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.420937    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ef74cc1-eb94-4550-af70-38d4f4149963-istio-token\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:37:34.421030 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.421014    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9btb7\" (UniqueName: \"kubernetes.io/projected/7ef74cc1-eb94-4550-af70-38d4f4149963-kube-api-access-9btb7\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:37:34.421275 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.421248    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-547rh\" (UniqueName: \"kubernetes.io/projected/8732b808-436c-4bf7-868e-1e934515e700-kube-api-access-547rh\") pod \"maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt\" (UID: \"8732b808-436c-4bf7-868e-1e934515e700\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt"
Oct 30 13:37:34.430209 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.429957    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt"
Oct 30 13:37:34.432539 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.432513    4544 scope.go:122] "RemoveContainer" containerID="a5d7aabf9d2045725d76d8bf5c5bea75194491daa70b774fbc0d32e810bfb8fe"
Oct 30 13:37:34.432889 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:37:34.432861    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5d7aabf9d2045725d76d8bf5c5bea75194491daa70b774fbc0d32e810bfb8fe\": container with ID starting with a5d7aabf9d2045725d76d8bf5c5bea75194491daa70b774fbc0d32e810bfb8fe not found: ID does not exist" containerID="a5d7aabf9d2045725d76d8bf5c5bea75194491daa70b774fbc0d32e810bfb8fe"
Oct 30 13:37:34.432961 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.432900    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5d7aabf9d2045725d76d8bf5c5bea75194491daa70b774fbc0d32e810bfb8fe"} err="failed to get container status \"a5d7aabf9d2045725d76d8bf5c5bea75194491daa70b774fbc0d32e810bfb8fe\": rpc error: code = NotFound desc = could not find container \"a5d7aabf9d2045725d76d8bf5c5bea75194491daa70b774fbc0d32e810bfb8fe\": container with ID starting with a5d7aabf9d2045725d76d8bf5c5bea75194491daa70b774fbc0d32e810bfb8fe not found: ID does not exist"
Oct 30 13:37:34.458527 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.458466    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5" podStartSLOduration=2.458445493 podStartE2EDuration="2.458445493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:34.453070225 +0000 UTC m=+3149.500204179" watchObservedRunningTime="2026-10-30 13:37:34.458445493 +0000 UTC m=+3149.505579448"
Oct 30 13:37:34.500679 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.499946    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:37:34.522314 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.522284    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"]
Oct 30 13:37:34.531545 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.531509    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt"]
Oct 30 13:37:34.591580 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.591545    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"]
Oct 30 13:37:34.599951 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.599754    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj"]
Oct 30 13:37:34.762954 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.762923    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt"]
Oct 30 13:37:34.799489 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:37:34.799455    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8732b808_436c_4bf7_868e_1e934515e700.slice/crio-b9b6f622c1d93e57686ad6841c3c5a46ca258b8bec5a2cc8c4f486208f9f1eb6 WatchSource:0}: Error finding container b9b6f622c1d93e57686ad6841c3c5a46ca258b8bec5a2cc8c4f486208f9f1eb6: Status 404 returned error can't find the container with id b9b6f622c1d93e57686ad6841c3c5a46ca258b8bec5a2cc8c4f486208f9f1eb6
Oct 30 13:37:34.846723 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.846696    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"]
Oct 30 13:37:34.848547 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:37:34.848517    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef74cc1_eb94_4550_af70_38d4f4149963.slice/crio-99014900ff6fbb3789befd6933876bb2809fe8041c9616b892974d36ef82b323 WatchSource:0}: Error finding container 99014900ff6fbb3789befd6933876bb2809fe8041c9616b892974d36ef82b323: Status 404 returned error can't find the container with id 99014900ff6fbb3789befd6933876bb2809fe8041c9616b892974d36ef82b323
Oct 30 13:37:34.851796 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.851762    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:37:34.851888 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.851834    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:37:34.851931 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:34.851885    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:37:35.092252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:35.092199    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10885581-4aa8-4607-b81a-158066a7dd27" path="/var/lib/kubelet/pods/10885581-4aa8-4607-b81a-158066a7dd27/volumes"
Oct 30 13:37:35.093056 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:35.093035    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66d450d8-b5a4-406f-a427-015307e5a9f6" path="/var/lib/kubelet/pods/66d450d8-b5a4-406f-a427-015307e5a9f6/volumes"
Oct 30 13:37:35.266121 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:35.266071    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:35.422358 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:35.422315    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r" event={"ID":"7ef74cc1-eb94-4550-af70-38d4f4149963","Type":"ContainerStarted","Data":"2fab71d50be57d2748266648f6d21cd9f1d1639dd149f9083673f7216b8353b4"}
Oct 30 13:37:35.422781 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:35.422384    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r" event={"ID":"7ef74cc1-eb94-4550-af70-38d4f4149963","Type":"ContainerStarted","Data":"99014900ff6fbb3789befd6933876bb2809fe8041c9616b892974d36ef82b323"}
Oct 30 13:37:35.424472 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:35.424438    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt" event={"ID":"8732b808-436c-4bf7-868e-1e934515e700","Type":"ContainerStarted","Data":"01d0ea9d4c967ad51d1bb50e9a9ece0c8450b987ff26030d476e0360fe9e8455"}
Oct 30 13:37:35.424614 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:35.424495    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt" event={"ID":"8732b808-436c-4bf7-868e-1e934515e700","Type":"ContainerStarted","Data":"b9b6f622c1d93e57686ad6841c3c5a46ca258b8bec5a2cc8c4f486208f9f1eb6"}
Oct 30 13:37:35.424614 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:35.424516    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt"
Oct 30 13:37:35.427475 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:35.427450    4544 generic.go:356] "Generic (PLEG): container finished" podID="01e24b49-04cb-4af7-a9cc-a0b6fa608451" containerID="f585dde75e1459c7410e3db22f94a388bda2c60845b3f09cbc2cb49fba4ec22b" exitCode=7
Oct 30 13:37:35.427583 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:35.427507    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5" event={"ID":"01e24b49-04cb-4af7-a9cc-a0b6fa608451","Type":"ContainerDied","Data":"f585dde75e1459c7410e3db22f94a388bda2c60845b3f09cbc2cb49fba4ec22b"}
Oct 30 13:37:35.427770 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:35.427758    4544 scope.go:122] "RemoveContainer" containerID="f585dde75e1459c7410e3db22f94a388bda2c60845b3f09cbc2cb49fba4ec22b"
Oct 30 13:37:35.474427 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:35.474373    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r" podStartSLOduration=170.474357694 podStartE2EDuration="2m50.474357694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:35.467983028 +0000 UTC m=+3150.515116982" watchObservedRunningTime="2026-10-30 13:37:35.474357694 +0000 UTC m=+3150.521491702"
Oct 30 13:37:35.500901 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:35.500857    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:37:35.507284 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:35.506346    4544 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" start-of-body=
Oct 30 13:37:35.507284 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:35.506411    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r" podUID="7ef74cc1-eb94-4550-af70-38d4f4149963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused"
Oct 30 13:37:35.584213 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:35.584142    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt" podStartSLOduration=172.256828624 podStartE2EDuration="2m52.584119068s" totalImagesPullingTime="327.290444ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:35.56140005 +0000 UTC m=+3150.608533997" watchObservedRunningTime="2026-10-30 13:37:35.584119068 +0000 UTC m=+3150.631253025"
Oct 30 13:37:36.449599 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:36.449559    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5" event={"ID":"01e24b49-04cb-4af7-a9cc-a0b6fa608451","Type":"ContainerStarted","Data":"fa495ef862e2ccacf50595f14e569e40bfd4f33545d071914e3b5f122fe7ab70"}
Oct 30 13:37:36.553242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:36.549624    4544 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" start-of-body=
Oct 30 13:37:36.553242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:36.549688    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r" podUID="7ef74cc1-eb94-4550-af70-38d4f4149963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused"
Oct 30 13:37:37.201978 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:37.201939    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76405c-zt5nd"]
Oct 30 13:37:37.207353 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:37.207327    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76405c-zt5nd"
Oct 30 13:37:37.245916 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:37.245878    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76405c-zt5nd"]
Oct 30 13:37:37.338944 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:37.338902    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:37.365989 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:37.365841    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77wnl\" (UniqueName: \"kubernetes.io/projected/5490d4aa-6c4d-486f-b953-05331cc36305-kube-api-access-77wnl\") pod \"maas-api-revoke-keys-e2e-ipp-a-76405c-zt5nd\" (UID: \"5490d4aa-6c4d-486f-b953-05331cc36305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76405c-zt5nd"
Oct 30 13:37:37.366512 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:37.366481    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5490d4aa-6c4d-486f-b953-05331cc36305-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-76405c-zt5nd\" (UID: \"5490d4aa-6c4d-486f-b953-05331cc36305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76405c-zt5nd"
Oct 30 13:37:37.387357 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:37.387299    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5490d4aa-6c4d-486f-b953-05331cc36305-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-76405c-zt5nd\" (UID: \"5490d4aa-6c4d-486f-b953-05331cc36305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76405c-zt5nd"
Oct 30 13:37:37.463898 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:37.463809    4544 generic.go:356] "Generic (PLEG): container finished" podID="01e24b49-04cb-4af7-a9cc-a0b6fa608451" containerID="fa495ef862e2ccacf50595f14e569e40bfd4f33545d071914e3b5f122fe7ab70" exitCode=7
Oct 30 13:37:37.464671 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:37.464634    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5" event={"ID":"01e24b49-04cb-4af7-a9cc-a0b6fa608451","Type":"ContainerDied","Data":"fa495ef862e2ccacf50595f14e569e40bfd4f33545d071914e3b5f122fe7ab70"}
Oct 30 13:37:37.464831 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:37.464717    4544 scope.go:122] "RemoveContainer" containerID="f585dde75e1459c7410e3db22f94a388bda2c60845b3f09cbc2cb49fba4ec22b"
Oct 30 13:37:37.466151 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:37.465712    4544 scope.go:122] "RemoveContainer" containerID="fa495ef862e2ccacf50595f14e569e40bfd4f33545d071914e3b5f122fe7ab70"
Oct 30 13:37:37.466151 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:37:37.466066    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5_odh-ai-gateway-infra(01e24b49-04cb-4af7-a9cc-a0b6fa608451)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5" podUID="01e24b49-04cb-4af7-a9cc-a0b6fa608451"
Oct 30 13:37:37.489137 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:37.489104    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5490d4aa-6c4d-486f-b953-05331cc36305-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-76405c-zt5nd\" (UID: \"5490d4aa-6c4d-486f-b953-05331cc36305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76405c-zt5nd"
Oct 30 13:37:37.489324 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:37.489212    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5490d4aa-6c4d-486f-b953-05331cc36305-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-76405c-zt5nd\" (UID: \"5490d4aa-6c4d-486f-b953-05331cc36305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76405c-zt5nd"
Oct 30 13:37:37.489530 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:37.489509    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77wnl\" (UniqueName: \"kubernetes.io/projected/5490d4aa-6c4d-486f-b953-05331cc36305-kube-api-access-77wnl\") pod \"maas-api-revoke-keys-e2e-ipp-a-76405c-zt5nd\" (UID: \"5490d4aa-6c4d-486f-b953-05331cc36305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76405c-zt5nd"
Oct 30 13:37:37.490140 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:37.490118    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5490d4aa-6c4d-486f-b953-05331cc36305-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-76405c-zt5nd\" (UID: \"5490d4aa-6c4d-486f-b953-05331cc36305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76405c-zt5nd"
Oct 30 13:37:37.491732 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:37.491707    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5490d4aa-6c4d-486f-b953-05331cc36305-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-76405c-zt5nd\" (UID: \"5490d4aa-6c4d-486f-b953-05331cc36305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76405c-zt5nd"
Oct 30 13:37:37.501471 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:37.501443    4544 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" start-of-body=
Oct 30 13:37:37.501589 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:37.501502    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r" podUID="7ef74cc1-eb94-4550-af70-38d4f4149963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused"
Oct 30 13:37:37.540048 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:37.540010    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77wnl\" (UniqueName: \"kubernetes.io/projected/5490d4aa-6c4d-486f-b953-05331cc36305-kube-api-access-77wnl\") pod \"maas-api-revoke-keys-e2e-ipp-a-76405c-zt5nd\" (UID: \"5490d4aa-6c4d-486f-b953-05331cc36305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76405c-zt5nd"
Oct 30 13:37:37.548893 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:37.548862    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76405c-zt5nd"
Oct 30 13:37:37.890041 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:37.890007    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76405c-zt5nd"]
Oct 30 13:37:37.891124 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:37:37.891095    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5490d4aa_6c4d_486f_b953_05331cc36305.slice/crio-5ec8c05ad0471bf13cbe7e408cac1b70b73f180d00fe7035e116819f3bc74db5 WatchSource:0}: Error finding container 5ec8c05ad0471bf13cbe7e408cac1b70b73f180d00fe7035e116819f3bc74db5: Status 404 returned error can't find the container with id 5ec8c05ad0471bf13cbe7e408cac1b70b73f180d00fe7035e116819f3bc74db5
Oct 30 13:37:38.276821 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:38.276787    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:38.433318 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:38.428772    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:38.474834 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:38.474804    4544 scope.go:122] "RemoveContainer" containerID="fa495ef862e2ccacf50595f14e569e40bfd4f33545d071914e3b5f122fe7ab70"
Oct 30 13:37:38.475320 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:37:38.475258    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5_odh-ai-gateway-infra(01e24b49-04cb-4af7-a9cc-a0b6fa608451)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5" podUID="01e24b49-04cb-4af7-a9cc-a0b6fa608451"
Oct 30 13:37:38.475829 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:38.475804    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76405c-zt5nd" event={"ID":"5490d4aa-6c4d-486f-b953-05331cc36305","Type":"ContainerStarted","Data":"5ec8c05ad0471bf13cbe7e408cac1b70b73f180d00fe7035e116819f3bc74db5"}
Oct 30 13:37:38.501405 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:38.501365    4544 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" start-of-body=
Oct 30 13:37:38.501564 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:38.501416    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r" podUID="7ef74cc1-eb94-4550-af70-38d4f4149963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused"
Oct 30 13:37:38.649642 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:38.649539    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:38.660245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:38.659382    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"]
Oct 30 13:37:38.660245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:38.659695    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2" podUID="5a60c9a2-4e23-4e4e-be07-09609b79f023" containerName="istio-proxy" containerID="cri-o://2773c3435f6f1ba8746bfb50ec5af3da37ee191eecb9e36e167615f6e1e96ae5" gracePeriod=30
Oct 30 13:37:38.672472 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:38.672429    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"]
Oct 30 13:37:38.672908 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:38.672863    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp" podUID="f90bdf7e-ba66-4f37-9585-5173e85566fb" containerName="istio-proxy" containerID="cri-o://29bc0e6821304515e028dab5ad10551f3b28ec78e428dd737391a7098f100555" gracePeriod=30
Oct 30 13:37:38.694467 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:38.694365    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"]
Oct 30 13:37:38.694693 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:38.694663    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g" podUID="8b6f4e05-5105-4596-b248-bc018f59ce81" containerName="istio-proxy" containerID="cri-o://b80f1275112c4ee5ff483cf8352a3d6cd269f111fde61e65075c0648ed69b0a5" gracePeriod=30
Oct 30 13:37:39.163406 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:39.161141    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:39.354825 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:39.354789    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-09db16-kmnhr"]
Oct 30 13:37:39.360713 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:39.360685    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-09db16-kmnhr"
Oct 30 13:37:39.385337 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:39.380714    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-09db16-kmnhr"]
Oct 30 13:37:39.769305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:39.767207    4544 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" start-of-body=
Oct 30 13:37:39.769305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:39.767317    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r" podUID="7ef74cc1-eb94-4550-af70-38d4f4149963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused"
Oct 30 13:37:39.792117 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:39.792080    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94261b46-b158-4895-bfd1-21365792a055-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-09db16-kmnhr\" (UID: \"94261b46-b158-4895-bfd1-21365792a055\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-09db16-kmnhr"
Oct 30 13:37:39.792492 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:39.792463    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94261b46-b158-4895-bfd1-21365792a055-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-09db16-kmnhr\" (UID: \"94261b46-b158-4895-bfd1-21365792a055\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-09db16-kmnhr"
Oct 30 13:37:39.793066 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:39.793033    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5g8m\" (UniqueName: \"kubernetes.io/projected/94261b46-b158-4895-bfd1-21365792a055-kube-api-access-g5g8m\") pod \"maas-api-revoke-keys-e2e-worker-w3-09db16-kmnhr\" (UID: \"94261b46-b158-4895-bfd1-21365792a055\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-09db16-kmnhr"
Oct 30 13:37:39.806661 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:39.806627    4544 generic.go:356] "Generic (PLEG): container finished" podID="5490d4aa-6c4d-486f-b953-05331cc36305" containerID="4feccb12d9e729a8befd620d4094d93676e9c29b6ec3b5b21ca1b76b59d6ea64" exitCode=0
Oct 30 13:37:39.806958 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:39.806934    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76405c-zt5nd" event={"ID":"5490d4aa-6c4d-486f-b953-05331cc36305","Type":"ContainerDied","Data":"4feccb12d9e729a8befd620d4094d93676e9c29b6ec3b5b21ca1b76b59d6ea64"}
Oct 30 13:37:39.894804 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:39.894762    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5g8m\" (UniqueName: \"kubernetes.io/projected/94261b46-b158-4895-bfd1-21365792a055-kube-api-access-g5g8m\") pod \"maas-api-revoke-keys-e2e-worker-w3-09db16-kmnhr\" (UID: \"94261b46-b158-4895-bfd1-21365792a055\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-09db16-kmnhr"
Oct 30 13:37:39.894975 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:39.894920    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94261b46-b158-4895-bfd1-21365792a055-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-09db16-kmnhr\" (UID: \"94261b46-b158-4895-bfd1-21365792a055\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-09db16-kmnhr"
Oct 30 13:37:39.895063 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:39.895025    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94261b46-b158-4895-bfd1-21365792a055-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-09db16-kmnhr\" (UID: \"94261b46-b158-4895-bfd1-21365792a055\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-09db16-kmnhr"
Oct 30 13:37:39.895912 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:39.895883    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94261b46-b158-4895-bfd1-21365792a055-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-09db16-kmnhr\" (UID: \"94261b46-b158-4895-bfd1-21365792a055\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-09db16-kmnhr"
Oct 30 13:37:39.897599 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:39.897567    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94261b46-b158-4895-bfd1-21365792a055-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-09db16-kmnhr\" (UID: \"94261b46-b158-4895-bfd1-21365792a055\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-09db16-kmnhr"
Oct 30 13:37:39.955154 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:39.954270    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5g8m\" (UniqueName: \"kubernetes.io/projected/94261b46-b158-4895-bfd1-21365792a055-kube-api-access-g5g8m\") pod \"maas-api-revoke-keys-e2e-worker-w3-09db16-kmnhr\" (UID: \"94261b46-b158-4895-bfd1-21365792a055\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-09db16-kmnhr"
Oct 30 13:37:40.039804 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:40.039709    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:40.072955 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:40.072921    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-09db16-kmnhr"
Oct 30 13:37:40.501570 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:40.501535    4544 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" start-of-body=
Oct 30 13:37:40.501716 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:40.501577    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-09db16-kmnhr"]
Oct 30 13:37:40.501716 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:40.501602    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r" podUID="7ef74cc1-eb94-4550-af70-38d4f4149963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused"
Oct 30 13:37:40.501842 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:37:40.501780    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94261b46_b158_4895_bfd1_21365792a055.slice/crio-1d590465132a21fadc621cd34e6923f99a2655d0298e916352abcff85cc70571 WatchSource:0}: Error finding container 1d590465132a21fadc621cd34e6923f99a2655d0298e916352abcff85cc70571: Status 404 returned error can't find the container with id 1d590465132a21fadc621cd34e6923f99a2655d0298e916352abcff85cc70571
Oct 30 13:37:40.843588 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:40.843544    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-09db16-kmnhr" event={"ID":"94261b46-b158-4895-bfd1-21365792a055","Type":"ContainerStarted","Data":"1d590465132a21fadc621cd34e6923f99a2655d0298e916352abcff85cc70571"}
Oct 30 13:37:41.098215 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:41.098184    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76405c-zt5nd"
Oct 30 13:37:41.178123 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:41.175916    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5490d4aa-6c4d-486f-b953-05331cc36305-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5490d4aa-6c4d-486f-b953-05331cc36305-maas-api-xks-ca\") pod \"5490d4aa-6c4d-486f-b953-05331cc36305\" (UID: \"5490d4aa-6c4d-486f-b953-05331cc36305\") "
Oct 30 13:37:41.178123 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:41.176000    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5490d4aa-6c4d-486f-b953-05331cc36305-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5490d4aa-6c4d-486f-b953-05331cc36305-maas-api-openshift-ca\") pod \"5490d4aa-6c4d-486f-b953-05331cc36305\" (UID: \"5490d4aa-6c4d-486f-b953-05331cc36305\") "
Oct 30 13:37:41.178123 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:41.176170    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5490d4aa-6c4d-486f-b953-05331cc36305-kube-api-access-77wnl\" (UniqueName: \"kubernetes.io/projected/5490d4aa-6c4d-486f-b953-05331cc36305-kube-api-access-77wnl\") pod \"5490d4aa-6c4d-486f-b953-05331cc36305\" (UID: \"5490d4aa-6c4d-486f-b953-05331cc36305\") "
Oct 30 13:37:41.178123 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:41.176447    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5490d4aa-6c4d-486f-b953-05331cc36305-maas-api-openshift-ca" pod "5490d4aa-6c4d-486f-b953-05331cc36305" (UID: "5490d4aa-6c4d-486f-b953-05331cc36305"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:37:41.178123 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:41.177373    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5490d4aa-6c4d-486f-b953-05331cc36305-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:41.178772 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:41.178740    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5490d4aa-6c4d-486f-b953-05331cc36305-maas-api-xks-ca" pod "5490d4aa-6c4d-486f-b953-05331cc36305" (UID: "5490d4aa-6c4d-486f-b953-05331cc36305"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:37:41.178919 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:41.178901    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5490d4aa-6c4d-486f-b953-05331cc36305-kube-api-access-77wnl" pod "5490d4aa-6c4d-486f-b953-05331cc36305" (UID: "5490d4aa-6c4d-486f-b953-05331cc36305"). InnerVolumeSpecName "kube-api-access-77wnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:41.278685 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:41.278645    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5490d4aa-6c4d-486f-b953-05331cc36305-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:41.278685 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:41.278679    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77wnl\" (UniqueName: \"kubernetes.io/projected/5490d4aa-6c4d-486f-b953-05331cc36305-kube-api-access-77wnl\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:41.457083 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:41.457050    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt"
Oct 30 13:37:41.500644 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:41.500607    4544 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" start-of-body=
Oct 30 13:37:41.500804 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:41.500663    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r" podUID="7ef74cc1-eb94-4550-af70-38d4f4149963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused"
Oct 30 13:37:41.855089 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:41.855055    4544 generic.go:356] "Generic (PLEG): container finished" podID="94261b46-b158-4895-bfd1-21365792a055" containerID="c3dd8a4783604edbf2573dabe8d9519cdefd2c7c38036800dd3490d7cb7e120b" exitCode=0
Oct 30 13:37:41.855608 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:41.855145    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-09db16-kmnhr" event={"ID":"94261b46-b158-4895-bfd1-21365792a055","Type":"ContainerDied","Data":"c3dd8a4783604edbf2573dabe8d9519cdefd2c7c38036800dd3490d7cb7e120b"}
Oct 30 13:37:41.856954 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:41.856933    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76405c-zt5nd"
Oct 30 13:37:41.857091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:41.856965    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76405c-zt5nd" event={"ID":"5490d4aa-6c4d-486f-b953-05331cc36305","Type":"ContainerDied","Data":"5ec8c05ad0471bf13cbe7e408cac1b70b73f180d00fe7035e116819f3bc74db5"}
Oct 30 13:37:41.857091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:41.857003    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ec8c05ad0471bf13cbe7e408cac1b70b73f180d00fe7035e116819f3bc74db5"
Oct 30 13:37:42.153775 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:42.153737    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:42.507317 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:42.507202    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:37:42.507610 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:42.507584    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:37:42.508672 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:42.508649    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:37:43.068777 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.068733    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4f385-t528x"]
Oct 30 13:37:43.069908 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.069885    4544 cpu_manager.go:414] "containerMap: removing container" podUID="5490d4aa-6c4d-486f-b953-05331cc36305" containerName="revoke-keys"
Oct 30 13:37:43.069908 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.069909    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5490d4aa-6c4d-486f-b953-05331cc36305" containerName="revoke-keys"
Oct 30 13:37:43.070079 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.070069    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="5490d4aa-6c4d-486f-b953-05331cc36305" containerName="revoke-keys"
Oct 30 13:37:43.074657 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.074614    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4f385-t528x"
Oct 30 13:37:43.083696 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.083666    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-09db16-kmnhr"
Oct 30 13:37:43.103407 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.103370    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4f385-t528x"]
Oct 30 13:37:43.131442 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.130129    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94261b46-b158-4895-bfd1-21365792a055-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94261b46-b158-4895-bfd1-21365792a055-maas-api-xks-ca\") pod \"94261b46-b158-4895-bfd1-21365792a055\" (UID: \"94261b46-b158-4895-bfd1-21365792a055\") "
Oct 30 13:37:43.131442 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.130315    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94261b46-b158-4895-bfd1-21365792a055-kube-api-access-g5g8m\" (UniqueName: \"kubernetes.io/projected/94261b46-b158-4895-bfd1-21365792a055-kube-api-access-g5g8m\") pod \"94261b46-b158-4895-bfd1-21365792a055\" (UID: \"94261b46-b158-4895-bfd1-21365792a055\") "
Oct 30 13:37:43.131442 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.130375    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94261b46-b158-4895-bfd1-21365792a055-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94261b46-b158-4895-bfd1-21365792a055-maas-api-openshift-ca\") pod \"94261b46-b158-4895-bfd1-21365792a055\" (UID: \"94261b46-b158-4895-bfd1-21365792a055\") "
Oct 30 13:37:43.131442 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.130787    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxgx8\" (UniqueName: \"kubernetes.io/projected/ee5e270c-f424-4803-992f-a49fbc67c6dd-kube-api-access-qxgx8\") pod \"maas-api-revoke-keys-e2e-ipp-a-a4f385-t528x\" (UID: \"ee5e270c-f424-4803-992f-a49fbc67c6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4f385-t528x"
Oct 30 13:37:43.131442 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.131092    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5e270c-f424-4803-992f-a49fbc67c6dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a4f385-t528x\" (UID: \"ee5e270c-f424-4803-992f-a49fbc67c6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4f385-t528x"
Oct 30 13:37:43.131442 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.131249    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5e270c-f424-4803-992f-a49fbc67c6dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a4f385-t528x\" (UID: \"ee5e270c-f424-4803-992f-a49fbc67c6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4f385-t528x"
Oct 30 13:37:43.134271 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.132818    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94261b46-b158-4895-bfd1-21365792a055-maas-api-openshift-ca" pod "94261b46-b158-4895-bfd1-21365792a055" (UID: "94261b46-b158-4895-bfd1-21365792a055"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:37:43.139834 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.139232    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94261b46-b158-4895-bfd1-21365792a055-maas-api-xks-ca" pod "94261b46-b158-4895-bfd1-21365792a055" (UID: "94261b46-b158-4895-bfd1-21365792a055"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:37:43.141134 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.141100    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94261b46-b158-4895-bfd1-21365792a055-kube-api-access-g5g8m" pod "94261b46-b158-4895-bfd1-21365792a055" (UID: "94261b46-b158-4895-bfd1-21365792a055"). InnerVolumeSpecName "kube-api-access-g5g8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:43.233555 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.233512    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5e270c-f424-4803-992f-a49fbc67c6dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a4f385-t528x\" (UID: \"ee5e270c-f424-4803-992f-a49fbc67c6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4f385-t528x"
Oct 30 13:37:43.233812 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.233792    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxgx8\" (UniqueName: \"kubernetes.io/projected/ee5e270c-f424-4803-992f-a49fbc67c6dd-kube-api-access-qxgx8\") pod \"maas-api-revoke-keys-e2e-ipp-a-a4f385-t528x\" (UID: \"ee5e270c-f424-4803-992f-a49fbc67c6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4f385-t528x"
Oct 30 13:37:43.233979 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.233958    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5e270c-f424-4803-992f-a49fbc67c6dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a4f385-t528x\" (UID: \"ee5e270c-f424-4803-992f-a49fbc67c6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4f385-t528x"
Oct 30 13:37:43.234125 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.234068    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94261b46-b158-4895-bfd1-21365792a055-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:43.234125 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.234082    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5g8m\" (UniqueName: \"kubernetes.io/projected/94261b46-b158-4895-bfd1-21365792a055-kube-api-access-g5g8m\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:43.234125 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.234092    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94261b46-b158-4895-bfd1-21365792a055-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:43.234646 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.234619    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5e270c-f424-4803-992f-a49fbc67c6dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a4f385-t528x\" (UID: \"ee5e270c-f424-4803-992f-a49fbc67c6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4f385-t528x"
Oct 30 13:37:43.236192 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.236170    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5e270c-f424-4803-992f-a49fbc67c6dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a4f385-t528x\" (UID: \"ee5e270c-f424-4803-992f-a49fbc67c6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4f385-t528x"
Oct 30 13:37:43.257741 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.257696    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxgx8\" (UniqueName: \"kubernetes.io/projected/ee5e270c-f424-4803-992f-a49fbc67c6dd-kube-api-access-qxgx8\") pod \"maas-api-revoke-keys-e2e-ipp-a-a4f385-t528x\" (UID: \"ee5e270c-f424-4803-992f-a49fbc67c6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4f385-t528x"
Oct 30 13:37:43.397030 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.396941    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4f385-t528x"
Oct 30 13:37:43.749174 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.749132    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4f385-t528x"]
Oct 30 13:37:43.886302 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.886077    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4f385-t528x" event={"ID":"ee5e270c-f424-4803-992f-a49fbc67c6dd","Type":"ContainerStarted","Data":"1e00236fa32083a42414cc7b663534949f7dc133713c215160f0cbe5ac34222e"}
Oct 30 13:37:43.888136 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.888102    4544 generic.go:356] "Generic (PLEG): container finished" podID="5a60c9a2-4e23-4e4e-be07-09609b79f023" containerID="2773c3435f6f1ba8746bfb50ec5af3da37ee191eecb9e36e167615f6e1e96ae5" exitCode=0
Oct 30 13:37:43.888339 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.888173    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2" event={"ID":"5a60c9a2-4e23-4e4e-be07-09609b79f023","Type":"ContainerDied","Data":"2773c3435f6f1ba8746bfb50ec5af3da37ee191eecb9e36e167615f6e1e96ae5"}
Oct 30 13:37:43.890047 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.890019    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-09db16-kmnhr" event={"ID":"94261b46-b158-4895-bfd1-21365792a055","Type":"ContainerDied","Data":"1d590465132a21fadc621cd34e6923f99a2655d0298e916352abcff85cc70571"}
Oct 30 13:37:43.890185 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.890071    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d590465132a21fadc621cd34e6923f99a2655d0298e916352abcff85cc70571"
Oct 30 13:37:43.890185 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.890150    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-09db16-kmnhr"
Oct 30 13:37:43.893617 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.893586    4544 generic.go:356] "Generic (PLEG): container finished" podID="8b6f4e05-5105-4596-b248-bc018f59ce81" containerID="b80f1275112c4ee5ff483cf8352a3d6cd269f111fde61e65075c0648ed69b0a5" exitCode=0
Oct 30 13:37:43.893810 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.893685    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g" event={"ID":"8b6f4e05-5105-4596-b248-bc018f59ce81","Type":"ContainerDied","Data":"b80f1275112c4ee5ff483cf8352a3d6cd269f111fde61e65075c0648ed69b0a5"}
Oct 30 13:37:43.895810 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.895784    4544 generic.go:356] "Generic (PLEG): container finished" podID="f90bdf7e-ba66-4f37-9585-5173e85566fb" containerID="29bc0e6821304515e028dab5ad10551f3b28ec78e428dd737391a7098f100555" exitCode=0
Oct 30 13:37:43.896102 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:43.896079    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp" event={"ID":"f90bdf7e-ba66-4f37-9585-5173e85566fb","Type":"ContainerDied","Data":"29bc0e6821304515e028dab5ad10551f3b28ec78e428dd737391a7098f100555"}
Oct 30 13:37:44.245785 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.242733    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"]
Oct 30 13:37:44.309495 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.296969    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:37:44.389310 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.386417    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:44.477312 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.477272    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a60c9a2-4e23-4e4e-be07-09609b79f023-istio-token\" (UniqueName: \"kubernetes.io/projected/5a60c9a2-4e23-4e4e-be07-09609b79f023-istio-token\") pod \"5a60c9a2-4e23-4e4e-be07-09609b79f023\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") "
Oct 30 13:37:44.477509 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.477490    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a60c9a2-4e23-4e4e-be07-09609b79f023-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a60c9a2-4e23-4e4e-be07-09609b79f023-istiod-ca-cert\") pod \"5a60c9a2-4e23-4e4e-be07-09609b79f023\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") "
Oct 30 13:37:44.477590 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.477572    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a60c9a2-4e23-4e4e-be07-09609b79f023-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a60c9a2-4e23-4e4e-be07-09609b79f023-istio-podinfo\") pod \"5a60c9a2-4e23-4e4e-be07-09609b79f023\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") "
Oct 30 13:37:44.477664 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.477649    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-credential-socket\") pod \"5a60c9a2-4e23-4e4e-be07-09609b79f023\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") "
Oct 30 13:37:44.477735 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.477703    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-workload-certs\") pod \"5a60c9a2-4e23-4e4e-be07-09609b79f023\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") "
Oct 30 13:37:44.477774 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.477741    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-workload-socket\") pod \"5a60c9a2-4e23-4e4e-be07-09609b79f023\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") "
Oct 30 13:37:44.477807 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.477768    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-istio-envoy\") pod \"5a60c9a2-4e23-4e4e-be07-09609b79f023\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") "
Oct 30 13:37:44.477840 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.477817    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-istio-data\") pod \"5a60c9a2-4e23-4e4e-be07-09609b79f023\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") "
Oct 30 13:37:44.478000 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.477983    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a60c9a2-4e23-4e4e-be07-09609b79f023-kube-api-access-hgpk8\" (UniqueName: \"kubernetes.io/projected/5a60c9a2-4e23-4e4e-be07-09609b79f023-kube-api-access-hgpk8\") pod \"5a60c9a2-4e23-4e4e-be07-09609b79f023\" (UID: \"5a60c9a2-4e23-4e4e-be07-09609b79f023\") "
Oct 30 13:37:44.480264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.480124    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-workload-socket" pod "5a60c9a2-4e23-4e4e-be07-09609b79f023" (UID: "5a60c9a2-4e23-4e4e-be07-09609b79f023"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:44.480264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.480161    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-workload-certs" pod "5a60c9a2-4e23-4e4e-be07-09609b79f023" (UID: "5a60c9a2-4e23-4e4e-be07-09609b79f023"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:44.481155 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.480943    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a60c9a2-4e23-4e4e-be07-09609b79f023-istiod-ca-cert" pod "5a60c9a2-4e23-4e4e-be07-09609b79f023" (UID: "5a60c9a2-4e23-4e4e-be07-09609b79f023"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:37:44.481951 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.481918    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a60c9a2-4e23-4e4e-be07-09609b79f023-istio-token" pod "5a60c9a2-4e23-4e4e-be07-09609b79f023" (UID: "5a60c9a2-4e23-4e4e-be07-09609b79f023"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:44.482449 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.482427    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-credential-socket" pod "5a60c9a2-4e23-4e4e-be07-09609b79f023" (UID: "5a60c9a2-4e23-4e4e-be07-09609b79f023"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:44.482633 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.482615    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-istio-data" pod "5a60c9a2-4e23-4e4e-be07-09609b79f023" (UID: "5a60c9a2-4e23-4e4e-be07-09609b79f023"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:44.483192 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.483164    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a60c9a2-4e23-4e4e-be07-09609b79f023-istio-podinfo" pod "5a60c9a2-4e23-4e4e-be07-09609b79f023" (UID: "5a60c9a2-4e23-4e4e-be07-09609b79f023"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:37:44.483616 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.483581    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a60c9a2-4e23-4e4e-be07-09609b79f023-kube-api-access-hgpk8" pod "5a60c9a2-4e23-4e4e-be07-09609b79f023" (UID: "5a60c9a2-4e23-4e4e-be07-09609b79f023"). InnerVolumeSpecName "kube-api-access-hgpk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:44.483828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.483803    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-istio-envoy" pod "5a60c9a2-4e23-4e4e-be07-09609b79f023" (UID: "5a60c9a2-4e23-4e4e-be07-09609b79f023"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:44.520605 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.520568    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:37:44.526964 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.526928    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:37:44.580428 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.580390    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a60c9a2-4e23-4e4e-be07-09609b79f023-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.580428 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.580421    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a60c9a2-4e23-4e4e-be07-09609b79f023-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.580428 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.580432    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.580691 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.580441    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.580691 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.580450    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.580691 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.580458    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.580691 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.580466    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a60c9a2-4e23-4e4e-be07-09609b79f023-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.580691 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.580475    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgpk8\" (UniqueName: \"kubernetes.io/projected/5a60c9a2-4e23-4e4e-be07-09609b79f023-kube-api-access-hgpk8\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.580691 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.580485    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a60c9a2-4e23-4e4e-be07-09609b79f023-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.720246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.682813    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b6f4e05-5105-4596-b248-bc018f59ce81-kube-api-access-kc2xs\" (UniqueName: \"kubernetes.io/projected/8b6f4e05-5105-4596-b248-bc018f59ce81-kube-api-access-kc2xs\") pod \"8b6f4e05-5105-4596-b248-bc018f59ce81\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") "
Oct 30 13:37:44.720664 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.720629    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f90bdf7e-ba66-4f37-9585-5173e85566fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f90bdf7e-ba66-4f37-9585-5173e85566fb-istiod-ca-cert\") pod \"f90bdf7e-ba66-4f37-9585-5173e85566fb\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") "
Oct 30 13:37:44.720813 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.720707    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-credential-socket\") pod \"f90bdf7e-ba66-4f37-9585-5173e85566fb\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") "
Oct 30 13:37:44.720813 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.720767    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-workload-socket\") pod \"f90bdf7e-ba66-4f37-9585-5173e85566fb\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") "
Oct 30 13:37:44.720813 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.720802    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-workload-certs\") pod \"f90bdf7e-ba66-4f37-9585-5173e85566fb\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") "
Oct 30 13:37:44.721028 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.720834    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-istio-data\") pod \"f90bdf7e-ba66-4f37-9585-5173e85566fb\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") "
Oct 30 13:37:44.721028 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.721008    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b6f4e05-5105-4596-b248-bc018f59ce81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6f4e05-5105-4596-b248-bc018f59ce81-istio-podinfo\") pod \"8b6f4e05-5105-4596-b248-bc018f59ce81\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") "
Oct 30 13:37:44.721148 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.721059    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b6f4e05-5105-4596-b248-bc018f59ce81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6f4e05-5105-4596-b248-bc018f59ce81-istiod-ca-cert\") pod \"8b6f4e05-5105-4596-b248-bc018f59ce81\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") "
Oct 30 13:37:44.721148 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.721098    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-workload-socket\") pod \"8b6f4e05-5105-4596-b248-bc018f59ce81\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") "
Oct 30 13:37:44.721359 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.721147    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-istio-envoy\") pod \"f90bdf7e-ba66-4f37-9585-5173e85566fb\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") "
Oct 30 13:37:44.721359 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.721211    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-credential-socket\") pod \"8b6f4e05-5105-4596-b248-bc018f59ce81\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") "
Oct 30 13:37:44.721359 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.721268    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f90bdf7e-ba66-4f37-9585-5173e85566fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f90bdf7e-ba66-4f37-9585-5173e85566fb-istio-podinfo\") pod \"f90bdf7e-ba66-4f37-9585-5173e85566fb\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") "
Oct 30 13:37:44.721359 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.721349    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-istio-data\") pod \"8b6f4e05-5105-4596-b248-bc018f59ce81\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") "
Oct 30 13:37:44.721757 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.721402    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f90bdf7e-ba66-4f37-9585-5173e85566fb-istio-token\" (UniqueName: \"kubernetes.io/projected/f90bdf7e-ba66-4f37-9585-5173e85566fb-istio-token\") pod \"f90bdf7e-ba66-4f37-9585-5173e85566fb\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") "
Oct 30 13:37:44.721757 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.721452    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-workload-certs\") pod \"8b6f4e05-5105-4596-b248-bc018f59ce81\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") "
Oct 30 13:37:44.721757 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.721499    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f90bdf7e-ba66-4f37-9585-5173e85566fb-kube-api-access-k9967\" (UniqueName: \"kubernetes.io/projected/f90bdf7e-ba66-4f37-9585-5173e85566fb-kube-api-access-k9967\") pod \"f90bdf7e-ba66-4f37-9585-5173e85566fb\" (UID: \"f90bdf7e-ba66-4f37-9585-5173e85566fb\") "
Oct 30 13:37:44.721757 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.721606    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b6f4e05-5105-4596-b248-bc018f59ce81-istio-token\" (UniqueName: \"kubernetes.io/projected/8b6f4e05-5105-4596-b248-bc018f59ce81-istio-token\") pod \"8b6f4e05-5105-4596-b248-bc018f59ce81\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") "
Oct 30 13:37:44.721757 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.721636    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-istio-envoy\") pod \"8b6f4e05-5105-4596-b248-bc018f59ce81\" (UID: \"8b6f4e05-5105-4596-b248-bc018f59ce81\") "
Oct 30 13:37:44.728814 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.728776    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-istio-data" pod "8b6f4e05-5105-4596-b248-bc018f59ce81" (UID: "8b6f4e05-5105-4596-b248-bc018f59ce81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:44.731599 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.731557    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-istio-envoy" pod "f90bdf7e-ba66-4f37-9585-5173e85566fb" (UID: "f90bdf7e-ba66-4f37-9585-5173e85566fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:44.731866 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.731829    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-credential-socket" pod "8b6f4e05-5105-4596-b248-bc018f59ce81" (UID: "8b6f4e05-5105-4596-b248-bc018f59ce81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:44.734205 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.734170    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-workload-socket" pod "f90bdf7e-ba66-4f37-9585-5173e85566fb" (UID: "f90bdf7e-ba66-4f37-9585-5173e85566fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:44.734205 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.695995    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:44.734593 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.734572    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f90bdf7e-ba66-4f37-9585-5173e85566fb-istiod-ca-cert" pod "f90bdf7e-ba66-4f37-9585-5173e85566fb" (UID: "f90bdf7e-ba66-4f37-9585-5173e85566fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:37:44.735066 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.734881    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-credential-socket" pod "f90bdf7e-ba66-4f37-9585-5173e85566fb" (UID: "f90bdf7e-ba66-4f37-9585-5173e85566fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:44.735734 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.735492    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-workload-certs" pod "8b6f4e05-5105-4596-b248-bc018f59ce81" (UID: "8b6f4e05-5105-4596-b248-bc018f59ce81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:44.736121 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.735901    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-istio-data" pod "f90bdf7e-ba66-4f37-9585-5173e85566fb" (UID: "f90bdf7e-ba66-4f37-9585-5173e85566fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:44.736510 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.736313    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-workload-certs" pod "f90bdf7e-ba66-4f37-9585-5173e85566fb" (UID: "f90bdf7e-ba66-4f37-9585-5173e85566fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:44.736809 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.736771    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b6f4e05-5105-4596-b248-bc018f59ce81-istiod-ca-cert" pod "8b6f4e05-5105-4596-b248-bc018f59ce81" (UID: "8b6f4e05-5105-4596-b248-bc018f59ce81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:37:44.737240 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.737204    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-workload-socket" pod "8b6f4e05-5105-4596-b248-bc018f59ce81" (UID: "8b6f4e05-5105-4596-b248-bc018f59ce81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:44.752706 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.744908    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b6f4e05-5105-4596-b248-bc018f59ce81-kube-api-access-kc2xs" pod "8b6f4e05-5105-4596-b248-bc018f59ce81" (UID: "8b6f4e05-5105-4596-b248-bc018f59ce81"). InnerVolumeSpecName "kube-api-access-kc2xs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:44.752706 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.745575    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f90bdf7e-ba66-4f37-9585-5173e85566fb-istio-token" pod "f90bdf7e-ba66-4f37-9585-5173e85566fb" (UID: "f90bdf7e-ba66-4f37-9585-5173e85566fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:44.752706 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.745656    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f90bdf7e-ba66-4f37-9585-5173e85566fb-istio-podinfo" pod "f90bdf7e-ba66-4f37-9585-5173e85566fb" (UID: "f90bdf7e-ba66-4f37-9585-5173e85566fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:37:44.756465 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.756425    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-istio-envoy" pod "8b6f4e05-5105-4596-b248-bc018f59ce81" (UID: "8b6f4e05-5105-4596-b248-bc018f59ce81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:44.756613 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.756431    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b6f4e05-5105-4596-b248-bc018f59ce81-istio-podinfo" pod "8b6f4e05-5105-4596-b248-bc018f59ce81" (UID: "8b6f4e05-5105-4596-b248-bc018f59ce81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:37:44.756613 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.756563    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f90bdf7e-ba66-4f37-9585-5173e85566fb-kube-api-access-k9967" pod "f90bdf7e-ba66-4f37-9585-5173e85566fb" (UID: "f90bdf7e-ba66-4f37-9585-5173e85566fb"). InnerVolumeSpecName "kube-api-access-k9967". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:44.757001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.756961    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b6f4e05-5105-4596-b248-bc018f59ce81-istio-token" pod "8b6f4e05-5105-4596-b248-bc018f59ce81" (UID: "8b6f4e05-5105-4596-b248-bc018f59ce81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:44.825419 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.825380    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.825419 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.825418    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f90bdf7e-ba66-4f37-9585-5173e85566fb-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.825647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.825437    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.825647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.825447    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f90bdf7e-ba66-4f37-9585-5173e85566fb-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.825647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.825455    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.825647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.825463    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9967\" (UniqueName: \"kubernetes.io/projected/f90bdf7e-ba66-4f37-9585-5173e85566fb-kube-api-access-k9967\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.825647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.825475    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b6f4e05-5105-4596-b248-bc018f59ce81-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.825647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.825488    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.825647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.825498    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc2xs\" (UniqueName: \"kubernetes.io/projected/8b6f4e05-5105-4596-b248-bc018f59ce81-kube-api-access-kc2xs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.825647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.825506    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f90bdf7e-ba66-4f37-9585-5173e85566fb-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.825647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.825515    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.825647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.825526    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.825647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.825542    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.825647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.825556    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.825647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.825568    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6f4e05-5105-4596-b248-bc018f59ce81-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.825647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.825577    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6f4e05-5105-4596-b248-bc018f59ce81-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.825647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.825585    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6f4e05-5105-4596-b248-bc018f59ce81-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.825647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.825593    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f90bdf7e-ba66-4f37-9585-5173e85566fb-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.919305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.911211    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"
Oct 30 13:37:44.919305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.911268    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2" event={"ID":"5a60c9a2-4e23-4e4e-be07-09609b79f023","Type":"ContainerDied","Data":"d6ff3cd1995e2295bf36c296f324a3ddc27ea2a491c0b7f32df677a3c9a19952"}
Oct 30 13:37:44.919305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.911352    4544 scope.go:122] "RemoveContainer" containerID="2773c3435f6f1ba8746bfb50ec5af3da37ee191eecb9e36e167615f6e1e96ae5"
Oct 30 13:37:44.919305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.913404    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g" event={"ID":"8b6f4e05-5105-4596-b248-bc018f59ce81","Type":"ContainerDied","Data":"b299268c320da03565c480f057a8553a7ab2071612ff52b9fdc481b84d826e0f"}
Oct 30 13:37:44.919305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.913522    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"
Oct 30 13:37:44.932628 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.931310    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp" event={"ID":"f90bdf7e-ba66-4f37-9585-5173e85566fb","Type":"ContainerDied","Data":"90c3040c250e658afdc45b34137ff9839d9217dc9864200fb265e068aacef586"}
Oct 30 13:37:44.932628 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.932023    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"
Oct 30 13:37:44.937405 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.936552    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4f385-t528x" event={"ID":"ee5e270c-f424-4803-992f-a49fbc67c6dd","Type":"ContainerStarted","Data":"e50f322507ae05231c350075560556cc697d72b5ea682bcd0932ebc4a7662456"}
Oct 30 13:37:44.945601 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.944609    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:37:44.945601 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.944669    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2" podUID="5a60c9a2-4e23-4e4e-be07-09609b79f023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:37:44.949832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.949745    4544 scope.go:122] "RemoveContainer" containerID="b80f1275112c4ee5ff483cf8352a3d6cd269f111fde61e65075c0648ed69b0a5"
Oct 30 13:37:44.966347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.966321    4544 scope.go:122] "RemoveContainer" containerID="29bc0e6821304515e028dab5ad10551f3b28ec78e428dd737391a7098f100555"
Oct 30 13:37:44.988875 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.988826    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"]
Oct 30 13:37:44.997804 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:44.997766    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g"]
Oct 30 13:37:45.059613 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:45.059559    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4f385-t528x" podStartSLOduration=2.05954404 podStartE2EDuration="2.05954404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:45.058682794 +0000 UTC m=+3160.105816746" watchObservedRunningTime="2026-10-30 13:37:45.05954404 +0000 UTC m=+3160.106677992"
Oct 30 13:37:45.090817 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:45.090781    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b6f4e05-5105-4596-b248-bc018f59ce81" path="/var/lib/kubelet/pods/8b6f4e05-5105-4596-b248-bc018f59ce81/volumes"
Oct 30 13:37:45.098789 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:45.098756    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"]
Oct 30 13:37:45.104192 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:45.104154    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2"]
Oct 30 13:37:45.152585 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:45.152541    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"]
Oct 30 13:37:45.161555 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:45.161508    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp"]
Oct 30 13:37:45.947129 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:45.947092    4544 generic.go:356] "Generic (PLEG): container finished" podID="ee5e270c-f424-4803-992f-a49fbc67c6dd" containerID="e50f322507ae05231c350075560556cc697d72b5ea682bcd0932ebc4a7662456" exitCode=0
Oct 30 13:37:45.947655 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:45.947172    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4f385-t528x" event={"ID":"ee5e270c-f424-4803-992f-a49fbc67c6dd","Type":"ContainerDied","Data":"e50f322507ae05231c350075560556cc697d72b5ea682bcd0932ebc4a7662456"}
Oct 30 13:37:46.078605 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:46.078571    4544 scope.go:122] "RemoveContainer" containerID="4c0b0f15d3a23f14b42ae835200a1924674449a5d5e370e3277514886209cb08"
Oct 30 13:37:46.917882 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:46.917829    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:47.002436 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:47.002384    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl" event={"ID":"eb8eebdc-5933-463b-a511-264945e917e9","Type":"ContainerStarted","Data":"c4b29abc4bba9dc8b6e8cce9e20baeaa9a42b17bdf61b7deed106bf581f606dc"}
Oct 30 13:37:47.096981 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:47.096938    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a60c9a2-4e23-4e4e-be07-09609b79f023" path="/var/lib/kubelet/pods/5a60c9a2-4e23-4e4e-be07-09609b79f023/volumes"
Oct 30 13:37:47.097743 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:47.097712    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f90bdf7e-ba66-4f37-9585-5173e85566fb" path="/var/lib/kubelet/pods/f90bdf7e-ba66-4f37-9585-5173e85566fb/volumes"
Oct 30 13:37:47.199726 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:47.199699    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4f385-t528x"
Oct 30 13:37:47.293306 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:47.293251    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee5e270c-f424-4803-992f-a49fbc67c6dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5e270c-f424-4803-992f-a49fbc67c6dd-maas-api-xks-ca\") pod \"ee5e270c-f424-4803-992f-a49fbc67c6dd\" (UID: \"ee5e270c-f424-4803-992f-a49fbc67c6dd\") "
Oct 30 13:37:47.293580 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:47.293557    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee5e270c-f424-4803-992f-a49fbc67c6dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5e270c-f424-4803-992f-a49fbc67c6dd-maas-api-openshift-ca\") pod \"ee5e270c-f424-4803-992f-a49fbc67c6dd\" (UID: \"ee5e270c-f424-4803-992f-a49fbc67c6dd\") "
Oct 30 13:37:47.293677 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:47.293646    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee5e270c-f424-4803-992f-a49fbc67c6dd-kube-api-access-qxgx8\" (UniqueName: \"kubernetes.io/projected/ee5e270c-f424-4803-992f-a49fbc67c6dd-kube-api-access-qxgx8\") pod \"ee5e270c-f424-4803-992f-a49fbc67c6dd\" (UID: \"ee5e270c-f424-4803-992f-a49fbc67c6dd\") "
Oct 30 13:37:47.293997 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:47.293967    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5e270c-f424-4803-992f-a49fbc67c6dd-maas-api-openshift-ca" pod "ee5e270c-f424-4803-992f-a49fbc67c6dd" (UID: "ee5e270c-f424-4803-992f-a49fbc67c6dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:37:47.294686 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:47.294663    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5e270c-f424-4803-992f-a49fbc67c6dd-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:47.295659 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:47.295629    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee5e270c-f424-4803-992f-a49fbc67c6dd-maas-api-xks-ca" pod "ee5e270c-f424-4803-992f-a49fbc67c6dd" (UID: "ee5e270c-f424-4803-992f-a49fbc67c6dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:37:47.295865 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:47.295845    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5e270c-f424-4803-992f-a49fbc67c6dd-kube-api-access-qxgx8" pod "ee5e270c-f424-4803-992f-a49fbc67c6dd" (UID: "ee5e270c-f424-4803-992f-a49fbc67c6dd"). InnerVolumeSpecName "kube-api-access-qxgx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:47.396311 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:47.396212    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxgx8\" (UniqueName: \"kubernetes.io/projected/ee5e270c-f424-4803-992f-a49fbc67c6dd-kube-api-access-qxgx8\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:47.396311 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:47.396310    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5e270c-f424-4803-992f-a49fbc67c6dd-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:48.014443 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:48.014403    4544 generic.go:356] "Generic (PLEG): container finished" podID="eb8eebdc-5933-463b-a511-264945e917e9" containerID="c4b29abc4bba9dc8b6e8cce9e20baeaa9a42b17bdf61b7deed106bf581f606dc" exitCode=0
Oct 30 13:37:48.014925 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:48.014472    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl" event={"ID":"eb8eebdc-5933-463b-a511-264945e917e9","Type":"ContainerDied","Data":"c4b29abc4bba9dc8b6e8cce9e20baeaa9a42b17bdf61b7deed106bf581f606dc"}
Oct 30 13:37:48.014925 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:48.014538    4544 scope.go:122] "RemoveContainer" containerID="4c0b0f15d3a23f14b42ae835200a1924674449a5d5e370e3277514886209cb08"
Oct 30 13:37:48.016480 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:48.016457    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4f385-t528x"
Oct 30 13:37:48.016576 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:48.016480    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4f385-t528x" event={"ID":"ee5e270c-f424-4803-992f-a49fbc67c6dd","Type":"ContainerDied","Data":"1e00236fa32083a42414cc7b663534949f7dc133713c215160f0cbe5ac34222e"}
Oct 30 13:37:48.016576 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:48.016529    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e00236fa32083a42414cc7b663534949f7dc133713c215160f0cbe5ac34222e"
Oct 30 13:37:48.079204 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:48.079173    4544 scope.go:122] "RemoveContainer" containerID="fa495ef862e2ccacf50595f14e569e40bfd4f33545d071914e3b5f122fe7ab70"
Oct 30 13:37:48.094769 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:48.094732    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl"]
Oct 30 13:37:48.904720 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:48.904685    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:49.032540 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:49.032496    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5" event={"ID":"01e24b49-04cb-4af7-a9cc-a0b6fa608451","Type":"ContainerStarted","Data":"51960c74027d772c41a8d22a93196f755b2bf7653912db1fdbf3b1f4cb0865f6"}
Oct 30 13:37:49.233969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:49.233933    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl"
Oct 30 13:37:49.363779 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:49.363741    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb8eebdc-5933-463b-a511-264945e917e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb8eebdc-5933-463b-a511-264945e917e9-maas-api-openshift-ca\") pod \"eb8eebdc-5933-463b-a511-264945e917e9\" (UID: \"eb8eebdc-5933-463b-a511-264945e917e9\") "
Oct 30 13:37:49.364027 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:49.364009    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb8eebdc-5933-463b-a511-264945e917e9-kube-api-access-79rz4\" (UniqueName: \"kubernetes.io/projected/eb8eebdc-5933-463b-a511-264945e917e9-kube-api-access-79rz4\") pod \"eb8eebdc-5933-463b-a511-264945e917e9\" (UID: \"eb8eebdc-5933-463b-a511-264945e917e9\") "
Oct 30 13:37:49.364110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:49.364056    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb8eebdc-5933-463b-a511-264945e917e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb8eebdc-5933-463b-a511-264945e917e9-maas-api-xks-ca\") pod \"eb8eebdc-5933-463b-a511-264945e917e9\" (UID: \"eb8eebdc-5933-463b-a511-264945e917e9\") "
Oct 30 13:37:49.364308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:49.364211    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb8eebdc-5933-463b-a511-264945e917e9-maas-api-openshift-ca" pod "eb8eebdc-5933-463b-a511-264945e917e9" (UID: "eb8eebdc-5933-463b-a511-264945e917e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:37:49.364767 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:49.364744    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb8eebdc-5933-463b-a511-264945e917e9-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:49.366262 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:49.366198    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb8eebdc-5933-463b-a511-264945e917e9-maas-api-xks-ca" pod "eb8eebdc-5933-463b-a511-264945e917e9" (UID: "eb8eebdc-5933-463b-a511-264945e917e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:37:49.366361 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:49.366339    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb8eebdc-5933-463b-a511-264945e917e9-kube-api-access-79rz4" pod "eb8eebdc-5933-463b-a511-264945e917e9" (UID: "eb8eebdc-5933-463b-a511-264945e917e9"). InnerVolumeSpecName "kube-api-access-79rz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:49.466768 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:49.466653    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79rz4\" (UniqueName: \"kubernetes.io/projected/eb8eebdc-5933-463b-a511-264945e917e9-kube-api-access-79rz4\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:49.466768 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:49.466706    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb8eebdc-5933-463b-a511-264945e917e9-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:49.851160 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:49.851114    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4"]
Oct 30 13:37:49.981291 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:49.981198    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:50.042958 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:50.042925    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl"
Oct 30 13:37:50.042958 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:50.042944    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl" event={"ID":"eb8eebdc-5933-463b-a511-264945e917e9","Type":"ContainerDied","Data":"faa0f508ff6e00f6587b279c7b25d9564ca5de28d837846e4d1d3f509dc63f51"}
Oct 30 13:37:50.043512 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:50.042996    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="faa0f508ff6e00f6587b279c7b25d9564ca5de28d837846e4d1d3f509dc63f51"
Oct 30 13:37:50.045187 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:50.045160    4544 generic.go:356] "Generic (PLEG): container finished" podID="01e24b49-04cb-4af7-a9cc-a0b6fa608451" containerID="51960c74027d772c41a8d22a93196f755b2bf7653912db1fdbf3b1f4cb0865f6" exitCode=0
Oct 30 13:37:50.045375 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:50.045245    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5" event={"ID":"01e24b49-04cb-4af7-a9cc-a0b6fa608451","Type":"ContainerDied","Data":"51960c74027d772c41a8d22a93196f755b2bf7653912db1fdbf3b1f4cb0865f6"}
Oct 30 13:37:50.045375 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:50.045297    4544 scope.go:122] "RemoveContainer" containerID="fa495ef862e2ccacf50595f14e569e40bfd4f33545d071914e3b5f122fe7ab70"
Oct 30 13:37:50.164539 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:50.162905    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:50.167063 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:50.167029    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5"]
Oct 30 13:37:50.196884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:50.196843    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl"]
Oct 30 13:37:50.202327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:50.202295    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-72208f-cwgnl"]
Oct 30 13:37:51.074772 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:51.074739    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:51.133582 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:51.133541    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb8eebdc-5933-463b-a511-264945e917e9" path="/var/lib/kubelet/pods/eb8eebdc-5933-463b-a511-264945e917e9/volumes"
Oct 30 13:37:51.304056 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:51.304027    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5"
Oct 30 13:37:51.325128 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:51.325093    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01e24b49-04cb-4af7-a9cc-a0b6fa608451-kube-api-access-vghrl\" (UniqueName: \"kubernetes.io/projected/01e24b49-04cb-4af7-a9cc-a0b6fa608451-kube-api-access-vghrl\") pod \"01e24b49-04cb-4af7-a9cc-a0b6fa608451\" (UID: \"01e24b49-04cb-4af7-a9cc-a0b6fa608451\") "
Oct 30 13:37:51.325540 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:51.325509    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01e24b49-04cb-4af7-a9cc-a0b6fa608451-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01e24b49-04cb-4af7-a9cc-a0b6fa608451-maas-api-xks-ca\") pod \"01e24b49-04cb-4af7-a9cc-a0b6fa608451\" (UID: \"01e24b49-04cb-4af7-a9cc-a0b6fa608451\") "
Oct 30 13:37:51.325662 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:51.325550    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01e24b49-04cb-4af7-a9cc-a0b6fa608451-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01e24b49-04cb-4af7-a9cc-a0b6fa608451-maas-api-openshift-ca\") pod \"01e24b49-04cb-4af7-a9cc-a0b6fa608451\" (UID: \"01e24b49-04cb-4af7-a9cc-a0b6fa608451\") "
Oct 30 13:37:51.325968 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:51.325946    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01e24b49-04cb-4af7-a9cc-a0b6fa608451-maas-api-openshift-ca" pod "01e24b49-04cb-4af7-a9cc-a0b6fa608451" (UID: "01e24b49-04cb-4af7-a9cc-a0b6fa608451"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:37:51.326211 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:51.326194    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01e24b49-04cb-4af7-a9cc-a0b6fa608451-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:51.327458 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:51.327433    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01e24b49-04cb-4af7-a9cc-a0b6fa608451-kube-api-access-vghrl" pod "01e24b49-04cb-4af7-a9cc-a0b6fa608451" (UID: "01e24b49-04cb-4af7-a9cc-a0b6fa608451"). InnerVolumeSpecName "kube-api-access-vghrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:51.327834 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:51.327814    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01e24b49-04cb-4af7-a9cc-a0b6fa608451-maas-api-xks-ca" pod "01e24b49-04cb-4af7-a9cc-a0b6fa608451" (UID: "01e24b49-04cb-4af7-a9cc-a0b6fa608451"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:37:51.427933 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:51.427891    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vghrl\" (UniqueName: \"kubernetes.io/projected/01e24b49-04cb-4af7-a9cc-a0b6fa608451-kube-api-access-vghrl\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:51.427933 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:51.427935    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01e24b49-04cb-4af7-a9cc-a0b6fa608451-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:52.067179 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:52.067147    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5" event={"ID":"01e24b49-04cb-4af7-a9cc-a0b6fa608451","Type":"ContainerDied","Data":"8dae0828007976cddd46d7e847ffc1bf85ca40e83eabdfff99d1de17d343dcef"}
Oct 30 13:37:52.067386 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:52.067200    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8dae0828007976cddd46d7e847ffc1bf85ca40e83eabdfff99d1de17d343dcef"
Oct 30 13:37:52.067386 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:52.067158    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5"
Oct 30 13:37:52.148577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:52.148539    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5"]
Oct 30 13:37:52.154139 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:52.154100    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78543b-5xpk5"]
Oct 30 13:37:52.789691 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:52.789660    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:53.101933 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:53.101896    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01e24b49-04cb-4af7-a9cc-a0b6fa608451" path="/var/lib/kubelet/pods/01e24b49-04cb-4af7-a9cc-a0b6fa608451/volumes"
Oct 30 13:37:54.641697 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:54.641659    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:54.852199 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:54.852132    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4" podUID="fa3df092-b18c-47fb-8793-90f9869a1078" containerName="payload-pre-processing" containerID="cri-o://04ff21d5441999697d91f7826ef964e1d1c457a79ac66a37b767125b11cdd8fb" gracePeriod=25
Oct 30 13:37:56.828654 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:56.828602    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:56.969292 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:56.946016    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:57.555841 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:57.555799    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4" podUID="fa3df092-b18c-47fb-8793-90f9869a1078" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:37:58.117557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:58.117516    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:58.408890 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:58.408794    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:58.661646 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:58.661559    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:37:59.009180 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:37:59.009086    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:37:59.477023 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:59.476979    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main" containerID="cri-o://ec8d4660164a489650899c9662278b1fd4ce42117fef8bfa0bf32b7155990a26" gracePeriod=45
Oct 30 13:37:59.814808 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:59.814771    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 13:37:59.941494 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:59.941453    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-home\" (UniqueName: \"kubernetes.io/empty-dir/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-home\") pod \"91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2\" (UID: \"91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2\") "
Oct 30 13:37:59.941693 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:59.941622    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-tls-certs\" (UniqueName: \"kubernetes.io/secret/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-tls-certs\") pod \"91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2\" (UID: \"91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2\") "
Oct 30 13:37:59.941785 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:59.941765    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-model-cache\") pod \"91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2\" (UID: \"91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2\") "
Oct 30 13:37:59.941785 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:59.941771    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-home" pod "91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" (UID: "91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:59.941943 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:59.941844    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-tmp-dir\") pod \"91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2\" (UID: \"91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2\") "
Oct 30 13:37:59.942038 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:59.942017    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-model-cache" pod "91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" (UID: "91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:59.942115 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:59.942041    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-dshm\") pod \"91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2\" (UID: \"91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2\") "
Oct 30 13:37:59.942179 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:59.942128    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-kube-api-access-hsqbf\" (UniqueName: \"kubernetes.io/projected/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-kube-api-access-hsqbf\") pod \"91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2\" (UID: \"91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2\") "
Oct 30 13:37:59.942301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:59.942189    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-tmp-dir" pod "91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" (UID: "91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:59.943304 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:59.943279    4544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-model-cache\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:59.943304 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:59.943306    4544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-tmp-dir\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:59.943524 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:59.943320    4544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-home\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:37:59.944997 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:59.944973    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-tls-certs" pod "91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" (UID: "91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:37:59.944997 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:59.944979    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-dshm" pod "91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" (UID: "91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:59.945526 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:37:59.945504    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-kube-api-access-hsqbf" pod "91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" (UID: "91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2"). InnerVolumeSpecName "kube-api-access-hsqbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:38:00.045518 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:00.045425    4544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-tls-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:00.045518 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:00.045462    4544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-dshm\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:00.045518 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:00.045472    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsqbf\" (UniqueName: \"kubernetes.io/projected/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2-kube-api-access-hsqbf\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:00.158057 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:00.158022    4544 generic.go:356] "Generic (PLEG): container finished" podID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerID="ec8d4660164a489650899c9662278b1fd4ce42117fef8bfa0bf32b7155990a26" exitCode=0
Oct 30 13:38:00.158310 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:00.158077    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"
Oct 30 13:38:00.158310 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:00.158101    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" event={"ID":"91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2","Type":"ContainerDied","Data":"ec8d4660164a489650899c9662278b1fd4ce42117fef8bfa0bf32b7155990a26"}
Oct 30 13:38:00.158310 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:00.158157    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf" event={"ID":"91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2","Type":"ContainerDied","Data":"925dfece6b0b78b9171040b7322218a7032d41ad764799e11d1dcdac855fb6de"}
Oct 30 13:38:00.158310 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:00.158181    4544 scope.go:122] "RemoveContainer" containerID="ec8d4660164a489650899c9662278b1fd4ce42117fef8bfa0bf32b7155990a26"
Oct 30 13:38:00.170904 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:00.170878    4544 scope.go:122] "RemoveContainer" containerID="ec8d4660164a489650899c9662278b1fd4ce42117fef8bfa0bf32b7155990a26"
Oct 30 13:38:00.171322 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:38:00.171299    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec8d4660164a489650899c9662278b1fd4ce42117fef8bfa0bf32b7155990a26\": container with ID starting with ec8d4660164a489650899c9662278b1fd4ce42117fef8bfa0bf32b7155990a26 not found: ID does not exist" containerID="ec8d4660164a489650899c9662278b1fd4ce42117fef8bfa0bf32b7155990a26"
Oct 30 13:38:00.171419 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:00.171337    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec8d4660164a489650899c9662278b1fd4ce42117fef8bfa0bf32b7155990a26"} err="failed to get container status \"ec8d4660164a489650899c9662278b1fd4ce42117fef8bfa0bf32b7155990a26\": rpc error: code = NotFound desc = could not find container \"ec8d4660164a489650899c9662278b1fd4ce42117fef8bfa0bf32b7155990a26\": container with ID starting with ec8d4660164a489650899c9662278b1fd4ce42117fef8bfa0bf32b7155990a26 not found: ID does not exist"
Oct 30 13:38:00.214272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:00.214231    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"]
Oct 30 13:38:00.226278 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:00.226237    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-09db16/simulator-w3-09db16-kserve-687f7bfcc6-hl6tf"]
Oct 30 13:38:00.480081 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:00.480038    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:01.091120 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:01.091069    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" path="/var/lib/kubelet/pods/91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2/volumes"
Oct 30 13:38:03.953367 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:03.953332    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:04.639731 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:04.639682    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:06.332944 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:06.332897    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:07.547565 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:07.547513    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4" podUID="fa3df092-b18c-47fb-8793-90f9869a1078" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:38:08.246817 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:08.246769    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:10.199151 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:10.199110    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:11.872565 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:11.872530    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:13.782807 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:13.782769    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:15.172511 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:15.172479    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4"
Oct 30 13:38:15.314027 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:15.313992    4544 generic.go:356] "Generic (PLEG): container finished" podID="fa3df092-b18c-47fb-8793-90f9869a1078" containerID="04ff21d5441999697d91f7826ef964e1d1c457a79ac66a37b767125b11cdd8fb" exitCode=0
Oct 30 13:38:15.314280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:15.314058    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4"
Oct 30 13:38:15.314280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:15.314069    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4" event={"ID":"fa3df092-b18c-47fb-8793-90f9869a1078","Type":"ContainerDied","Data":"04ff21d5441999697d91f7826ef964e1d1c457a79ac66a37b767125b11cdd8fb"}
Oct 30 13:38:15.314280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:15.314122    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4" event={"ID":"fa3df092-b18c-47fb-8793-90f9869a1078","Type":"ContainerDied","Data":"821861ec49d958e9af2b872c0b60a4cae3c68833574ef71aacb20069c672df68"}
Oct 30 13:38:15.314280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:15.314151    4544 scope.go:122] "RemoveContainer" containerID="04ff21d5441999697d91f7826ef964e1d1c457a79ac66a37b767125b11cdd8fb"
Oct 30 13:38:15.329268 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:15.329204    4544 scope.go:122] "RemoveContainer" containerID="04ff21d5441999697d91f7826ef964e1d1c457a79ac66a37b767125b11cdd8fb"
Oct 30 13:38:15.329622 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:38:15.329602    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04ff21d5441999697d91f7826ef964e1d1c457a79ac66a37b767125b11cdd8fb\": container with ID starting with 04ff21d5441999697d91f7826ef964e1d1c457a79ac66a37b767125b11cdd8fb not found: ID does not exist" containerID="04ff21d5441999697d91f7826ef964e1d1c457a79ac66a37b767125b11cdd8fb"
Oct 30 13:38:15.329690 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:15.329633    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04ff21d5441999697d91f7826ef964e1d1c457a79ac66a37b767125b11cdd8fb"} err="failed to get container status \"04ff21d5441999697d91f7826ef964e1d1c457a79ac66a37b767125b11cdd8fb\": rpc error: code = NotFound desc = could not find container \"04ff21d5441999697d91f7826ef964e1d1c457a79ac66a37b767125b11cdd8fb\": container with ID starting with 04ff21d5441999697d91f7826ef964e1d1c457a79ac66a37b767125b11cdd8fb not found: ID does not exist"
Oct 30 13:38:15.354561 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:15.354518    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa3df092-b18c-47fb-8793-90f9869a1078-kube-api-access-fzdk8\" (UniqueName: \"kubernetes.io/projected/fa3df092-b18c-47fb-8793-90f9869a1078-kube-api-access-fzdk8\") pod \"fa3df092-b18c-47fb-8793-90f9869a1078\" (UID: \"fa3df092-b18c-47fb-8793-90f9869a1078\") "
Oct 30 13:38:15.354748 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:15.354612    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa3df092-b18c-47fb-8793-90f9869a1078-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa3df092-b18c-47fb-8793-90f9869a1078-plugins-config-volume\") pod \"fa3df092-b18c-47fb-8793-90f9869a1078\" (UID: \"fa3df092-b18c-47fb-8793-90f9869a1078\") "
Oct 30 13:38:15.354987 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:15.354966    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa3df092-b18c-47fb-8793-90f9869a1078-plugins-config-volume" pod "fa3df092-b18c-47fb-8793-90f9869a1078" (UID: "fa3df092-b18c-47fb-8793-90f9869a1078"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:38:15.355495 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:15.355476    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa3df092-b18c-47fb-8793-90f9869a1078-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:15.356943 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:15.356921    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa3df092-b18c-47fb-8793-90f9869a1078-kube-api-access-fzdk8" pod "fa3df092-b18c-47fb-8793-90f9869a1078" (UID: "fa3df092-b18c-47fb-8793-90f9869a1078"). InnerVolumeSpecName "kube-api-access-fzdk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:38:15.457380 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:15.457283    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzdk8\" (UniqueName: \"kubernetes.io/projected/fa3df092-b18c-47fb-8793-90f9869a1078-kube-api-access-fzdk8\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:15.575513 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:15.575473    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:15.698054 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:15.698015    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4"]
Oct 30 13:38:15.702994 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:15.702962    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4"]
Oct 30 13:38:17.089654 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:17.089581    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa3df092-b18c-47fb-8793-90f9869a1078" path="/var/lib/kubelet/pods/fa3df092-b18c-47fb-8793-90f9869a1078/volumes"
Oct 30 13:38:17.283832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:17.283801    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:18.806045 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:18.806003    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:19.047812 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:19.047775    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:19.177252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:19.177199    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:19.564934 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:19.564841    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:19.789750 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:19.789704    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:19.801953 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:19.801917    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4f385-t528x"]
Oct 30 13:38:19.812105 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:19.812067    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4f385-t528x"]
Oct 30 13:38:19.914832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:19.914800    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:20.033706 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:20.033165    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:20.342518 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:20.342487    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:21.089004 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:21.088963    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee5e270c-f424-4803-992f-a49fbc67c6dd" path="/var/lib/kubelet/pods/ee5e270c-f424-4803-992f-a49fbc67c6dd/volumes"
Oct 30 13:38:21.724795 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:21.724756    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:22.039302 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:22.039163    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:22.170188 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:22.170155    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:22.307868 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:22.307760    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:22.886981 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:22.886937    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:23.095889 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:23.095834    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:23.135441 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:23.135405    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76405c-zt5nd"]
Oct 30 13:38:23.147305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:23.147210    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-76405c-zt5nd"]
Oct 30 13:38:23.223273 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:23.222234    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:23.524357 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:23.524253    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:23.689763 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:23.689722    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:25.090851 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:25.090816    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5490d4aa-6c4d-486f-b953-05331cc36305" path="/var/lib/kubelet/pods/5490d4aa-6c4d-486f-b953-05331cc36305/volumes"
Oct 30 13:38:25.422699 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:25.422595    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:26.955094 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:26.955060    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:27.128363 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:27.128332    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:27.364179 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:27.364122    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"]
Oct 30 13:38:27.364575 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:27.364535    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt" podUID="68fa1f90-25f8-442f-96ff-12e6a716db84" containerName="istio-proxy" containerID="cri-o://67f408b3ed9734ca22c1321171a4edb0cb40399d8e76a9cffe9c96bf93cb9bc2" gracePeriod=30
Oct 30 13:38:27.385349 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:27.385297    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:27.760491 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:27.760373    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:29.011140 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:38:29.011103    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:38:30.000652 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:30.000617    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:30.011828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:30.011788    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:38:30.012270 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:30.011849    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt" podUID="68fa1f90-25f8-442f-96ff-12e6a716db84" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:38:31.445234 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:31.445188    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:31.749785 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:31.749677    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:31.905392 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:31.905328    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:32.068683 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.068570    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"]
Oct 30 13:38:32.069004 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.068976    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7" podUID="820333af-678e-40c7-a1b7-24cab9591ea2" containerName="istio-proxy" containerID="cri-o://c705b45706084fb81e7b52018a64159c3b77147b07870ef42a989b8cf6e09fd2" gracePeriod=30
Oct 30 13:38:32.358178 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.358141    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:32.512133 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.512096    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:32.744335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.744297    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:38:32.895578 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.895548    4544 generic.go:356] "Generic (PLEG): container finished" podID="68fa1f90-25f8-442f-96ff-12e6a716db84" containerID="67f408b3ed9734ca22c1321171a4edb0cb40399d8e76a9cffe9c96bf93cb9bc2" exitCode=0
Oct 30 13:38:32.895775 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.895597    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"
Oct 30 13:38:32.895775 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.895629    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt" event={"ID":"68fa1f90-25f8-442f-96ff-12e6a716db84","Type":"ContainerDied","Data":"67f408b3ed9734ca22c1321171a4edb0cb40399d8e76a9cffe9c96bf93cb9bc2"}
Oct 30 13:38:32.895775 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.895685    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt" event={"ID":"68fa1f90-25f8-442f-96ff-12e6a716db84","Type":"ContainerDied","Data":"39463f8324d3e8f456aa4f1046b0a7619495d383011e3148539c6826e54458cb"}
Oct 30 13:38:32.895775 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.895708    4544 scope.go:122] "RemoveContainer" containerID="67f408b3ed9734ca22c1321171a4edb0cb40399d8e76a9cffe9c96bf93cb9bc2"
Oct 30 13:38:32.907758 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.907737    4544 scope.go:122] "RemoveContainer" containerID="1b5b1864e485d3ba2317e500e7781d058ddcd9712fb6a5eaaf154303bec16dd3"
Oct 30 13:38:32.912762 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.912736    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-istio-envoy\") pod \"68fa1f90-25f8-442f-96ff-12e6a716db84\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") "
Oct 30 13:38:32.912915 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.912797    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-workload-certs\") pod \"68fa1f90-25f8-442f-96ff-12e6a716db84\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") "
Oct 30 13:38:32.912915 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.912862    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-istio-data\") pod \"68fa1f90-25f8-442f-96ff-12e6a716db84\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") "
Oct 30 13:38:32.913074 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.913051    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68fa1f90-25f8-442f-96ff-12e6a716db84-istio-token\" (UniqueName: \"kubernetes.io/projected/68fa1f90-25f8-442f-96ff-12e6a716db84-istio-token\") pod \"68fa1f90-25f8-442f-96ff-12e6a716db84\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") "
Oct 30 13:38:32.913207 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.913110    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-workload-certs" pod "68fa1f90-25f8-442f-96ff-12e6a716db84" (UID: "68fa1f90-25f8-442f-96ff-12e6a716db84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:38:32.913207 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.913123    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68fa1f90-25f8-442f-96ff-12e6a716db84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68fa1f90-25f8-442f-96ff-12e6a716db84-istio-podinfo\") pod \"68fa1f90-25f8-442f-96ff-12e6a716db84\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") "
Oct 30 13:38:32.913417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.913288    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-istio-data" pod "68fa1f90-25f8-442f-96ff-12e6a716db84" (UID: "68fa1f90-25f8-442f-96ff-12e6a716db84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:38:32.913417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.913300    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68fa1f90-25f8-442f-96ff-12e6a716db84-kube-api-access-5nnlx\" (UniqueName: \"kubernetes.io/projected/68fa1f90-25f8-442f-96ff-12e6a716db84-kube-api-access-5nnlx\") pod \"68fa1f90-25f8-442f-96ff-12e6a716db84\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") "
Oct 30 13:38:32.913417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.913396    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-workload-socket\") pod \"68fa1f90-25f8-442f-96ff-12e6a716db84\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") "
Oct 30 13:38:32.913852 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.913476    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68fa1f90-25f8-442f-96ff-12e6a716db84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68fa1f90-25f8-442f-96ff-12e6a716db84-istiod-ca-cert\") pod \"68fa1f90-25f8-442f-96ff-12e6a716db84\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") "
Oct 30 13:38:32.913852 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.913519    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-credential-socket\") pod \"68fa1f90-25f8-442f-96ff-12e6a716db84\" (UID: \"68fa1f90-25f8-442f-96ff-12e6a716db84\") "
Oct 30 13:38:32.913852 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.913706    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-workload-socket" pod "68fa1f90-25f8-442f-96ff-12e6a716db84" (UID: "68fa1f90-25f8-442f-96ff-12e6a716db84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:38:32.914335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.913929    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-credential-socket" pod "68fa1f90-25f8-442f-96ff-12e6a716db84" (UID: "68fa1f90-25f8-442f-96ff-12e6a716db84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:38:32.914653 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.914628    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:32.914701 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.914662    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:32.914701 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.914679    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:32.914701 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.914693    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:32.914889 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.914870    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68fa1f90-25f8-442f-96ff-12e6a716db84-istiod-ca-cert" pod "68fa1f90-25f8-442f-96ff-12e6a716db84" (UID: "68fa1f90-25f8-442f-96ff-12e6a716db84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:38:32.915482 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.915461    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68fa1f90-25f8-442f-96ff-12e6a716db84-istio-podinfo" pod "68fa1f90-25f8-442f-96ff-12e6a716db84" (UID: "68fa1f90-25f8-442f-96ff-12e6a716db84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:38:32.915737 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.915700    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-istio-envoy" pod "68fa1f90-25f8-442f-96ff-12e6a716db84" (UID: "68fa1f90-25f8-442f-96ff-12e6a716db84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:38:32.915737 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.915721    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68fa1f90-25f8-442f-96ff-12e6a716db84-istio-token" pod "68fa1f90-25f8-442f-96ff-12e6a716db84" (UID: "68fa1f90-25f8-442f-96ff-12e6a716db84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:38:32.915996 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.915976    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68fa1f90-25f8-442f-96ff-12e6a716db84-kube-api-access-5nnlx" pod "68fa1f90-25f8-442f-96ff-12e6a716db84" (UID: "68fa1f90-25f8-442f-96ff-12e6a716db84"). InnerVolumeSpecName "kube-api-access-5nnlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:38:32.922324 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.922300    4544 scope.go:122] "RemoveContainer" containerID="67f408b3ed9734ca22c1321171a4edb0cb40399d8e76a9cffe9c96bf93cb9bc2"
Oct 30 13:38:32.922701 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:38:32.922677    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67f408b3ed9734ca22c1321171a4edb0cb40399d8e76a9cffe9c96bf93cb9bc2\": container with ID starting with 67f408b3ed9734ca22c1321171a4edb0cb40399d8e76a9cffe9c96bf93cb9bc2 not found: ID does not exist" containerID="67f408b3ed9734ca22c1321171a4edb0cb40399d8e76a9cffe9c96bf93cb9bc2"
Oct 30 13:38:32.922776 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.922709    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67f408b3ed9734ca22c1321171a4edb0cb40399d8e76a9cffe9c96bf93cb9bc2"} err="failed to get container status \"67f408b3ed9734ca22c1321171a4edb0cb40399d8e76a9cffe9c96bf93cb9bc2\": rpc error: code = NotFound desc = could not find container \"67f408b3ed9734ca22c1321171a4edb0cb40399d8e76a9cffe9c96bf93cb9bc2\": container with ID starting with 67f408b3ed9734ca22c1321171a4edb0cb40399d8e76a9cffe9c96bf93cb9bc2 not found: ID does not exist"
Oct 30 13:38:32.922776 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.922726    4544 scope.go:122] "RemoveContainer" containerID="1b5b1864e485d3ba2317e500e7781d058ddcd9712fb6a5eaaf154303bec16dd3"
Oct 30 13:38:32.923058 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:38:32.923024    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b5b1864e485d3ba2317e500e7781d058ddcd9712fb6a5eaaf154303bec16dd3\": container with ID starting with 1b5b1864e485d3ba2317e500e7781d058ddcd9712fb6a5eaaf154303bec16dd3 not found: ID does not exist" containerID="1b5b1864e485d3ba2317e500e7781d058ddcd9712fb6a5eaaf154303bec16dd3"
Oct 30 13:38:32.923138 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.923068    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b5b1864e485d3ba2317e500e7781d058ddcd9712fb6a5eaaf154303bec16dd3"} err="failed to get container status \"1b5b1864e485d3ba2317e500e7781d058ddcd9712fb6a5eaaf154303bec16dd3\": rpc error: code = NotFound desc = could not find container \"1b5b1864e485d3ba2317e500e7781d058ddcd9712fb6a5eaaf154303bec16dd3\": container with ID starting with 1b5b1864e485d3ba2317e500e7781d058ddcd9712fb6a5eaaf154303bec16dd3 not found: ID does not exist"
Oct 30 13:38:32.998785 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.998702    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:38:32.998942 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:32.998785    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7" podUID="820333af-678e-40c7-a1b7-24cab9591ea2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:38:33.016474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:33.016431    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68fa1f90-25f8-442f-96ff-12e6a716db84-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:33.016474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:33.016463    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68fa1f90-25f8-442f-96ff-12e6a716db84-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:33.016474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:33.016474    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nnlx\" (UniqueName: \"kubernetes.io/projected/68fa1f90-25f8-442f-96ff-12e6a716db84-kube-api-access-5nnlx\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:33.016723 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:33.016484    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68fa1f90-25f8-442f-96ff-12e6a716db84-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:33.016723 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:33.016493    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68fa1f90-25f8-442f-96ff-12e6a716db84-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:33.247511 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:33.247473    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"]
Oct 30 13:38:33.253311 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:33.253210    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt"]
Oct 30 13:38:34.424272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:34.423271    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt"]
Oct 30 13:38:34.424272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:34.423981    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt" podUID="8732b808-436c-4bf7-868e-1e934515e700" containerName="maas-api" containerID="cri-o://01d0ea9d4c967ad51d1bb50e9a9ece0c8450b987ff26030d476e0360fe9e8455" gracePeriod=30
Oct 30 13:38:34.526975 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:34.526935    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:34.537908 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:34.537874    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg"]
Oct 30 13:38:34.718382 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:34.712980    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:34.764212 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:34.764179    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt"
Oct 30 13:38:34.846086 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:34.846055    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8732b808-436c-4bf7-868e-1e934515e700-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8732b808-436c-4bf7-868e-1e934515e700-maas-api-tls\") pod \"8732b808-436c-4bf7-868e-1e934515e700\" (UID: \"8732b808-436c-4bf7-868e-1e934515e700\") "
Oct 30 13:38:34.846086 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:34.846092    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8732b808-436c-4bf7-868e-1e934515e700-kube-api-access-547rh\" (UniqueName: \"kubernetes.io/projected/8732b808-436c-4bf7-868e-1e934515e700-kube-api-access-547rh\") pod \"8732b808-436c-4bf7-868e-1e934515e700\" (UID: \"8732b808-436c-4bf7-868e-1e934515e700\") "
Oct 30 13:38:34.846409 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:34.846167    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8732b808-436c-4bf7-868e-1e934515e700-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8732b808-436c-4bf7-868e-1e934515e700-openshift-service-ca\") pod \"8732b808-436c-4bf7-868e-1e934515e700\" (UID: \"8732b808-436c-4bf7-868e-1e934515e700\") "
Oct 30 13:38:34.846513 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:34.846488    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8732b808-436c-4bf7-868e-1e934515e700-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8732b808-436c-4bf7-868e-1e934515e700-metrics-certs\") pod \"8732b808-436c-4bf7-868e-1e934515e700\" (UID: \"8732b808-436c-4bf7-868e-1e934515e700\") "
Oct 30 13:38:34.846745 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:34.846713    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8732b808-436c-4bf7-868e-1e934515e700-openshift-service-ca" pod "8732b808-436c-4bf7-868e-1e934515e700" (UID: "8732b808-436c-4bf7-868e-1e934515e700"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:38:34.847394 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:34.847371    4544 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8732b808-436c-4bf7-868e-1e934515e700-openshift-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:34.848926 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:34.848902    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8732b808-436c-4bf7-868e-1e934515e700-maas-api-tls" pod "8732b808-436c-4bf7-868e-1e934515e700" (UID: "8732b808-436c-4bf7-868e-1e934515e700"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:38:34.849020 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:34.848987    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8732b808-436c-4bf7-868e-1e934515e700-kube-api-access-547rh" pod "8732b808-436c-4bf7-868e-1e934515e700" (UID: "8732b808-436c-4bf7-868e-1e934515e700"). InnerVolumeSpecName "kube-api-access-547rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:38:34.849174 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:34.849156    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8732b808-436c-4bf7-868e-1e934515e700-metrics-certs" pod "8732b808-436c-4bf7-868e-1e934515e700" (UID: "8732b808-436c-4bf7-868e-1e934515e700"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:38:34.886635 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:34.886598    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:34.921348 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:34.921307    4544 generic.go:356] "Generic (PLEG): container finished" podID="8732b808-436c-4bf7-868e-1e934515e700" containerID="01d0ea9d4c967ad51d1bb50e9a9ece0c8450b987ff26030d476e0360fe9e8455" exitCode=0
Oct 30 13:38:34.921547 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:34.921368    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt"
Oct 30 13:38:34.921547 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:34.921384    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt" event={"ID":"8732b808-436c-4bf7-868e-1e934515e700","Type":"ContainerDied","Data":"01d0ea9d4c967ad51d1bb50e9a9ece0c8450b987ff26030d476e0360fe9e8455"}
Oct 30 13:38:34.921547 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:34.921445    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt" event={"ID":"8732b808-436c-4bf7-868e-1e934515e700","Type":"ContainerDied","Data":"b9b6f622c1d93e57686ad6841c3c5a46ca258b8bec5a2cc8c4f486208f9f1eb6"}
Oct 30 13:38:34.921547 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:34.921469    4544 scope.go:122] "RemoveContainer" containerID="01d0ea9d4c967ad51d1bb50e9a9ece0c8450b987ff26030d476e0360fe9e8455"
Oct 30 13:38:34.937561 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:34.937540    4544 scope.go:122] "RemoveContainer" containerID="01d0ea9d4c967ad51d1bb50e9a9ece0c8450b987ff26030d476e0360fe9e8455"
Oct 30 13:38:34.938082 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:38:34.938055    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01d0ea9d4c967ad51d1bb50e9a9ece0c8450b987ff26030d476e0360fe9e8455\": container with ID starting with 01d0ea9d4c967ad51d1bb50e9a9ece0c8450b987ff26030d476e0360fe9e8455 not found: ID does not exist" containerID="01d0ea9d4c967ad51d1bb50e9a9ece0c8450b987ff26030d476e0360fe9e8455"
Oct 30 13:38:34.938185 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:34.938092    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01d0ea9d4c967ad51d1bb50e9a9ece0c8450b987ff26030d476e0360fe9e8455"} err="failed to get container status \"01d0ea9d4c967ad51d1bb50e9a9ece0c8450b987ff26030d476e0360fe9e8455\": rpc error: code = NotFound desc = could not find container \"01d0ea9d4c967ad51d1bb50e9a9ece0c8450b987ff26030d476e0360fe9e8455\": container with ID starting with 01d0ea9d4c967ad51d1bb50e9a9ece0c8450b987ff26030d476e0360fe9e8455 not found: ID does not exist"
Oct 30 13:38:34.962261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:34.959817    4544 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8732b808-436c-4bf7-868e-1e934515e700-metrics-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:34.962261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:34.959860    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8732b808-436c-4bf7-868e-1e934515e700-maas-api-tls\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:34.962261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:34.959874    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-547rh\" (UniqueName: \"kubernetes.io/projected/8732b808-436c-4bf7-868e-1e934515e700-kube-api-access-547rh\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:35.206373 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:35.206131    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68fa1f90-25f8-442f-96ff-12e6a716db84" path="/var/lib/kubelet/pods/68fa1f90-25f8-442f-96ff-12e6a716db84/volumes"
Oct 30 13:38:35.208958 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:35.208930    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt"]
Oct 30 13:38:35.209169 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:35.209157    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-78543b-68cc465d-r4gjt"]
Oct 30 13:38:35.593826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:35.593786    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:36.179776 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:36.179745    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:36.324053 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:36.324015    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:36.817874 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:36.817838    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:37.089759 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.089719    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8732b808-436c-4bf7-868e-1e934515e700" path="/var/lib/kubelet/pods/8732b808-436c-4bf7-868e-1e934515e700/volumes"
Oct 30 13:38:37.379940 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.379913    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:38:37.506721 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.506684    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-credential-socket\") pod \"820333af-678e-40c7-a1b7-24cab9591ea2\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") "
Oct 30 13:38:37.506938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.506790    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/820333af-678e-40c7-a1b7-24cab9591ea2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/820333af-678e-40c7-a1b7-24cab9591ea2-istiod-ca-cert\") pod \"820333af-678e-40c7-a1b7-24cab9591ea2\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") "
Oct 30 13:38:37.506938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.506854    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-istio-data\") pod \"820333af-678e-40c7-a1b7-24cab9591ea2\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") "
Oct 30 13:38:37.506938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.506914    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-workload-socket\") pod \"820333af-678e-40c7-a1b7-24cab9591ea2\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") "
Oct 30 13:38:37.507143 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.506977    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-credential-socket" pod "820333af-678e-40c7-a1b7-24cab9591ea2" (UID: "820333af-678e-40c7-a1b7-24cab9591ea2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:38:37.507143 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.507008    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-istio-envoy\") pod \"820333af-678e-40c7-a1b7-24cab9591ea2\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") "
Oct 30 13:38:37.507143 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.507049    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/820333af-678e-40c7-a1b7-24cab9591ea2-kube-api-access-vhzhv\" (UniqueName: \"kubernetes.io/projected/820333af-678e-40c7-a1b7-24cab9591ea2-kube-api-access-vhzhv\") pod \"820333af-678e-40c7-a1b7-24cab9591ea2\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") "
Oct 30 13:38:37.507143 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.507108    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/820333af-678e-40c7-a1b7-24cab9591ea2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/820333af-678e-40c7-a1b7-24cab9591ea2-istio-podinfo\") pod \"820333af-678e-40c7-a1b7-24cab9591ea2\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") "
Oct 30 13:38:37.507430 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.507202    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-istio-data" pod "820333af-678e-40c7-a1b7-24cab9591ea2" (UID: "820333af-678e-40c7-a1b7-24cab9591ea2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:38:37.507430 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.507233    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-workload-certs\") pod \"820333af-678e-40c7-a1b7-24cab9591ea2\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") "
Oct 30 13:38:37.507430 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.507328    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/820333af-678e-40c7-a1b7-24cab9591ea2-istiod-ca-cert" pod "820333af-678e-40c7-a1b7-24cab9591ea2" (UID: "820333af-678e-40c7-a1b7-24cab9591ea2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:38:37.507430 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.507352    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/820333af-678e-40c7-a1b7-24cab9591ea2-istio-token\" (UniqueName: \"kubernetes.io/projected/820333af-678e-40c7-a1b7-24cab9591ea2-istio-token\") pod \"820333af-678e-40c7-a1b7-24cab9591ea2\" (UID: \"820333af-678e-40c7-a1b7-24cab9591ea2\") "
Oct 30 13:38:37.507652 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.507626    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-workload-certs" pod "820333af-678e-40c7-a1b7-24cab9591ea2" (UID: "820333af-678e-40c7-a1b7-24cab9591ea2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:38:37.507780 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.507643    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-workload-socket" pod "820333af-678e-40c7-a1b7-24cab9591ea2" (UID: "820333af-678e-40c7-a1b7-24cab9591ea2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:38:37.508205 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.508177    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:37.508327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.508209    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:37.508327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.508239    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/820333af-678e-40c7-a1b7-24cab9591ea2-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:37.508327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.508253    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:37.508327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.508266    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:37.510031 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.510004    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/820333af-678e-40c7-a1b7-24cab9591ea2-istio-token" pod "820333af-678e-40c7-a1b7-24cab9591ea2" (UID: "820333af-678e-40c7-a1b7-24cab9591ea2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:38:37.510138 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.510116    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/820333af-678e-40c7-a1b7-24cab9591ea2-kube-api-access-vhzhv" pod "820333af-678e-40c7-a1b7-24cab9591ea2" (UID: "820333af-678e-40c7-a1b7-24cab9591ea2"). InnerVolumeSpecName "kube-api-access-vhzhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:38:37.510189 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.510135    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/820333af-678e-40c7-a1b7-24cab9591ea2-istio-podinfo" pod "820333af-678e-40c7-a1b7-24cab9591ea2" (UID: "820333af-678e-40c7-a1b7-24cab9591ea2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:38:37.510463 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.510444    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-istio-envoy" pod "820333af-678e-40c7-a1b7-24cab9591ea2" (UID: "820333af-678e-40c7-a1b7-24cab9591ea2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:38:37.609558 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.609474    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/820333af-678e-40c7-a1b7-24cab9591ea2-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:37.609558 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.609509    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhzhv\" (UniqueName: \"kubernetes.io/projected/820333af-678e-40c7-a1b7-24cab9591ea2-kube-api-access-vhzhv\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:37.609558 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.609519    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/820333af-678e-40c7-a1b7-24cab9591ea2-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:37.609558 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.609529    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/820333af-678e-40c7-a1b7-24cab9591ea2-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:37.839163 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.839128    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:37.951842 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.951755    4544 generic.go:356] "Generic (PLEG): container finished" podID="820333af-678e-40c7-a1b7-24cab9591ea2" containerID="c705b45706084fb81e7b52018a64159c3b77147b07870ef42a989b8cf6e09fd2" exitCode=0
Oct 30 13:38:37.951842 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.951809    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"
Oct 30 13:38:37.952066 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.951834    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7" event={"ID":"820333af-678e-40c7-a1b7-24cab9591ea2","Type":"ContainerDied","Data":"c705b45706084fb81e7b52018a64159c3b77147b07870ef42a989b8cf6e09fd2"}
Oct 30 13:38:37.952066 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.951889    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7" event={"ID":"820333af-678e-40c7-a1b7-24cab9591ea2","Type":"ContainerDied","Data":"22fffec9aab71f0764f5f003683a7a98eae5a03441d5469b21a96dff657ea312"}
Oct 30 13:38:37.952066 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.951915    4544 scope.go:122] "RemoveContainer" containerID="c705b45706084fb81e7b52018a64159c3b77147b07870ef42a989b8cf6e09fd2"
Oct 30 13:38:37.965738 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.965718    4544 scope.go:122] "RemoveContainer" containerID="2c60a75236026d1d1f0f915292164c351adeebb4e0101cae02351b5b402b083f"
Oct 30 13:38:37.976890 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.976863    4544 scope.go:122] "RemoveContainer" containerID="c705b45706084fb81e7b52018a64159c3b77147b07870ef42a989b8cf6e09fd2"
Oct 30 13:38:37.977279 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:38:37.977253    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c705b45706084fb81e7b52018a64159c3b77147b07870ef42a989b8cf6e09fd2\": container with ID starting with c705b45706084fb81e7b52018a64159c3b77147b07870ef42a989b8cf6e09fd2 not found: ID does not exist" containerID="c705b45706084fb81e7b52018a64159c3b77147b07870ef42a989b8cf6e09fd2"
Oct 30 13:38:37.977408 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.977293    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c705b45706084fb81e7b52018a64159c3b77147b07870ef42a989b8cf6e09fd2"} err="failed to get container status \"c705b45706084fb81e7b52018a64159c3b77147b07870ef42a989b8cf6e09fd2\": rpc error: code = NotFound desc = could not find container \"c705b45706084fb81e7b52018a64159c3b77147b07870ef42a989b8cf6e09fd2\": container with ID starting with c705b45706084fb81e7b52018a64159c3b77147b07870ef42a989b8cf6e09fd2 not found: ID does not exist"
Oct 30 13:38:37.977408 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.977319    4544 scope.go:122] "RemoveContainer" containerID="2c60a75236026d1d1f0f915292164c351adeebb4e0101cae02351b5b402b083f"
Oct 30 13:38:37.977685 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:38:37.977658    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c60a75236026d1d1f0f915292164c351adeebb4e0101cae02351b5b402b083f\": container with ID starting with 2c60a75236026d1d1f0f915292164c351adeebb4e0101cae02351b5b402b083f not found: ID does not exist" containerID="2c60a75236026d1d1f0f915292164c351adeebb4e0101cae02351b5b402b083f"
Oct 30 13:38:37.977766 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:37.977694    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c60a75236026d1d1f0f915292164c351adeebb4e0101cae02351b5b402b083f"} err="failed to get container status \"2c60a75236026d1d1f0f915292164c351adeebb4e0101cae02351b5b402b083f\": rpc error: code = NotFound desc = could not find container \"2c60a75236026d1d1f0f915292164c351adeebb4e0101cae02351b5b402b083f\": container with ID starting with 2c60a75236026d1d1f0f915292164c351adeebb4e0101cae02351b5b402b083f not found: ID does not exist"
Oct 30 13:38:38.011072 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:38.011039    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"]
Oct 30 13:38:38.021929 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:38.021899    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7"]
Oct 30 13:38:39.090530 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:39.090495    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="820333af-678e-40c7-a1b7-24cab9591ea2" path="/var/lib/kubelet/pods/820333af-678e-40c7-a1b7-24cab9591ea2/volumes"
Oct 30 13:38:39.538521 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:39.538390    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg" podUID="45260af2-724d-4065-b966-542af12813a6" containerName="payload-pre-processing" containerID="cri-o://915424c5beba64e95cb93c60770d6a155b8792482cba716a18ecfe771f1cabdf" gracePeriod=25
Oct 30 13:38:39.844853 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:39.844823    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg"
Oct 30 13:38:39.934599 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:39.934552    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:39.952998 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:39.952965    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45260af2-724d-4065-b966-542af12813a6-kube-api-access-bgjxn\" (UniqueName: \"kubernetes.io/projected/45260af2-724d-4065-b966-542af12813a6-kube-api-access-bgjxn\") pod \"45260af2-724d-4065-b966-542af12813a6\" (UID: \"45260af2-724d-4065-b966-542af12813a6\") "
Oct 30 13:38:39.953178 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:39.953010    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45260af2-724d-4065-b966-542af12813a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45260af2-724d-4065-b966-542af12813a6-plugins-config-volume\") pod \"45260af2-724d-4065-b966-542af12813a6\" (UID: \"45260af2-724d-4065-b966-542af12813a6\") "
Oct 30 13:38:39.953442 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:39.953420    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45260af2-724d-4065-b966-542af12813a6-plugins-config-volume" pod "45260af2-724d-4065-b966-542af12813a6" (UID: "45260af2-724d-4065-b966-542af12813a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:38:39.953921 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:39.953900    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45260af2-724d-4065-b966-542af12813a6-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:39.955199 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:39.955179    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45260af2-724d-4065-b966-542af12813a6-kube-api-access-bgjxn" pod "45260af2-724d-4065-b966-542af12813a6" (UID: "45260af2-724d-4065-b966-542af12813a6"). InnerVolumeSpecName "kube-api-access-bgjxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:38:40.027019 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:40.026980    4544 generic.go:356] "Generic (PLEG): container finished" podID="45260af2-724d-4065-b966-542af12813a6" containerID="915424c5beba64e95cb93c60770d6a155b8792482cba716a18ecfe771f1cabdf" exitCode=0
Oct 30 13:38:40.027390 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:40.027352    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg" event={"ID":"45260af2-724d-4065-b966-542af12813a6","Type":"ContainerDied","Data":"915424c5beba64e95cb93c60770d6a155b8792482cba716a18ecfe771f1cabdf"}
Oct 30 13:38:40.027390 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:40.027377    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg"
Oct 30 13:38:40.027568 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:40.027444    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg" event={"ID":"45260af2-724d-4065-b966-542af12813a6","Type":"ContainerDied","Data":"1c30d580a00f0f6b3ef63c294e0bbf6e3222d9a8213c424b8353d666daf86495"}
Oct 30 13:38:40.027568 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:40.027471    4544 scope.go:122] "RemoveContainer" containerID="915424c5beba64e95cb93c60770d6a155b8792482cba716a18ecfe771f1cabdf"
Oct 30 13:38:40.043371 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:40.043347    4544 scope.go:122] "RemoveContainer" containerID="915424c5beba64e95cb93c60770d6a155b8792482cba716a18ecfe771f1cabdf"
Oct 30 13:38:40.043669 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:38:40.043645    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"915424c5beba64e95cb93c60770d6a155b8792482cba716a18ecfe771f1cabdf\": container with ID starting with 915424c5beba64e95cb93c60770d6a155b8792482cba716a18ecfe771f1cabdf not found: ID does not exist" containerID="915424c5beba64e95cb93c60770d6a155b8792482cba716a18ecfe771f1cabdf"
Oct 30 13:38:40.043718 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:40.043679    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"915424c5beba64e95cb93c60770d6a155b8792482cba716a18ecfe771f1cabdf"} err="failed to get container status \"915424c5beba64e95cb93c60770d6a155b8792482cba716a18ecfe771f1cabdf\": rpc error: code = NotFound desc = could not find container \"915424c5beba64e95cb93c60770d6a155b8792482cba716a18ecfe771f1cabdf\": container with ID starting with 915424c5beba64e95cb93c60770d6a155b8792482cba716a18ecfe771f1cabdf not found: ID does not exist"
Oct 30 13:38:40.055491 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:40.055456    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgjxn\" (UniqueName: \"kubernetes.io/projected/45260af2-724d-4065-b966-542af12813a6-kube-api-access-bgjxn\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:40.073602 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:40.073569    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4"]
Oct 30 13:38:40.083879 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:40.083838    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg"]
Oct 30 13:38:40.089538 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:40.089509    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg"]
Oct 30 13:38:40.106860 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:40.106822    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl"]
Oct 30 13:38:40.222426 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:40.222378    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:40.360303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:40.349374    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:41.090695 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:41.090658    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45260af2-724d-4065-b966-542af12813a6" path="/var/lib/kubelet/pods/45260af2-724d-4065-b966-542af12813a6/volumes"
Oct 30 13:38:42.062683 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:42.062568    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:43.146842 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:43.146794    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:43.355488 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:43.355452    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:43.519346 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:43.519211    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:44.049852 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:44.049818    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:45.074242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:45.074173    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4" podUID="aba97076-557c-4b61-9c1a-374341fbe152" containerName="payload-processing" containerID="cri-o://447e5c02621aded4a7bf2b3daabe0916be0cf13244f6150dcc39e435693966c9" gracePeriod=25
Oct 30 13:38:45.107775 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:45.107700    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl" podUID="d232746e-9562-4578-b95f-09895eece465" containerName="payload-pre-processing" containerID="cri-o://3f363054c1f041fb06b97602f9739316f56be031b99aa81eebe5c6aa2f58d2dc" gracePeriod=25
Oct 30 13:38:45.422470 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:45.422438    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4"
Oct 30 13:38:45.476387 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:45.475946    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl"
Oct 30 13:38:45.561941 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:45.561897    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d232746e-9562-4578-b95f-09895eece465-kube-api-access-67zv4\" (UniqueName: \"kubernetes.io/projected/d232746e-9562-4578-b95f-09895eece465-kube-api-access-67zv4\") pod \"d232746e-9562-4578-b95f-09895eece465\" (UID: \"d232746e-9562-4578-b95f-09895eece465\") "
Oct 30 13:38:45.562197 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:45.562165    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aba97076-557c-4b61-9c1a-374341fbe152-kube-api-access-nhjz8\" (UniqueName: \"kubernetes.io/projected/aba97076-557c-4b61-9c1a-374341fbe152-kube-api-access-nhjz8\") pod \"aba97076-557c-4b61-9c1a-374341fbe152\" (UID: \"aba97076-557c-4b61-9c1a-374341fbe152\") "
Oct 30 13:38:45.562388 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:45.562255    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d232746e-9562-4578-b95f-09895eece465-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d232746e-9562-4578-b95f-09895eece465-plugins-config-volume\") pod \"d232746e-9562-4578-b95f-09895eece465\" (UID: \"d232746e-9562-4578-b95f-09895eece465\") "
Oct 30 13:38:45.562510 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:45.562491    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aba97076-557c-4b61-9c1a-374341fbe152-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aba97076-557c-4b61-9c1a-374341fbe152-plugins-config-volume\") pod \"aba97076-557c-4b61-9c1a-374341fbe152\" (UID: \"aba97076-557c-4b61-9c1a-374341fbe152\") "
Oct 30 13:38:45.562703 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:45.562666    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d232746e-9562-4578-b95f-09895eece465-plugins-config-volume" pod "d232746e-9562-4578-b95f-09895eece465" (UID: "d232746e-9562-4578-b95f-09895eece465"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:38:45.563215 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:45.563186    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aba97076-557c-4b61-9c1a-374341fbe152-plugins-config-volume" pod "aba97076-557c-4b61-9c1a-374341fbe152" (UID: "aba97076-557c-4b61-9c1a-374341fbe152"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:38:45.563428 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:45.563407    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aba97076-557c-4b61-9c1a-374341fbe152-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:45.563474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:45.563436    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d232746e-9562-4578-b95f-09895eece465-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:45.564859 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:45.564829    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d232746e-9562-4578-b95f-09895eece465-kube-api-access-67zv4" pod "d232746e-9562-4578-b95f-09895eece465" (UID: "d232746e-9562-4578-b95f-09895eece465"). InnerVolumeSpecName "kube-api-access-67zv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:38:45.565364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:45.565335    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba97076-557c-4b61-9c1a-374341fbe152-kube-api-access-nhjz8" pod "aba97076-557c-4b61-9c1a-374341fbe152" (UID: "aba97076-557c-4b61-9c1a-374341fbe152"). InnerVolumeSpecName "kube-api-access-nhjz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:38:45.641632 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:45.641540    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:45.665786 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:45.665728    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67zv4\" (UniqueName: \"kubernetes.io/projected/d232746e-9562-4578-b95f-09895eece465-kube-api-access-67zv4\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:45.665786 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:45.665788    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhjz8\" (UniqueName: \"kubernetes.io/projected/aba97076-557c-4b61-9c1a-374341fbe152-kube-api-access-nhjz8\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:46.249247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:46.238862    4544 generic.go:356] "Generic (PLEG): container finished" podID="aba97076-557c-4b61-9c1a-374341fbe152" containerID="447e5c02621aded4a7bf2b3daabe0916be0cf13244f6150dcc39e435693966c9" exitCode=0
Oct 30 13:38:46.249247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:46.239089    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4" event={"ID":"aba97076-557c-4b61-9c1a-374341fbe152","Type":"ContainerDied","Data":"447e5c02621aded4a7bf2b3daabe0916be0cf13244f6150dcc39e435693966c9"}
Oct 30 13:38:46.249247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:46.239148    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4" event={"ID":"aba97076-557c-4b61-9c1a-374341fbe152","Type":"ContainerDied","Data":"ae7adad940036cc1aed8f095cf489b301f04a2fa7c7f393e4185ca5657433088"}
Oct 30 13:38:46.249247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:46.239180    4544 scope.go:122] "RemoveContainer" containerID="447e5c02621aded4a7bf2b3daabe0916be0cf13244f6150dcc39e435693966c9"
Oct 30 13:38:46.249247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:46.239405    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4"
Oct 30 13:38:46.266560 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:46.264844    4544 generic.go:356] "Generic (PLEG): container finished" podID="d232746e-9562-4578-b95f-09895eece465" containerID="3f363054c1f041fb06b97602f9739316f56be031b99aa81eebe5c6aa2f58d2dc" exitCode=0
Oct 30 13:38:46.266560 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:46.265044    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl" event={"ID":"d232746e-9562-4578-b95f-09895eece465","Type":"ContainerDied","Data":"3f363054c1f041fb06b97602f9739316f56be031b99aa81eebe5c6aa2f58d2dc"}
Oct 30 13:38:46.266560 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:46.265118    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl" event={"ID":"d232746e-9562-4578-b95f-09895eece465","Type":"ContainerDied","Data":"6e58582a32cf08261188d2a5af5654273131a3b9fdc017aab32615b246907054"}
Oct 30 13:38:46.266560 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:46.265241    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl"
Oct 30 13:38:46.291181 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:46.291035    4544 scope.go:122] "RemoveContainer" containerID="447e5c02621aded4a7bf2b3daabe0916be0cf13244f6150dcc39e435693966c9"
Oct 30 13:38:46.291839 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:38:46.291669    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"447e5c02621aded4a7bf2b3daabe0916be0cf13244f6150dcc39e435693966c9\": container with ID starting with 447e5c02621aded4a7bf2b3daabe0916be0cf13244f6150dcc39e435693966c9 not found: ID does not exist" containerID="447e5c02621aded4a7bf2b3daabe0916be0cf13244f6150dcc39e435693966c9"
Oct 30 13:38:46.291839 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:46.291709    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"447e5c02621aded4a7bf2b3daabe0916be0cf13244f6150dcc39e435693966c9"} err="failed to get container status \"447e5c02621aded4a7bf2b3daabe0916be0cf13244f6150dcc39e435693966c9\": rpc error: code = NotFound desc = could not find container \"447e5c02621aded4a7bf2b3daabe0916be0cf13244f6150dcc39e435693966c9\": container with ID starting with 447e5c02621aded4a7bf2b3daabe0916be0cf13244f6150dcc39e435693966c9 not found: ID does not exist"
Oct 30 13:38:46.291839 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:46.291736    4544 scope.go:122] "RemoveContainer" containerID="3f363054c1f041fb06b97602f9739316f56be031b99aa81eebe5c6aa2f58d2dc"
Oct 30 13:38:46.316210 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:46.315827    4544 scope.go:122] "RemoveContainer" containerID="3f363054c1f041fb06b97602f9739316f56be031b99aa81eebe5c6aa2f58d2dc"
Oct 30 13:38:46.317334 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:38:46.317167    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f363054c1f041fb06b97602f9739316f56be031b99aa81eebe5c6aa2f58d2dc\": container with ID starting with 3f363054c1f041fb06b97602f9739316f56be031b99aa81eebe5c6aa2f58d2dc not found: ID does not exist" containerID="3f363054c1f041fb06b97602f9739316f56be031b99aa81eebe5c6aa2f58d2dc"
Oct 30 13:38:46.317562 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:46.317454    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f363054c1f041fb06b97602f9739316f56be031b99aa81eebe5c6aa2f58d2dc"} err="failed to get container status \"3f363054c1f041fb06b97602f9739316f56be031b99aa81eebe5c6aa2f58d2dc\": rpc error: code = NotFound desc = could not find container \"3f363054c1f041fb06b97602f9739316f56be031b99aa81eebe5c6aa2f58d2dc\": container with ID starting with 3f363054c1f041fb06b97602f9739316f56be031b99aa81eebe5c6aa2f58d2dc not found: ID does not exist"
Oct 30 13:38:46.382328 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:46.380510    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4"]
Oct 30 13:38:46.391375 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:46.391333    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4"]
Oct 30 13:38:46.441755 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:46.441717    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl"]
Oct 30 13:38:46.464123 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:46.464084    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl"]
Oct 30 13:38:46.755482 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:46.755395    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:46.878235 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:46.878184    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:46.998205 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:46.998166    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:47.089628 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:47.089595    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aba97076-557c-4b61-9c1a-374341fbe152" path="/var/lib/kubelet/pods/aba97076-557c-4b61-9c1a-374341fbe152/volumes"
Oct 30 13:38:47.090108 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:47.090093    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d232746e-9562-4578-b95f-09895eece465" path="/var/lib/kubelet/pods/d232746e-9562-4578-b95f-09895eece465/volumes"
Oct 30 13:38:47.488275 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:47.488150    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:48.981452 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:48.981414    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:50.718205 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:50.718170    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:50.838530 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:50.838475    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck"]
Oct 30 13:38:50.838818 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:50.838787    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck" podUID="7ab2ca89-081f-411d-8ec4-20a8d399010f" containerName="payload-pre-processing" containerID="cri-o://49fb526bc42cf0f223ef2e63d670e46bb2815851dea7a1a1ec4368e6188f8149" gracePeriod=30
Oct 30 13:38:50.908849 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:50.908801    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:51.041185 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:51.041083    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:51.147435 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:51.147404    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck"
Oct 30 13:38:51.187755 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:51.187721    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ab2ca89-081f-411d-8ec4-20a8d399010f-kube-api-access-w59wl\" (UniqueName: \"kubernetes.io/projected/7ab2ca89-081f-411d-8ec4-20a8d399010f-kube-api-access-w59wl\") pod \"7ab2ca89-081f-411d-8ec4-20a8d399010f\" (UID: \"7ab2ca89-081f-411d-8ec4-20a8d399010f\") "
Oct 30 13:38:51.187938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:51.187885    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ab2ca89-081f-411d-8ec4-20a8d399010f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab2ca89-081f-411d-8ec4-20a8d399010f-plugins-config-volume\") pod \"7ab2ca89-081f-411d-8ec4-20a8d399010f\" (UID: \"7ab2ca89-081f-411d-8ec4-20a8d399010f\") "
Oct 30 13:38:51.188372 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:51.188335    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ab2ca89-081f-411d-8ec4-20a8d399010f-plugins-config-volume" pod "7ab2ca89-081f-411d-8ec4-20a8d399010f" (UID: "7ab2ca89-081f-411d-8ec4-20a8d399010f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:38:51.188602 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:51.188582    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab2ca89-081f-411d-8ec4-20a8d399010f-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:51.191123 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:51.191092    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab2ca89-081f-411d-8ec4-20a8d399010f-kube-api-access-w59wl" pod "7ab2ca89-081f-411d-8ec4-20a8d399010f" (UID: "7ab2ca89-081f-411d-8ec4-20a8d399010f"). InnerVolumeSpecName "kube-api-access-w59wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:38:51.300300 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:51.291929    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w59wl\" (UniqueName: \"kubernetes.io/projected/7ab2ca89-081f-411d-8ec4-20a8d399010f-kube-api-access-w59wl\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:38:51.313255 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:51.302841    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:51.480533 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:51.473293    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:51.486244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:51.480977    4544 generic.go:356] "Generic (PLEG): container finished" podID="7ab2ca89-081f-411d-8ec4-20a8d399010f" containerID="49fb526bc42cf0f223ef2e63d670e46bb2815851dea7a1a1ec4368e6188f8149" exitCode=0
Oct 30 13:38:51.486244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:51.481096    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck" event={"ID":"7ab2ca89-081f-411d-8ec4-20a8d399010f","Type":"ContainerDied","Data":"49fb526bc42cf0f223ef2e63d670e46bb2815851dea7a1a1ec4368e6188f8149"}
Oct 30 13:38:51.486244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:51.481171    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck" event={"ID":"7ab2ca89-081f-411d-8ec4-20a8d399010f","Type":"ContainerDied","Data":"6f2f1e069afaca79e82def92eea22a4da59231aced4f55c2b276579a4f12af63"}
Oct 30 13:38:51.486244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:51.481206    4544 scope.go:122] "RemoveContainer" containerID="49fb526bc42cf0f223ef2e63d670e46bb2815851dea7a1a1ec4368e6188f8149"
Oct 30 13:38:51.486244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:51.481445    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck"
Oct 30 13:38:51.583433 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:51.573400    4544 scope.go:122] "RemoveContainer" containerID="49fb526bc42cf0f223ef2e63d670e46bb2815851dea7a1a1ec4368e6188f8149"
Oct 30 13:38:51.583433 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:38:51.581362    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49fb526bc42cf0f223ef2e63d670e46bb2815851dea7a1a1ec4368e6188f8149\": container with ID starting with 49fb526bc42cf0f223ef2e63d670e46bb2815851dea7a1a1ec4368e6188f8149 not found: ID does not exist" containerID="49fb526bc42cf0f223ef2e63d670e46bb2815851dea7a1a1ec4368e6188f8149"
Oct 30 13:38:51.583433 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:51.581413    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49fb526bc42cf0f223ef2e63d670e46bb2815851dea7a1a1ec4368e6188f8149"} err="failed to get container status \"49fb526bc42cf0f223ef2e63d670e46bb2815851dea7a1a1ec4368e6188f8149\": rpc error: code = NotFound desc = could not find container \"49fb526bc42cf0f223ef2e63d670e46bb2815851dea7a1a1ec4368e6188f8149\": container with ID starting with 49fb526bc42cf0f223ef2e63d670e46bb2815851dea7a1a1ec4368e6188f8149 not found: ID does not exist"
Oct 30 13:38:51.608899 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:51.608563    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:51.612869 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:51.612738    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck"]
Oct 30 13:38:51.622471 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:51.622434    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck"]
Oct 30 13:38:51.994755 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:51.994625    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:53.162075 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:53.162035    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ab2ca89-081f-411d-8ec4-20a8d399010f" path="/var/lib/kubelet/pods/7ab2ca89-081f-411d-8ec4-20a8d399010f/volumes"
Oct 30 13:38:54.104531 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:54.104492    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:54.353447 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:54.353408    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:54.499103 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:54.499007    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:54.698882 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:54.698846    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"]
Oct 30 13:38:54.699158 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:54.699126    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn" podUID="ff60abd5-8f12-4e67-99f4-e6baeaee47ce" containerName="istio-proxy" containerID="cri-o://b859c49495aa286dbbb49625bd0e28cdb6f01b56f18f257120889e82a3b6279d" gracePeriod=30
Oct 30 13:38:54.709666 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:54.709617    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"]
Oct 30 13:38:54.709937 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:54.709907    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs" podUID="e6aed883-80e6-44a5-ac18-855437d32862" containerName="istio-proxy" containerID="cri-o://7dcecf6dfb4b226246e8f47b96862623b45f2a3b508bf93b475327ec6305dcf7" gracePeriod=30
Oct 30 13:38:54.714479 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:54.714450    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"]
Oct 30 13:38:54.714707 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:54.714683    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k" podUID="128497c2-9f64-4902-a913-5acb1489b88b" containerName="istio-proxy" containerID="cri-o://ca0c9e30de327cc6d9419739768d5a50e4af39540c170332e3e4222dae75619e" gracePeriod=30
Oct 30 13:38:54.761123 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:54.761023    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"]
Oct 30 13:38:54.761316 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:54.761261    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j" podUID="e3559421-6424-4480-8d49-5ed4a11deb3d" containerName="istio-proxy" containerID="cri-o://a9f7334495eef7f6dc76371fa6eee8c0a454b18686522a401d2a378e2514fa0c" gracePeriod=30
Oct 30 13:38:54.766592 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:54.766563    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"]
Oct 30 13:38:54.766791 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:54.766769    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww" podUID="62db3dba-ac93-4845-9e85-6d83a6fb0cb1" containerName="istio-proxy" containerID="cri-o://93aecdd1f121efade27a99382f811c8b68b4aa47009bbcb61c09ecd3ba3a3096" gracePeriod=30
Oct 30 13:38:54.769647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:54.769615    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:55.020887 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:55.020798    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"]
Oct 30 13:38:55.021034 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:55.021007    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g" podUID="80b08e28-171e-49cd-bafe-0fbf4efc09e3" containerName="istio-proxy" containerID="cri-o://caa31e59de79b72b8cc8b40f021564188634ab8d33cb4ce65747986721379178" gracePeriod=30
Oct 30 13:38:56.103071 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:56.103035    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:56.120014 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:56.119975    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:38:56.120213 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:56.120044    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k" podUID="128497c2-9f64-4902-a913-5acb1489b88b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:38:56.222464 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:56.218149    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:38:56.222464 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:56.218276    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww" podUID="62db3dba-ac93-4845-9e85-6d83a6fb0cb1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:38:56.239253 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:56.236933    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:56.527460 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:56.527357    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:58.169180 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:58.169141    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"]
Oct 30 13:38:58.169711 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:58.169385    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf" podUID="f44f3385-bd49-41d7-9add-718b415008f2" containerName="istio-proxy" containerID="cri-o://1da3ca37de71d501a467fe0530785de9ae1d35ea57150be91997d7199a83df0c" gracePeriod=30
Oct 30 13:38:58.236365 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:58.236314    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:58.524786 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:58.524644    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:58.621488 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:58.621449    4544 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:38:58.621670 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:58.621514    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn" podUID="ff60abd5-8f12-4e67-99f4-e6baeaee47ce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:38:58.663813 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:58.663770    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:58.869810 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:58.869766    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"]
Oct 30 13:38:58.870080 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:58.870052    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r" podUID="7ef74cc1-eb94-4550-af70-38d4f4149963" containerName="istio-proxy" containerID="cri-o://2fab71d50be57d2748266648f6d21cd9f1d1639dd149f9083673f7216b8353b4" gracePeriod=30
Oct 30 13:38:58.893632 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:58.893592    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:59.065245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:59.061607    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:38:59.138331 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:38:59.138182    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:38:59.760089 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:59.760047    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"]
Oct 30 13:38:59.760531 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:38:59.760306    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm" podUID="347e2f12-0089-42aa-86b8-d42d02a56175" containerName="istio-proxy" containerID="cri-o://33153249a7f6d8b9013d6842dd92d84c7c343ae48ab96cfc53ff0eba3e23fca1" gracePeriod=30
Oct 30 13:39:00.162458 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.162425    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:39:00.170125 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.170093    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:39:00.233321 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.233233    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6aed883-80e6-44a5-ac18-855437d32862-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6aed883-80e6-44a5-ac18-855437d32862-istiod-ca-cert\") pod \"e6aed883-80e6-44a5-ac18-855437d32862\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") "
Oct 30 13:39:00.233321 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.233299    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-istio-envoy\") pod \"e6aed883-80e6-44a5-ac18-855437d32862\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") "
Oct 30 13:39:00.233560 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.233347    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-istio-envoy\") pod \"128497c2-9f64-4902-a913-5acb1489b88b\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") "
Oct 30 13:39:00.233560 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.233543    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-workload-socket\") pod \"128497c2-9f64-4902-a913-5acb1489b88b\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") "
Oct 30 13:39:00.233689 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.233591    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-istio-data\") pod \"e6aed883-80e6-44a5-ac18-855437d32862\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") "
Oct 30 13:39:00.233743 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.233668    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/128497c2-9f64-4902-a913-5acb1489b88b-istio-token\" (UniqueName: \"kubernetes.io/projected/128497c2-9f64-4902-a913-5acb1489b88b-istio-token\") pod \"128497c2-9f64-4902-a913-5acb1489b88b\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") "
Oct 30 13:39:00.233810 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.233750    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-credential-socket\") pod \"e6aed883-80e6-44a5-ac18-855437d32862\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") "
Oct 30 13:39:00.233810 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.233801    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-workload-certs\") pod \"128497c2-9f64-4902-a913-5acb1489b88b\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") "
Oct 30 13:39:00.233926 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.233852    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6aed883-80e6-44a5-ac18-855437d32862-istio-token\" (UniqueName: \"kubernetes.io/projected/e6aed883-80e6-44a5-ac18-855437d32862-istio-token\") pod \"e6aed883-80e6-44a5-ac18-855437d32862\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") "
Oct 30 13:39:00.233926 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.233884    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/128497c2-9f64-4902-a913-5acb1489b88b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/128497c2-9f64-4902-a913-5acb1489b88b-istiod-ca-cert\") pod \"128497c2-9f64-4902-a913-5acb1489b88b\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") "
Oct 30 13:39:00.234039 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.233976    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/128497c2-9f64-4902-a913-5acb1489b88b-kube-api-access-6b5gf\" (UniqueName: \"kubernetes.io/projected/128497c2-9f64-4902-a913-5acb1489b88b-kube-api-access-6b5gf\") pod \"128497c2-9f64-4902-a913-5acb1489b88b\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") "
Oct 30 13:39:00.234039 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.234018    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-istio-data\") pod \"128497c2-9f64-4902-a913-5acb1489b88b\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") "
Oct 30 13:39:00.234152 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.234065    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6aed883-80e6-44a5-ac18-855437d32862-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6aed883-80e6-44a5-ac18-855437d32862-istio-podinfo\") pod \"e6aed883-80e6-44a5-ac18-855437d32862\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") "
Oct 30 13:39:00.234152 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.234108    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-workload-certs\") pod \"e6aed883-80e6-44a5-ac18-855437d32862\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") "
Oct 30 13:39:00.234289 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.234162    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/128497c2-9f64-4902-a913-5acb1489b88b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/128497c2-9f64-4902-a913-5acb1489b88b-istio-podinfo\") pod \"128497c2-9f64-4902-a913-5acb1489b88b\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") "
Oct 30 13:39:00.237123 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.234212    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6aed883-80e6-44a5-ac18-855437d32862-kube-api-access-j8mmp\" (UniqueName: \"kubernetes.io/projected/e6aed883-80e6-44a5-ac18-855437d32862-kube-api-access-j8mmp\") pod \"e6aed883-80e6-44a5-ac18-855437d32862\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") "
Oct 30 13:39:00.237123 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.234790    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-credential-socket\") pod \"128497c2-9f64-4902-a913-5acb1489b88b\" (UID: \"128497c2-9f64-4902-a913-5acb1489b88b\") "
Oct 30 13:39:00.237123 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.234910    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-workload-socket\") pod \"e6aed883-80e6-44a5-ac18-855437d32862\" (UID: \"e6aed883-80e6-44a5-ac18-855437d32862\") "
Oct 30 13:39:00.237123 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.235905    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/128497c2-9f64-4902-a913-5acb1489b88b-istiod-ca-cert" pod "128497c2-9f64-4902-a913-5acb1489b88b" (UID: "128497c2-9f64-4902-a913-5acb1489b88b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:00.237123 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.236246    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-workload-certs" pod "128497c2-9f64-4902-a913-5acb1489b88b" (UID: "128497c2-9f64-4902-a913-5acb1489b88b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.237123 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.236646    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6aed883-80e6-44a5-ac18-855437d32862-istiod-ca-cert" pod "e6aed883-80e6-44a5-ac18-855437d32862" (UID: "e6aed883-80e6-44a5-ac18-855437d32862"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:00.237623 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.237156    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-workload-certs" pod "e6aed883-80e6-44a5-ac18-855437d32862" (UID: "e6aed883-80e6-44a5-ac18-855437d32862"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.238136 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.238108    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-istio-data" pod "e6aed883-80e6-44a5-ac18-855437d32862" (UID: "e6aed883-80e6-44a5-ac18-855437d32862"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.238373 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.238351    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-workload-socket" pod "128497c2-9f64-4902-a913-5acb1489b88b" (UID: "128497c2-9f64-4902-a913-5acb1489b88b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.238543 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.238526    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-istio-data" pod "128497c2-9f64-4902-a913-5acb1489b88b" (UID: "128497c2-9f64-4902-a913-5acb1489b88b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.246384 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.238702    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-credential-socket" pod "128497c2-9f64-4902-a913-5acb1489b88b" (UID: "128497c2-9f64-4902-a913-5acb1489b88b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.246384 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.241419    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.246384 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.241448    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6aed883-80e6-44a5-ac18-855437d32862-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.246384 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.241469    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.246384 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.241482    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.246384 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.241496    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.246384 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.241509    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/128497c2-9f64-4902-a913-5acb1489b88b-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.246384 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.241520    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.246384 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.241532    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.246384 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.246033    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-credential-socket" pod "e6aed883-80e6-44a5-ac18-855437d32862" (UID: "e6aed883-80e6-44a5-ac18-855437d32862"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.246384 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.246057    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-workload-socket" pod "e6aed883-80e6-44a5-ac18-855437d32862" (UID: "e6aed883-80e6-44a5-ac18-855437d32862"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.259350 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.259165    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6aed883-80e6-44a5-ac18-855437d32862-istio-token" pod "e6aed883-80e6-44a5-ac18-855437d32862" (UID: "e6aed883-80e6-44a5-ac18-855437d32862"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:00.263951 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.262123    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/128497c2-9f64-4902-a913-5acb1489b88b-istio-podinfo" pod "128497c2-9f64-4902-a913-5acb1489b88b" (UID: "128497c2-9f64-4902-a913-5acb1489b88b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:39:00.263951 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.262280    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-istio-envoy" pod "128497c2-9f64-4902-a913-5acb1489b88b" (UID: "128497c2-9f64-4902-a913-5acb1489b88b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.263951 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.263518    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-istio-envoy" pod "e6aed883-80e6-44a5-ac18-855437d32862" (UID: "e6aed883-80e6-44a5-ac18-855437d32862"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.263951 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.263627    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/128497c2-9f64-4902-a913-5acb1489b88b-istio-token" pod "128497c2-9f64-4902-a913-5acb1489b88b" (UID: "128497c2-9f64-4902-a913-5acb1489b88b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:00.263951 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.263720    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6aed883-80e6-44a5-ac18-855437d32862-istio-podinfo" pod "e6aed883-80e6-44a5-ac18-855437d32862" (UID: "e6aed883-80e6-44a5-ac18-855437d32862"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:39:00.263951 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.263765    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/128497c2-9f64-4902-a913-5acb1489b88b-kube-api-access-6b5gf" pod "128497c2-9f64-4902-a913-5acb1489b88b" (UID: "128497c2-9f64-4902-a913-5acb1489b88b"). InnerVolumeSpecName "kube-api-access-6b5gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:00.282073 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.280607    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6aed883-80e6-44a5-ac18-855437d32862-kube-api-access-j8mmp" pod "e6aed883-80e6-44a5-ac18-855437d32862" (UID: "e6aed883-80e6-44a5-ac18-855437d32862"). InnerVolumeSpecName "kube-api-access-j8mmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:00.282073 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.280714    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:39:00.287772 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.287517    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:39:00.296012 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.295973    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:39:00.345363 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.344351    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-kube-api-access-js2cm\" (UniqueName: \"kubernetes.io/projected/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-kube-api-access-js2cm\") pod \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") "
Oct 30 13:39:00.345363 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.344541    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-workload-certs\") pod \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") "
Oct 30 13:39:00.345363 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.345154    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istio-data\") pod \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") "
Oct 30 13:39:00.345363 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.345195    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-workload-socket\") pod \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") "
Oct 30 13:39:00.345363 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.345286    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-credential-socket\") pod \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") "
Oct 30 13:39:00.345826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.345411    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istiod-ca-cert\") pod \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") "
Oct 30 13:39:00.345826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.345438    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istio-envoy\") pod \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") "
Oct 30 13:39:00.345826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.345491    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istio-token\" (UniqueName: \"kubernetes.io/projected/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istio-token\") pod \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") "
Oct 30 13:39:00.345826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.345520    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istio-podinfo\") pod \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\" (UID: \"ff60abd5-8f12-4e67-99f4-e6baeaee47ce\") "
Oct 30 13:39:00.346985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.346261    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-credential-socket" pod "ff60abd5-8f12-4e67-99f4-e6baeaee47ce" (UID: "ff60abd5-8f12-4e67-99f4-e6baeaee47ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.346985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.346441    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istio-data" pod "ff60abd5-8f12-4e67-99f4-e6baeaee47ce" (UID: "ff60abd5-8f12-4e67-99f4-e6baeaee47ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.346985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.346492    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/128497c2-9f64-4902-a913-5acb1489b88b-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.346985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.346516    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.346985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.346529    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6aed883-80e6-44a5-ac18-855437d32862-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.346985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.346659    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b5gf\" (UniqueName: \"kubernetes.io/projected/128497c2-9f64-4902-a913-5acb1489b88b-kube-api-access-6b5gf\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.346985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.346672    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6aed883-80e6-44a5-ac18-855437d32862-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.346985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.346823    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/128497c2-9f64-4902-a913-5acb1489b88b-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.346985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.346845    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8mmp\" (UniqueName: \"kubernetes.io/projected/e6aed883-80e6-44a5-ac18-855437d32862-kube-api-access-j8mmp\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.346985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.346843    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istiod-ca-cert" pod "ff60abd5-8f12-4e67-99f4-e6baeaee47ce" (UID: "ff60abd5-8f12-4e67-99f4-e6baeaee47ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:00.346985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.346855    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.346985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.346886    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6aed883-80e6-44a5-ac18-855437d32862-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.346985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.346906    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/128497c2-9f64-4902-a913-5acb1489b88b-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.346985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.346920    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.347836 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.347069    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-workload-socket" pod "ff60abd5-8f12-4e67-99f4-e6baeaee47ce" (UID: "ff60abd5-8f12-4e67-99f4-e6baeaee47ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.347836 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.347087    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-workload-certs" pod "ff60abd5-8f12-4e67-99f4-e6baeaee47ce" (UID: "ff60abd5-8f12-4e67-99f4-e6baeaee47ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.348919 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.348889    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istio-token" pod "ff60abd5-8f12-4e67-99f4-e6baeaee47ce" (UID: "ff60abd5-8f12-4e67-99f4-e6baeaee47ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:00.349474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.349430    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istio-envoy" pod "ff60abd5-8f12-4e67-99f4-e6baeaee47ce" (UID: "ff60abd5-8f12-4e67-99f4-e6baeaee47ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.350645 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.350565    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-kube-api-access-js2cm" pod "ff60abd5-8f12-4e67-99f4-e6baeaee47ce" (UID: "ff60abd5-8f12-4e67-99f4-e6baeaee47ce"). InnerVolumeSpecName "kube-api-access-js2cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:00.350976 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.350939    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istio-podinfo" pod "ff60abd5-8f12-4e67-99f4-e6baeaee47ce" (UID: "ff60abd5-8f12-4e67-99f4-e6baeaee47ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:39:00.437698 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.437663    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:39:00.447741 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.447707    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3559421-6424-4480-8d49-5ed4a11deb3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3559421-6424-4480-8d49-5ed4a11deb3d-istiod-ca-cert\") pod \"e3559421-6424-4480-8d49-5ed4a11deb3d\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") "
Oct 30 13:39:00.447952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.447805    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-workload-socket\") pod \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") "
Oct 30 13:39:00.447952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.447842    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-workload-certs\") pod \"e3559421-6424-4480-8d49-5ed4a11deb3d\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") "
Oct 30 13:39:00.447952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.447941    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-istio-data\") pod \"e3559421-6424-4480-8d49-5ed4a11deb3d\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") "
Oct 30 13:39:00.448130 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.448004    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-workload-certs\") pod \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") "
Oct 30 13:39:00.448130 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.448040    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istio-token\" (UniqueName: \"kubernetes.io/projected/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istio-token\") pod \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") "
Oct 30 13:39:00.448325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.448168    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-istio-envoy\") pod \"e3559421-6424-4480-8d49-5ed4a11deb3d\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") "
Oct 30 13:39:00.448325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.448204    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3559421-6424-4480-8d49-5ed4a11deb3d-kube-api-access-w8rsc\" (UniqueName: \"kubernetes.io/projected/e3559421-6424-4480-8d49-5ed4a11deb3d-kube-api-access-w8rsc\") pod \"e3559421-6424-4480-8d49-5ed4a11deb3d\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") "
Oct 30 13:39:00.448325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.448216    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3559421-6424-4480-8d49-5ed4a11deb3d-istiod-ca-cert" pod "e3559421-6424-4480-8d49-5ed4a11deb3d" (UID: "e3559421-6424-4480-8d49-5ed4a11deb3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:00.448325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.448275    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3559421-6424-4480-8d49-5ed4a11deb3d-istio-token\" (UniqueName: \"kubernetes.io/projected/e3559421-6424-4480-8d49-5ed4a11deb3d-istio-token\") pod \"e3559421-6424-4480-8d49-5ed4a11deb3d\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") "
Oct 30 13:39:00.448556 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.448390    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3559421-6424-4480-8d49-5ed4a11deb3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3559421-6424-4480-8d49-5ed4a11deb3d-istio-podinfo\") pod \"e3559421-6424-4480-8d49-5ed4a11deb3d\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") "
Oct 30 13:39:00.448556 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.448427    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-workload-socket\") pod \"e3559421-6424-4480-8d49-5ed4a11deb3d\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") "
Oct 30 13:39:00.448556 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.448489    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-credential-socket\") pod \"e3559421-6424-4480-8d49-5ed4a11deb3d\" (UID: \"e3559421-6424-4480-8d49-5ed4a11deb3d\") "
Oct 30 13:39:00.448556 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.448519    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istio-envoy\") pod \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") "
Oct 30 13:39:00.448781 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.448632    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-workload-certs" pod "62db3dba-ac93-4845-9e85-6d83a6fb0cb1" (UID: "62db3dba-ac93-4845-9e85-6d83a6fb0cb1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.448781 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.448672    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istiod-ca-cert\") pod \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") "
Oct 30 13:39:00.448781 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.448764    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-credential-socket\") pod \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") "
Oct 30 13:39:00.448996 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.448793    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istio-data\") pod \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") "
Oct 30 13:39:00.448996 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.448837    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-kube-api-access-mxx8g\" (UniqueName: \"kubernetes.io/projected/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-kube-api-access-mxx8g\") pod \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") "
Oct 30 13:39:00.448996 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.448856    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-workload-socket" pod "62db3dba-ac93-4845-9e85-6d83a6fb0cb1" (UID: "62db3dba-ac93-4845-9e85-6d83a6fb0cb1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.449164 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.448995    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istio-podinfo\") pod \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\" (UID: \"62db3dba-ac93-4845-9e85-6d83a6fb0cb1\") "
Oct 30 13:39:00.449164 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.449038    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-workload-certs" pod "e3559421-6424-4480-8d49-5ed4a11deb3d" (UID: "e3559421-6424-4480-8d49-5ed4a11deb3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.449309 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.449237    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-istio-data" pod "e3559421-6424-4480-8d49-5ed4a11deb3d" (UID: "e3559421-6424-4480-8d49-5ed4a11deb3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.449909 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.449864    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.449909 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.449896    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js2cm\" (UniqueName: \"kubernetes.io/projected/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-kube-api-access-js2cm\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.450111 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.449913    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.450111 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.449928    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.450111 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.449941    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.450111 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.449955    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.450111 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.449967    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.450111 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.449981    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.450111 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.449993    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.450111 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.450005    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff60abd5-8f12-4e67-99f4-e6baeaee47ce-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.450111 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.450017    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3559421-6424-4480-8d49-5ed4a11deb3d-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.450111 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.450029    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.450111 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.450043    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.452258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.451022    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-workload-socket" pod "e3559421-6424-4480-8d49-5ed4a11deb3d" (UID: "e3559421-6424-4480-8d49-5ed4a11deb3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.452258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.451073    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-credential-socket" pod "62db3dba-ac93-4845-9e85-6d83a6fb0cb1" (UID: "62db3dba-ac93-4845-9e85-6d83a6fb0cb1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.452258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.451294    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-credential-socket" pod "e3559421-6424-4480-8d49-5ed4a11deb3d" (UID: "e3559421-6424-4480-8d49-5ed4a11deb3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.452258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.451446    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istiod-ca-cert" pod "62db3dba-ac93-4845-9e85-6d83a6fb0cb1" (UID: "62db3dba-ac93-4845-9e85-6d83a6fb0cb1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:00.452258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.451523    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istio-data" pod "62db3dba-ac93-4845-9e85-6d83a6fb0cb1" (UID: "62db3dba-ac93-4845-9e85-6d83a6fb0cb1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.454517 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.454469    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3559421-6424-4480-8d49-5ed4a11deb3d-istio-token" pod "e3559421-6424-4480-8d49-5ed4a11deb3d" (UID: "e3559421-6424-4480-8d49-5ed4a11deb3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:00.458322 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.456599    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-istio-envoy" pod "e3559421-6424-4480-8d49-5ed4a11deb3d" (UID: "e3559421-6424-4480-8d49-5ed4a11deb3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.458322 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.457007    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3559421-6424-4480-8d49-5ed4a11deb3d-kube-api-access-w8rsc" pod "e3559421-6424-4480-8d49-5ed4a11deb3d" (UID: "e3559421-6424-4480-8d49-5ed4a11deb3d"). InnerVolumeSpecName "kube-api-access-w8rsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:00.459802 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.459758    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istio-envoy" pod "62db3dba-ac93-4845-9e85-6d83a6fb0cb1" (UID: "62db3dba-ac93-4845-9e85-6d83a6fb0cb1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.459802 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.459793    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3559421-6424-4480-8d49-5ed4a11deb3d-istio-podinfo" pod "e3559421-6424-4480-8d49-5ed4a11deb3d" (UID: "e3559421-6424-4480-8d49-5ed4a11deb3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:39:00.460289 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.460197    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istio-podinfo" pod "62db3dba-ac93-4845-9e85-6d83a6fb0cb1" (UID: "62db3dba-ac93-4845-9e85-6d83a6fb0cb1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:39:00.460457 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.460291    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istio-token" pod "62db3dba-ac93-4845-9e85-6d83a6fb0cb1" (UID: "62db3dba-ac93-4845-9e85-6d83a6fb0cb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:00.463809 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.463779    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-kube-api-access-mxx8g" pod "62db3dba-ac93-4845-9e85-6d83a6fb0cb1" (UID: "62db3dba-ac93-4845-9e85-6d83a6fb0cb1"). InnerVolumeSpecName "kube-api-access-mxx8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:00.550742 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.550637    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80b08e28-171e-49cd-bafe-0fbf4efc09e3-kube-api-access-zjhhc\" (UniqueName: \"kubernetes.io/projected/80b08e28-171e-49cd-bafe-0fbf4efc09e3-kube-api-access-zjhhc\") pod \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") "
Oct 30 13:39:00.550742 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.550726    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istio-podinfo\") pod \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") "
Oct 30 13:39:00.551009 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.550775    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-credential-socket\") pod \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") "
Oct 30 13:39:00.551009 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.550848    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-workload-certs\") pod \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") "
Oct 30 13:39:00.551009 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.550925    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istio-token\" (UniqueName: \"kubernetes.io/projected/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istio-token\") pod \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") "
Oct 30 13:39:00.551009 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.550963    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istio-data\") pod \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") "
Oct 30 13:39:00.551260 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.551029    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istio-envoy\") pod \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") "
Oct 30 13:39:00.551260 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.551169    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-workload-socket\") pod \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") "
Oct 30 13:39:00.551379 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.551284    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istiod-ca-cert\") pod \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\" (UID: \"80b08e28-171e-49cd-bafe-0fbf4efc09e3\") "
Oct 30 13:39:00.551767 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.551736    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-workload-certs" pod "80b08e28-171e-49cd-bafe-0fbf4efc09e3" (UID: "80b08e28-171e-49cd-bafe-0fbf4efc09e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.552557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.551892    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.552557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.552376    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.552557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.552391    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.552557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.552158    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-workload-socket" pod "80b08e28-171e-49cd-bafe-0fbf4efc09e3" (UID: "80b08e28-171e-49cd-bafe-0fbf4efc09e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.552557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.552405    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8rsc\" (UniqueName: \"kubernetes.io/projected/e3559421-6424-4480-8d49-5ed4a11deb3d-kube-api-access-w8rsc\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.552557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.552212    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-credential-socket" pod "80b08e28-171e-49cd-bafe-0fbf4efc09e3" (UID: "80b08e28-171e-49cd-bafe-0fbf4efc09e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.552557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.552249    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istiod-ca-cert" pod "80b08e28-171e-49cd-bafe-0fbf4efc09e3" (UID: "80b08e28-171e-49cd-bafe-0fbf4efc09e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:00.552557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.552419    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3559421-6424-4480-8d49-5ed4a11deb3d-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.552557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.552283    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istio-data" pod "80b08e28-171e-49cd-bafe-0fbf4efc09e3" (UID: "80b08e28-171e-49cd-bafe-0fbf4efc09e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.552557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.552437    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3559421-6424-4480-8d49-5ed4a11deb3d-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.552557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.552451    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.552557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.552465    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3559421-6424-4480-8d49-5ed4a11deb3d-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.552557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.552478    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.552557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.552496    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.552557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.552509    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.552557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.552521    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.552557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.552534    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxx8g\" (UniqueName: \"kubernetes.io/projected/62db3dba-ac93-4845-9e85-6d83a6fb0cb1-kube-api-access-mxx8g\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.557546 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.554763    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istio-token" pod "80b08e28-171e-49cd-bafe-0fbf4efc09e3" (UID: "80b08e28-171e-49cd-bafe-0fbf4efc09e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:00.557546 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.556123    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istio-envoy" pod "80b08e28-171e-49cd-bafe-0fbf4efc09e3" (UID: "80b08e28-171e-49cd-bafe-0fbf4efc09e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.562365 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.562319    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b08e28-171e-49cd-bafe-0fbf4efc09e3-kube-api-access-zjhhc" pod "80b08e28-171e-49cd-bafe-0fbf4efc09e3" (UID: "80b08e28-171e-49cd-bafe-0fbf4efc09e3"). InnerVolumeSpecName "kube-api-access-zjhhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:00.564427 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.564370    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istio-podinfo" pod "80b08e28-171e-49cd-bafe-0fbf4efc09e3" (UID: "80b08e28-171e-49cd-bafe-0fbf4efc09e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:39:00.569166 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.569126    4544 generic.go:356] "Generic (PLEG): container finished" podID="e3559421-6424-4480-8d49-5ed4a11deb3d" containerID="a9f7334495eef7f6dc76371fa6eee8c0a454b18686522a401d2a378e2514fa0c" exitCode=0
Oct 30 13:39:00.569395 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.569180    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"
Oct 30 13:39:00.569395 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.569306    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j" event={"ID":"e3559421-6424-4480-8d49-5ed4a11deb3d","Type":"ContainerDied","Data":"a9f7334495eef7f6dc76371fa6eee8c0a454b18686522a401d2a378e2514fa0c"}
Oct 30 13:39:00.569395 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.569359    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j" event={"ID":"e3559421-6424-4480-8d49-5ed4a11deb3d","Type":"ContainerDied","Data":"363c9b2c18c77f8395a6afaaed19b210f702a2c8d43ed7712f8c7fcff37fceb8"}
Oct 30 13:39:00.569395 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.569389    4544 scope.go:122] "RemoveContainer" containerID="a9f7334495eef7f6dc76371fa6eee8c0a454b18686522a401d2a378e2514fa0c"
Oct 30 13:39:00.572432 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.572200    4544 generic.go:356] "Generic (PLEG): container finished" podID="80b08e28-171e-49cd-bafe-0fbf4efc09e3" containerID="caa31e59de79b72b8cc8b40f021564188634ab8d33cb4ce65747986721379178" exitCode=0
Oct 30 13:39:00.572432 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.572322    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"
Oct 30 13:39:00.572432 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.572268    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g" event={"ID":"80b08e28-171e-49cd-bafe-0fbf4efc09e3","Type":"ContainerDied","Data":"caa31e59de79b72b8cc8b40f021564188634ab8d33cb4ce65747986721379178"}
Oct 30 13:39:00.572769 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.572460    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g" event={"ID":"80b08e28-171e-49cd-bafe-0fbf4efc09e3","Type":"ContainerDied","Data":"29d7dbca3c7840b3f6c0b7ab312f36243ec00073e678cdade51f291c902c0836"}
Oct 30 13:39:00.574526 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.574493    4544 generic.go:356] "Generic (PLEG): container finished" podID="ff60abd5-8f12-4e67-99f4-e6baeaee47ce" containerID="b859c49495aa286dbbb49625bd0e28cdb6f01b56f18f257120889e82a3b6279d" exitCode=0
Oct 30 13:39:00.574697 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.574626    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn" event={"ID":"ff60abd5-8f12-4e67-99f4-e6baeaee47ce","Type":"ContainerDied","Data":"b859c49495aa286dbbb49625bd0e28cdb6f01b56f18f257120889e82a3b6279d"}
Oct 30 13:39:00.574697 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.574667    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn" event={"ID":"ff60abd5-8f12-4e67-99f4-e6baeaee47ce","Type":"ContainerDied","Data":"70d3f83ae1e5d8899ec11db36cb4d5e5db4847eeace788d2f2940d2e01523582"}
Oct 30 13:39:00.574921 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.574748    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"
Oct 30 13:39:00.580596 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.580431    4544 generic.go:356] "Generic (PLEG): container finished" podID="62db3dba-ac93-4845-9e85-6d83a6fb0cb1" containerID="93aecdd1f121efade27a99382f811c8b68b4aa47009bbcb61c09ecd3ba3a3096" exitCode=0
Oct 30 13:39:00.580596 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.580576    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww" event={"ID":"62db3dba-ac93-4845-9e85-6d83a6fb0cb1","Type":"ContainerDied","Data":"93aecdd1f121efade27a99382f811c8b68b4aa47009bbcb61c09ecd3ba3a3096"}
Oct 30 13:39:00.580836 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.580646    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww" event={"ID":"62db3dba-ac93-4845-9e85-6d83a6fb0cb1","Type":"ContainerDied","Data":"5ca4caf10c1bc0512254af24b5835c36eaf56ffb06d3b56ba15da593e3c7aee1"}
Oct 30 13:39:00.580836 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.580688    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"
Oct 30 13:39:00.590114 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.589757    4544 generic.go:356] "Generic (PLEG): container finished" podID="e6aed883-80e6-44a5-ac18-855437d32862" containerID="7dcecf6dfb4b226246e8f47b96862623b45f2a3b508bf93b475327ec6305dcf7" exitCode=0
Oct 30 13:39:00.590114 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.590009    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"
Oct 30 13:39:00.590114 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.590016    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs" event={"ID":"e6aed883-80e6-44a5-ac18-855437d32862","Type":"ContainerDied","Data":"7dcecf6dfb4b226246e8f47b96862623b45f2a3b508bf93b475327ec6305dcf7"}
Oct 30 13:39:00.590114 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.590065    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs" event={"ID":"e6aed883-80e6-44a5-ac18-855437d32862","Type":"ContainerDied","Data":"6e90f0ff4567adf97083426bab796e21d1eb35113c8f40afcc1915c523a84521"}
Oct 30 13:39:00.594577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.592804    4544 generic.go:356] "Generic (PLEG): container finished" podID="128497c2-9f64-4902-a913-5acb1489b88b" containerID="ca0c9e30de327cc6d9419739768d5a50e4af39540c170332e3e4222dae75619e" exitCode=0
Oct 30 13:39:00.594577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.592864    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k" event={"ID":"128497c2-9f64-4902-a913-5acb1489b88b","Type":"ContainerDied","Data":"ca0c9e30de327cc6d9419739768d5a50e4af39540c170332e3e4222dae75619e"}
Oct 30 13:39:00.594577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.592899    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k" event={"ID":"128497c2-9f64-4902-a913-5acb1489b88b","Type":"ContainerDied","Data":"4b8c6d5ec853ae3fab61383f583f9be75bae9d1f9544578643665b38b6c48071"}
Oct 30 13:39:00.594577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.592972    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"
Oct 30 13:39:00.598474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.597318    4544 scope.go:122] "RemoveContainer" containerID="a9f7334495eef7f6dc76371fa6eee8c0a454b18686522a401d2a378e2514fa0c"
Oct 30 13:39:00.598474 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:39:00.597666    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9f7334495eef7f6dc76371fa6eee8c0a454b18686522a401d2a378e2514fa0c\": container with ID starting with a9f7334495eef7f6dc76371fa6eee8c0a454b18686522a401d2a378e2514fa0c not found: ID does not exist" containerID="a9f7334495eef7f6dc76371fa6eee8c0a454b18686522a401d2a378e2514fa0c"
Oct 30 13:39:00.598474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.597696    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9f7334495eef7f6dc76371fa6eee8c0a454b18686522a401d2a378e2514fa0c"} err="failed to get container status \"a9f7334495eef7f6dc76371fa6eee8c0a454b18686522a401d2a378e2514fa0c\": rpc error: code = NotFound desc = could not find container \"a9f7334495eef7f6dc76371fa6eee8c0a454b18686522a401d2a378e2514fa0c\": container with ID starting with a9f7334495eef7f6dc76371fa6eee8c0a454b18686522a401d2a378e2514fa0c not found: ID does not exist"
Oct 30 13:39:00.598474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.597715    4544 scope.go:122] "RemoveContainer" containerID="caa31e59de79b72b8cc8b40f021564188634ab8d33cb4ce65747986721379178"
Oct 30 13:39:00.611010 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.610984    4544 scope.go:122] "RemoveContainer" containerID="caa31e59de79b72b8cc8b40f021564188634ab8d33cb4ce65747986721379178"
Oct 30 13:39:00.611426 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:39:00.611407    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caa31e59de79b72b8cc8b40f021564188634ab8d33cb4ce65747986721379178\": container with ID starting with caa31e59de79b72b8cc8b40f021564188634ab8d33cb4ce65747986721379178 not found: ID does not exist" containerID="caa31e59de79b72b8cc8b40f021564188634ab8d33cb4ce65747986721379178"
Oct 30 13:39:00.611494 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.611437    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caa31e59de79b72b8cc8b40f021564188634ab8d33cb4ce65747986721379178"} err="failed to get container status \"caa31e59de79b72b8cc8b40f021564188634ab8d33cb4ce65747986721379178\": rpc error: code = NotFound desc = could not find container \"caa31e59de79b72b8cc8b40f021564188634ab8d33cb4ce65747986721379178\": container with ID starting with caa31e59de79b72b8cc8b40f021564188634ab8d33cb4ce65747986721379178 not found: ID does not exist"
Oct 30 13:39:00.611494 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.611454    4544 scope.go:122] "RemoveContainer" containerID="b859c49495aa286dbbb49625bd0e28cdb6f01b56f18f257120889e82a3b6279d"
Oct 30 13:39:00.623335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.623311    4544 scope.go:122] "RemoveContainer" containerID="b859c49495aa286dbbb49625bd0e28cdb6f01b56f18f257120889e82a3b6279d"
Oct 30 13:39:00.623614 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:39:00.623585    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b859c49495aa286dbbb49625bd0e28cdb6f01b56f18f257120889e82a3b6279d\": container with ID starting with b859c49495aa286dbbb49625bd0e28cdb6f01b56f18f257120889e82a3b6279d not found: ID does not exist" containerID="b859c49495aa286dbbb49625bd0e28cdb6f01b56f18f257120889e82a3b6279d"
Oct 30 13:39:00.623709 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.623616    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b859c49495aa286dbbb49625bd0e28cdb6f01b56f18f257120889e82a3b6279d"} err="failed to get container status \"b859c49495aa286dbbb49625bd0e28cdb6f01b56f18f257120889e82a3b6279d\": rpc error: code = NotFound desc = could not find container \"b859c49495aa286dbbb49625bd0e28cdb6f01b56f18f257120889e82a3b6279d\": container with ID starting with b859c49495aa286dbbb49625bd0e28cdb6f01b56f18f257120889e82a3b6279d not found: ID does not exist"
Oct 30 13:39:00.623709 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.623638    4544 scope.go:122] "RemoveContainer" containerID="93aecdd1f121efade27a99382f811c8b68b4aa47009bbcb61c09ecd3ba3a3096"
Oct 30 13:39:00.643122 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.643082    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"]
Oct 30 13:39:00.645142 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.645113    4544 scope.go:122] "RemoveContainer" containerID="93aecdd1f121efade27a99382f811c8b68b4aa47009bbcb61c09ecd3ba3a3096"
Oct 30 13:39:00.645572 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:39:00.645544    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93aecdd1f121efade27a99382f811c8b68b4aa47009bbcb61c09ecd3ba3a3096\": container with ID starting with 93aecdd1f121efade27a99382f811c8b68b4aa47009bbcb61c09ecd3ba3a3096 not found: ID does not exist" containerID="93aecdd1f121efade27a99382f811c8b68b4aa47009bbcb61c09ecd3ba3a3096"
Oct 30 13:39:00.645630 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.645584    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93aecdd1f121efade27a99382f811c8b68b4aa47009bbcb61c09ecd3ba3a3096"} err="failed to get container status \"93aecdd1f121efade27a99382f811c8b68b4aa47009bbcb61c09ecd3ba3a3096\": rpc error: code = NotFound desc = could not find container \"93aecdd1f121efade27a99382f811c8b68b4aa47009bbcb61c09ecd3ba3a3096\": container with ID starting with 93aecdd1f121efade27a99382f811c8b68b4aa47009bbcb61c09ecd3ba3a3096 not found: ID does not exist"
Oct 30 13:39:00.645630 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.645610    4544 scope.go:122] "RemoveContainer" containerID="7dcecf6dfb4b226246e8f47b96862623b45f2a3b508bf93b475327ec6305dcf7"
Oct 30 13:39:00.648444 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.648417    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g"]
Oct 30 13:39:00.653809 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.653757    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjhhc\" (UniqueName: \"kubernetes.io/projected/80b08e28-171e-49cd-bafe-0fbf4efc09e3-kube-api-access-zjhhc\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.653930 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.653811    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.653930 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.653826    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.653930 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.653835    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.653930 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.653845    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.653930 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.653855    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.653930 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.653863    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.653930 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.653871    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b08e28-171e-49cd-bafe-0fbf4efc09e3-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.653930 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.653878    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80b08e28-171e-49cd-bafe-0fbf4efc09e3-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.657390 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.657359    4544 scope.go:122] "RemoveContainer" containerID="7dcecf6dfb4b226246e8f47b96862623b45f2a3b508bf93b475327ec6305dcf7"
Oct 30 13:39:00.657832 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:39:00.657787    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dcecf6dfb4b226246e8f47b96862623b45f2a3b508bf93b475327ec6305dcf7\": container with ID starting with 7dcecf6dfb4b226246e8f47b96862623b45f2a3b508bf93b475327ec6305dcf7 not found: ID does not exist" containerID="7dcecf6dfb4b226246e8f47b96862623b45f2a3b508bf93b475327ec6305dcf7"
Oct 30 13:39:00.657943 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.657827    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dcecf6dfb4b226246e8f47b96862623b45f2a3b508bf93b475327ec6305dcf7"} err="failed to get container status \"7dcecf6dfb4b226246e8f47b96862623b45f2a3b508bf93b475327ec6305dcf7\": rpc error: code = NotFound desc = could not find container \"7dcecf6dfb4b226246e8f47b96862623b45f2a3b508bf93b475327ec6305dcf7\": container with ID starting with 7dcecf6dfb4b226246e8f47b96862623b45f2a3b508bf93b475327ec6305dcf7 not found: ID does not exist"
Oct 30 13:39:00.657943 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.657854    4544 scope.go:122] "RemoveContainer" containerID="ca0c9e30de327cc6d9419739768d5a50e4af39540c170332e3e4222dae75619e"
Oct 30 13:39:00.669357 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.669333    4544 scope.go:122] "RemoveContainer" containerID="ca0c9e30de327cc6d9419739768d5a50e4af39540c170332e3e4222dae75619e"
Oct 30 13:39:00.669797 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:39:00.669772    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca0c9e30de327cc6d9419739768d5a50e4af39540c170332e3e4222dae75619e\": container with ID starting with ca0c9e30de327cc6d9419739768d5a50e4af39540c170332e3e4222dae75619e not found: ID does not exist" containerID="ca0c9e30de327cc6d9419739768d5a50e4af39540c170332e3e4222dae75619e"
Oct 30 13:39:00.669862 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.669807    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca0c9e30de327cc6d9419739768d5a50e4af39540c170332e3e4222dae75619e"} err="failed to get container status \"ca0c9e30de327cc6d9419739768d5a50e4af39540c170332e3e4222dae75619e\": rpc error: code = NotFound desc = could not find container \"ca0c9e30de327cc6d9419739768d5a50e4af39540c170332e3e4222dae75619e\": container with ID starting with ca0c9e30de327cc6d9419739768d5a50e4af39540c170332e3e4222dae75619e not found: ID does not exist"
Oct 30 13:39:00.697086 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.697047    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"]
Oct 30 13:39:00.702641 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.702607    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j"]
Oct 30 13:39:00.741031 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.740985    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"]
Oct 30 13:39:00.746140 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.746101    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs"]
Oct 30 13:39:00.779404 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.779370    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"]
Oct 30 13:39:00.783930 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.783897    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn"]
Oct 30 13:39:00.823331 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.823296    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"]
Oct 30 13:39:00.828589 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.828555    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k"]
Oct 30 13:39:00.860574 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.860532    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"]
Oct 30 13:39:00.865676 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:00.865645    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww"]
Oct 30 13:39:01.089510 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:01.089405    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="128497c2-9f64-4902-a913-5acb1489b88b" path="/var/lib/kubelet/pods/128497c2-9f64-4902-a913-5acb1489b88b/volumes"
Oct 30 13:39:01.090060 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:01.090043    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62db3dba-ac93-4845-9e85-6d83a6fb0cb1" path="/var/lib/kubelet/pods/62db3dba-ac93-4845-9e85-6d83a6fb0cb1/volumes"
Oct 30 13:39:01.090597 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:01.090582    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80b08e28-171e-49cd-bafe-0fbf4efc09e3" path="/var/lib/kubelet/pods/80b08e28-171e-49cd-bafe-0fbf4efc09e3/volumes"
Oct 30 13:39:01.091466 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:01.091448    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3559421-6424-4480-8d49-5ed4a11deb3d" path="/var/lib/kubelet/pods/e3559421-6424-4480-8d49-5ed4a11deb3d/volumes"
Oct 30 13:39:01.092029 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:01.092013    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6aed883-80e6-44a5-ac18-855437d32862" path="/var/lib/kubelet/pods/e6aed883-80e6-44a5-ac18-855437d32862/volumes"
Oct 30 13:39:01.092510 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:01.092493    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff60abd5-8f12-4e67-99f4-e6baeaee47ce" path="/var/lib/kubelet/pods/ff60abd5-8f12-4e67-99f4-e6baeaee47ce/volumes"
Oct 30 13:39:01.120067 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:01.120023    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:02.973568 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:02.973528    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:03.461917 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.461890    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:39:03.609039 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.609003    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-workload-certs\") pod \"f44f3385-bd49-41d7-9add-718b415008f2\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") "
Oct 30 13:39:03.609039 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.609050    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f44f3385-bd49-41d7-9add-718b415008f2-kube-api-access-5z24b\" (UniqueName: \"kubernetes.io/projected/f44f3385-bd49-41d7-9add-718b415008f2-kube-api-access-5z24b\") pod \"f44f3385-bd49-41d7-9add-718b415008f2\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") "
Oct 30 13:39:03.609353 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.609107    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-workload-socket\") pod \"f44f3385-bd49-41d7-9add-718b415008f2\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") "
Oct 30 13:39:03.609353 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.609172    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f44f3385-bd49-41d7-9add-718b415008f2-istio-token\" (UniqueName: \"kubernetes.io/projected/f44f3385-bd49-41d7-9add-718b415008f2-istio-token\") pod \"f44f3385-bd49-41d7-9add-718b415008f2\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") "
Oct 30 13:39:03.609353 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.609282    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f44f3385-bd49-41d7-9add-718b415008f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f44f3385-bd49-41d7-9add-718b415008f2-istiod-ca-cert\") pod \"f44f3385-bd49-41d7-9add-718b415008f2\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") "
Oct 30 13:39:03.609525 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.609380    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-istio-envoy\") pod \"f44f3385-bd49-41d7-9add-718b415008f2\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") "
Oct 30 13:39:03.609525 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.609383    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-workload-certs" pod "f44f3385-bd49-41d7-9add-718b415008f2" (UID: "f44f3385-bd49-41d7-9add-718b415008f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:03.609525 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.609437    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-credential-socket\") pod \"f44f3385-bd49-41d7-9add-718b415008f2\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") "
Oct 30 13:39:03.609687 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.609556    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-istio-data\") pod \"f44f3385-bd49-41d7-9add-718b415008f2\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") "
Oct 30 13:39:03.609687 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.609576    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-workload-socket" pod "f44f3385-bd49-41d7-9add-718b415008f2" (UID: "f44f3385-bd49-41d7-9add-718b415008f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:03.609687 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.609586    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f44f3385-bd49-41d7-9add-718b415008f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f44f3385-bd49-41d7-9add-718b415008f2-istio-podinfo\") pod \"f44f3385-bd49-41d7-9add-718b415008f2\" (UID: \"f44f3385-bd49-41d7-9add-718b415008f2\") "
Oct 30 13:39:03.609858 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.609697    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-credential-socket" pod "f44f3385-bd49-41d7-9add-718b415008f2" (UID: "f44f3385-bd49-41d7-9add-718b415008f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:03.609858 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.609715    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f44f3385-bd49-41d7-9add-718b415008f2-istiod-ca-cert" pod "f44f3385-bd49-41d7-9add-718b415008f2" (UID: "f44f3385-bd49-41d7-9add-718b415008f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:03.609962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.609897    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-istio-data" pod "f44f3385-bd49-41d7-9add-718b415008f2" (UID: "f44f3385-bd49-41d7-9add-718b415008f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:03.610381 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.610358    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f44f3385-bd49-41d7-9add-718b415008f2-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:03.610476 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.610383    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:03.610476 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.610399    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:03.610476 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.610411    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:03.610476 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.610424    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:03.611704 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.611671    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f44f3385-bd49-41d7-9add-718b415008f2-istio-podinfo" pod "f44f3385-bd49-41d7-9add-718b415008f2" (UID: "f44f3385-bd49-41d7-9add-718b415008f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:39:03.611704 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.611683    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f44f3385-bd49-41d7-9add-718b415008f2-kube-api-access-5z24b" pod "f44f3385-bd49-41d7-9add-718b415008f2" (UID: "f44f3385-bd49-41d7-9add-718b415008f2"). InnerVolumeSpecName "kube-api-access-5z24b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:03.612056 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.612026    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-istio-envoy" pod "f44f3385-bd49-41d7-9add-718b415008f2" (UID: "f44f3385-bd49-41d7-9add-718b415008f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:03.612169 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.612029    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f44f3385-bd49-41d7-9add-718b415008f2-istio-token" pod "f44f3385-bd49-41d7-9add-718b415008f2" (UID: "f44f3385-bd49-41d7-9add-718b415008f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:03.640391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.639645    4544 generic.go:356] "Generic (PLEG): container finished" podID="f44f3385-bd49-41d7-9add-718b415008f2" containerID="1da3ca37de71d501a467fe0530785de9ae1d35ea57150be91997d7199a83df0c" exitCode=0
Oct 30 13:39:03.640391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.639959    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf" event={"ID":"f44f3385-bd49-41d7-9add-718b415008f2","Type":"ContainerDied","Data":"1da3ca37de71d501a467fe0530785de9ae1d35ea57150be91997d7199a83df0c"}
Oct 30 13:39:03.640391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.640062    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf" event={"ID":"f44f3385-bd49-41d7-9add-718b415008f2","Type":"ContainerDied","Data":"929a30ee4b707616f8bd7ab501442973e855518173c49a90a436415afafd72dd"}
Oct 30 13:39:03.640391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.640130    4544 scope.go:122] "RemoveContainer" containerID="1da3ca37de71d501a467fe0530785de9ae1d35ea57150be91997d7199a83df0c"
Oct 30 13:39:03.641530 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.641027    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"
Oct 30 13:39:03.655603 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.655571    4544 scope.go:122] "RemoveContainer" containerID="1da3ca37de71d501a467fe0530785de9ae1d35ea57150be91997d7199a83df0c"
Oct 30 13:39:03.655999 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:39:03.655979    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1da3ca37de71d501a467fe0530785de9ae1d35ea57150be91997d7199a83df0c\": container with ID starting with 1da3ca37de71d501a467fe0530785de9ae1d35ea57150be91997d7199a83df0c not found: ID does not exist" containerID="1da3ca37de71d501a467fe0530785de9ae1d35ea57150be91997d7199a83df0c"
Oct 30 13:39:03.656055 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.656010    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1da3ca37de71d501a467fe0530785de9ae1d35ea57150be91997d7199a83df0c"} err="failed to get container status \"1da3ca37de71d501a467fe0530785de9ae1d35ea57150be91997d7199a83df0c\": rpc error: code = NotFound desc = could not find container \"1da3ca37de71d501a467fe0530785de9ae1d35ea57150be91997d7199a83df0c\": container with ID starting with 1da3ca37de71d501a467fe0530785de9ae1d35ea57150be91997d7199a83df0c not found: ID does not exist"
Oct 30 13:39:03.696664 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.696629    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"]
Oct 30 13:39:03.701067 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.701026    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf"]
Oct 30 13:39:03.712279 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.712238    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f44f3385-bd49-41d7-9add-718b415008f2-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:03.712279 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.712272    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f44f3385-bd49-41d7-9add-718b415008f2-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:03.712279 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.712282    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z24b\" (UniqueName: \"kubernetes.io/projected/f44f3385-bd49-41d7-9add-718b415008f2-kube-api-access-5z24b\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:03.712546 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:03.712291    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f44f3385-bd49-41d7-9add-718b415008f2-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.155264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.155215    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:39:04.321781 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.321714    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ef74cc1-eb94-4550-af70-38d4f4149963-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ef74cc1-eb94-4550-af70-38d4f4149963-istiod-ca-cert\") pod \"7ef74cc1-eb94-4550-af70-38d4f4149963\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") "
Oct 30 13:39:04.321979 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.321828    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ef74cc1-eb94-4550-af70-38d4f4149963-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ef74cc1-eb94-4550-af70-38d4f4149963-istio-podinfo\") pod \"7ef74cc1-eb94-4550-af70-38d4f4149963\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") "
Oct 30 13:39:04.321979 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.321875    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-credential-socket\") pod \"7ef74cc1-eb94-4550-af70-38d4f4149963\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") "
Oct 30 13:39:04.322091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.321985    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-istio-envoy\") pod \"7ef74cc1-eb94-4550-af70-38d4f4149963\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") "
Oct 30 13:39:04.322091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.322033    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ef74cc1-eb94-4550-af70-38d4f4149963-kube-api-access-9btb7\" (UniqueName: \"kubernetes.io/projected/7ef74cc1-eb94-4550-af70-38d4f4149963-kube-api-access-9btb7\") pod \"7ef74cc1-eb94-4550-af70-38d4f4149963\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") "
Oct 30 13:39:04.322091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.322086    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-workload-certs\") pod \"7ef74cc1-eb94-4550-af70-38d4f4149963\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") "
Oct 30 13:39:04.322302 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.322144    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ef74cc1-eb94-4550-af70-38d4f4149963-istio-token\" (UniqueName: \"kubernetes.io/projected/7ef74cc1-eb94-4550-af70-38d4f4149963-istio-token\") pod \"7ef74cc1-eb94-4550-af70-38d4f4149963\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") "
Oct 30 13:39:04.322342 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.322299    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-istio-data\") pod \"7ef74cc1-eb94-4550-af70-38d4f4149963\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") "
Oct 30 13:39:04.322393 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.322362    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-workload-socket\") pod \"7ef74cc1-eb94-4550-af70-38d4f4149963\" (UID: \"7ef74cc1-eb94-4550-af70-38d4f4149963\") "
Oct 30 13:39:04.322454 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.322408    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ef74cc1-eb94-4550-af70-38d4f4149963-istiod-ca-cert" pod "7ef74cc1-eb94-4550-af70-38d4f4149963" (UID: "7ef74cc1-eb94-4550-af70-38d4f4149963"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:04.323001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.322930    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-credential-socket" pod "7ef74cc1-eb94-4550-af70-38d4f4149963" (UID: "7ef74cc1-eb94-4550-af70-38d4f4149963"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:04.323267 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.323187    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-workload-socket" pod "7ef74cc1-eb94-4550-af70-38d4f4149963" (UID: "7ef74cc1-eb94-4550-af70-38d4f4149963"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:04.323448 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.323423    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-istio-data" pod "7ef74cc1-eb94-4550-af70-38d4f4149963" (UID: "7ef74cc1-eb94-4550-af70-38d4f4149963"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:04.323569 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.323546    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.323640 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.323578    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ef74cc1-eb94-4550-af70-38d4f4149963-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.323640 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.323595    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.323640 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.323623    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-workload-certs" pod "7ef74cc1-eb94-4550-af70-38d4f4149963" (UID: "7ef74cc1-eb94-4550-af70-38d4f4149963"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:04.325197 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.325151    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ef74cc1-eb94-4550-af70-38d4f4149963-kube-api-access-9btb7" pod "7ef74cc1-eb94-4550-af70-38d4f4149963" (UID: "7ef74cc1-eb94-4550-af70-38d4f4149963"). InnerVolumeSpecName "kube-api-access-9btb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:04.326050 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.326019    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-istio-envoy" pod "7ef74cc1-eb94-4550-af70-38d4f4149963" (UID: "7ef74cc1-eb94-4550-af70-38d4f4149963"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:04.326155 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.326109    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ef74cc1-eb94-4550-af70-38d4f4149963-istio-token" pod "7ef74cc1-eb94-4550-af70-38d4f4149963" (UID: "7ef74cc1-eb94-4550-af70-38d4f4149963"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:04.327644 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.327618    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ef74cc1-eb94-4550-af70-38d4f4149963-istio-podinfo" pod "7ef74cc1-eb94-4550-af70-38d4f4149963" (UID: "7ef74cc1-eb94-4550-af70-38d4f4149963"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:39:04.424934 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.424834    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ef74cc1-eb94-4550-af70-38d4f4149963-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.424934 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.424868    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.424934 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.424880    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ef74cc1-eb94-4550-af70-38d4f4149963-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.424934 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.424892    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.424934 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.424904    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9btb7\" (UniqueName: \"kubernetes.io/projected/7ef74cc1-eb94-4550-af70-38d4f4149963-kube-api-access-9btb7\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.424934 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.424920    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ef74cc1-eb94-4550-af70-38d4f4149963-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.649833 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.649800    4544 generic.go:356] "Generic (PLEG): container finished" podID="7ef74cc1-eb94-4550-af70-38d4f4149963" containerID="2fab71d50be57d2748266648f6d21cd9f1d1639dd149f9083673f7216b8353b4" exitCode=0
Oct 30 13:39:04.650058 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.649852    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"
Oct 30 13:39:04.650058 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.649881    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r" event={"ID":"7ef74cc1-eb94-4550-af70-38d4f4149963","Type":"ContainerDied","Data":"2fab71d50be57d2748266648f6d21cd9f1d1639dd149f9083673f7216b8353b4"}
Oct 30 13:39:04.650058 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.649929    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r" event={"ID":"7ef74cc1-eb94-4550-af70-38d4f4149963","Type":"ContainerDied","Data":"99014900ff6fbb3789befd6933876bb2809fe8041c9616b892974d36ef82b323"}
Oct 30 13:39:04.650058 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.649951    4544 scope.go:122] "RemoveContainer" containerID="2fab71d50be57d2748266648f6d21cd9f1d1639dd149f9083673f7216b8353b4"
Oct 30 13:39:04.663984 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.663955    4544 scope.go:122] "RemoveContainer" containerID="2fab71d50be57d2748266648f6d21cd9f1d1639dd149f9083673f7216b8353b4"
Oct 30 13:39:04.664449 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:39:04.664420    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fab71d50be57d2748266648f6d21cd9f1d1639dd149f9083673f7216b8353b4\": container with ID starting with 2fab71d50be57d2748266648f6d21cd9f1d1639dd149f9083673f7216b8353b4 not found: ID does not exist" containerID="2fab71d50be57d2748266648f6d21cd9f1d1639dd149f9083673f7216b8353b4"
Oct 30 13:39:04.664579 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.664460    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fab71d50be57d2748266648f6d21cd9f1d1639dd149f9083673f7216b8353b4"} err="failed to get container status \"2fab71d50be57d2748266648f6d21cd9f1d1639dd149f9083673f7216b8353b4\": rpc error: code = NotFound desc = could not find container \"2fab71d50be57d2748266648f6d21cd9f1d1639dd149f9083673f7216b8353b4\": container with ID starting with 2fab71d50be57d2748266648f6d21cd9f1d1639dd149f9083673f7216b8353b4 not found: ID does not exist"
Oct 30 13:39:04.719054 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.718949    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"]
Oct 30 13:39:04.731100 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:04.731057    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r"]
Oct 30 13:39:05.016700 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.016588    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:05.044877 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.044845    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:39:05.103628 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.103594    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ef74cc1-eb94-4550-af70-38d4f4149963" path="/var/lib/kubelet/pods/7ef74cc1-eb94-4550-af70-38d4f4149963/volumes"
Oct 30 13:39:05.104355 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.104310    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f44f3385-bd49-41d7-9add-718b415008f2" path="/var/lib/kubelet/pods/f44f3385-bd49-41d7-9add-718b415008f2/volumes"
Oct 30 13:39:05.134196 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.134136    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/347e2f12-0089-42aa-86b8-d42d02a56175-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/347e2f12-0089-42aa-86b8-d42d02a56175-istiod-ca-cert\") pod \"347e2f12-0089-42aa-86b8-d42d02a56175\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") "
Oct 30 13:39:05.134432 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.134248    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-istio-envoy\") pod \"347e2f12-0089-42aa-86b8-d42d02a56175\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") "
Oct 30 13:39:05.134432 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.134315    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-istio-data\") pod \"347e2f12-0089-42aa-86b8-d42d02a56175\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") "
Oct 30 13:39:05.134586 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.134460    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-workload-certs\") pod \"347e2f12-0089-42aa-86b8-d42d02a56175\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") "
Oct 30 13:39:05.134586 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.134538    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-workload-socket\") pod \"347e2f12-0089-42aa-86b8-d42d02a56175\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") "
Oct 30 13:39:05.134749 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.134638    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-istio-data" pod "347e2f12-0089-42aa-86b8-d42d02a56175" (UID: "347e2f12-0089-42aa-86b8-d42d02a56175"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:05.134749 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.134664    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/347e2f12-0089-42aa-86b8-d42d02a56175-istiod-ca-cert" pod "347e2f12-0089-42aa-86b8-d42d02a56175" (UID: "347e2f12-0089-42aa-86b8-d42d02a56175"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:05.134749 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.134698    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/347e2f12-0089-42aa-86b8-d42d02a56175-kube-api-access-gwl2g\" (UniqueName: \"kubernetes.io/projected/347e2f12-0089-42aa-86b8-d42d02a56175-kube-api-access-gwl2g\") pod \"347e2f12-0089-42aa-86b8-d42d02a56175\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") "
Oct 30 13:39:05.135002 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.134768    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-credential-socket\") pod \"347e2f12-0089-42aa-86b8-d42d02a56175\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") "
Oct 30 13:39:05.135002 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.134815    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/347e2f12-0089-42aa-86b8-d42d02a56175-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/347e2f12-0089-42aa-86b8-d42d02a56175-istio-podinfo\") pod \"347e2f12-0089-42aa-86b8-d42d02a56175\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") "
Oct 30 13:39:05.135002 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.134849    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/347e2f12-0089-42aa-86b8-d42d02a56175-istio-token\" (UniqueName: \"kubernetes.io/projected/347e2f12-0089-42aa-86b8-d42d02a56175-istio-token\") pod \"347e2f12-0089-42aa-86b8-d42d02a56175\" (UID: \"347e2f12-0089-42aa-86b8-d42d02a56175\") "
Oct 30 13:39:05.135002 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.134868    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-workload-certs" pod "347e2f12-0089-42aa-86b8-d42d02a56175" (UID: "347e2f12-0089-42aa-86b8-d42d02a56175"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:05.135002 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.134887    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-workload-socket" pod "347e2f12-0089-42aa-86b8-d42d02a56175" (UID: "347e2f12-0089-42aa-86b8-d42d02a56175"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:05.135321 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.135066    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-credential-socket" pod "347e2f12-0089-42aa-86b8-d42d02a56175" (UID: "347e2f12-0089-42aa-86b8-d42d02a56175"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:05.135712 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.135687    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:05.135815 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.135719    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/347e2f12-0089-42aa-86b8-d42d02a56175-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:05.135815 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.135734    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:05.135815 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.135746    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:05.135815 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.135758    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:05.137496 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.137465    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/347e2f12-0089-42aa-86b8-d42d02a56175-istio-podinfo" pod "347e2f12-0089-42aa-86b8-d42d02a56175" (UID: "347e2f12-0089-42aa-86b8-d42d02a56175"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:39:05.137608 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.137587    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-istio-envoy" pod "347e2f12-0089-42aa-86b8-d42d02a56175" (UID: "347e2f12-0089-42aa-86b8-d42d02a56175"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:05.137676 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.137660    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/347e2f12-0089-42aa-86b8-d42d02a56175-kube-api-access-gwl2g" pod "347e2f12-0089-42aa-86b8-d42d02a56175" (UID: "347e2f12-0089-42aa-86b8-d42d02a56175"). InnerVolumeSpecName "kube-api-access-gwl2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:05.138060 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.138043    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/347e2f12-0089-42aa-86b8-d42d02a56175-istio-token" pod "347e2f12-0089-42aa-86b8-d42d02a56175" (UID: "347e2f12-0089-42aa-86b8-d42d02a56175"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:05.236769 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.236733    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/347e2f12-0089-42aa-86b8-d42d02a56175-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:05.236769 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.236766    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwl2g\" (UniqueName: \"kubernetes.io/projected/347e2f12-0089-42aa-86b8-d42d02a56175-kube-api-access-gwl2g\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:05.236769 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.236779    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/347e2f12-0089-42aa-86b8-d42d02a56175-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:05.237315 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.236789    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/347e2f12-0089-42aa-86b8-d42d02a56175-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:05.663702 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.663669    4544 generic.go:356] "Generic (PLEG): container finished" podID="347e2f12-0089-42aa-86b8-d42d02a56175" containerID="33153249a7f6d8b9013d6842dd92d84c7c343ae48ab96cfc53ff0eba3e23fca1" exitCode=0
Oct 30 13:39:05.663907 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.663715    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"
Oct 30 13:39:05.663907 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.663754    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm" event={"ID":"347e2f12-0089-42aa-86b8-d42d02a56175","Type":"ContainerDied","Data":"33153249a7f6d8b9013d6842dd92d84c7c343ae48ab96cfc53ff0eba3e23fca1"}
Oct 30 13:39:05.663907 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.663818    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm" event={"ID":"347e2f12-0089-42aa-86b8-d42d02a56175","Type":"ContainerDied","Data":"f9c209eac3ae95a3156acffcd481f20fc2feb2b56955675f01e9d44c1cad0122"}
Oct 30 13:39:05.663907 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.663842    4544 scope.go:122] "RemoveContainer" containerID="33153249a7f6d8b9013d6842dd92d84c7c343ae48ab96cfc53ff0eba3e23fca1"
Oct 30 13:39:05.680912 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.680875    4544 scope.go:122] "RemoveContainer" containerID="33153249a7f6d8b9013d6842dd92d84c7c343ae48ab96cfc53ff0eba3e23fca1"
Oct 30 13:39:05.681459 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:39:05.681427    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33153249a7f6d8b9013d6842dd92d84c7c343ae48ab96cfc53ff0eba3e23fca1\": container with ID starting with 33153249a7f6d8b9013d6842dd92d84c7c343ae48ab96cfc53ff0eba3e23fca1 not found: ID does not exist" containerID="33153249a7f6d8b9013d6842dd92d84c7c343ae48ab96cfc53ff0eba3e23fca1"
Oct 30 13:39:05.681573 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.681473    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33153249a7f6d8b9013d6842dd92d84c7c343ae48ab96cfc53ff0eba3e23fca1"} err="failed to get container status \"33153249a7f6d8b9013d6842dd92d84c7c343ae48ab96cfc53ff0eba3e23fca1\": rpc error: code = NotFound desc = could not find container \"33153249a7f6d8b9013d6842dd92d84c7c343ae48ab96cfc53ff0eba3e23fca1\": container with ID starting with 33153249a7f6d8b9013d6842dd92d84c7c343ae48ab96cfc53ff0eba3e23fca1 not found: ID does not exist"
Oct 30 13:39:05.730568 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.729823    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"]
Oct 30 13:39:05.737119 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:05.737088    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm"]
Oct 30 13:39:06.912544 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:06.912501    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:07.094641 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:07.094599    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="347e2f12-0089-42aa-86b8-d42d02a56175" path="/var/lib/kubelet/pods/347e2f12-0089-42aa-86b8-d42d02a56175/volumes"
Oct 30 13:39:07.327643 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:07.327603    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx"]
Oct 30 13:39:07.362614 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:07.362578    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq"]
Oct 30 13:39:07.488555 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:07.488510    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:07.603937 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:07.603833    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:08.365586 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:08.365548    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls"]
Oct 30 13:39:08.366119 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:08.365825    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls" podUID="01fb99d1-307e-4917-bb84-b0276b6f78a7" containerName="maas-api" containerID="cri-o://919edfeac1f9d1a694fafe8d9f7fe07cfaefee798b23156e86b479542d58ae74" gracePeriod=30
Oct 30 13:39:08.441433 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:08.441400    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:08.723108 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:08.723074    4544 generic.go:356] "Generic (PLEG): container finished" podID="01fb99d1-307e-4917-bb84-b0276b6f78a7" containerID="919edfeac1f9d1a694fafe8d9f7fe07cfaefee798b23156e86b479542d58ae74" exitCode=0
Oct 30 13:39:08.723336 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:08.723155    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls" event={"ID":"01fb99d1-307e-4917-bb84-b0276b6f78a7","Type":"ContainerDied","Data":"919edfeac1f9d1a694fafe8d9f7fe07cfaefee798b23156e86b479542d58ae74"}
Oct 30 13:39:08.739926 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:08.739892    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls"
Oct 30 13:39:08.903162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:08.902277    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01fb99d1-307e-4917-bb84-b0276b6f78a7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01fb99d1-307e-4917-bb84-b0276b6f78a7-maas-api-tls\") pod \"01fb99d1-307e-4917-bb84-b0276b6f78a7\" (UID: \"01fb99d1-307e-4917-bb84-b0276b6f78a7\") "
Oct 30 13:39:08.903162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:08.902364    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01fb99d1-307e-4917-bb84-b0276b6f78a7-kube-api-access-qxh6t\" (UniqueName: \"kubernetes.io/projected/01fb99d1-307e-4917-bb84-b0276b6f78a7-kube-api-access-qxh6t\") pod \"01fb99d1-307e-4917-bb84-b0276b6f78a7\" (UID: \"01fb99d1-307e-4917-bb84-b0276b6f78a7\") "
Oct 30 13:39:08.903162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:08.902442    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01fb99d1-307e-4917-bb84-b0276b6f78a7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01fb99d1-307e-4917-bb84-b0276b6f78a7-openshift-service-ca\") pod \"01fb99d1-307e-4917-bb84-b0276b6f78a7\" (UID: \"01fb99d1-307e-4917-bb84-b0276b6f78a7\") "
Oct 30 13:39:08.903162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:08.902522    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01fb99d1-307e-4917-bb84-b0276b6f78a7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/01fb99d1-307e-4917-bb84-b0276b6f78a7-metrics-certs\") pod \"01fb99d1-307e-4917-bb84-b0276b6f78a7\" (UID: \"01fb99d1-307e-4917-bb84-b0276b6f78a7\") "
Oct 30 13:39:08.911064 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:08.909632    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01fb99d1-307e-4917-bb84-b0276b6f78a7-openshift-service-ca" pod "01fb99d1-307e-4917-bb84-b0276b6f78a7" (UID: "01fb99d1-307e-4917-bb84-b0276b6f78a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:08.954803 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:08.954712    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01fb99d1-307e-4917-bb84-b0276b6f78a7-metrics-certs" pod "01fb99d1-307e-4917-bb84-b0276b6f78a7" (UID: "01fb99d1-307e-4917-bb84-b0276b6f78a7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:39:08.956191 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:08.956149    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01fb99d1-307e-4917-bb84-b0276b6f78a7-maas-api-tls" pod "01fb99d1-307e-4917-bb84-b0276b6f78a7" (UID: "01fb99d1-307e-4917-bb84-b0276b6f78a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:39:08.956393 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:08.956362    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01fb99d1-307e-4917-bb84-b0276b6f78a7-kube-api-access-qxh6t" pod "01fb99d1-307e-4917-bb84-b0276b6f78a7" (UID: "01fb99d1-307e-4917-bb84-b0276b6f78a7"). InnerVolumeSpecName "kube-api-access-qxh6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:09.004390 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:09.004355    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxh6t\" (UniqueName: \"kubernetes.io/projected/01fb99d1-307e-4917-bb84-b0276b6f78a7-kube-api-access-qxh6t\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:09.004390 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:09.004390    4544 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01fb99d1-307e-4917-bb84-b0276b6f78a7-openshift-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:09.004625 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:09.004407    4544 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01fb99d1-307e-4917-bb84-b0276b6f78a7-metrics-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:09.004625 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:09.004420    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01fb99d1-307e-4917-bb84-b0276b6f78a7-maas-api-tls\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:09.004724 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:09.004626    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:09.247985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:09.247898    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:09.374400 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:09.374369    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:09.733025 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:09.732988    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls"
Oct 30 13:39:09.733025 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:09.732998    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls" event={"ID":"01fb99d1-307e-4917-bb84-b0276b6f78a7","Type":"ContainerDied","Data":"9a19f54320c63b4f52d24d1a41fe2ba8f76cfdfc6c04c7f3e050148b929f0f2a"}
Oct 30 13:39:09.733256 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:09.733102    4544 scope.go:122] "RemoveContainer" containerID="919edfeac1f9d1a694fafe8d9f7fe07cfaefee798b23156e86b479542d58ae74"
Oct 30 13:39:09.790448 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:09.790408    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls"]
Oct 30 13:39:09.802532 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:09.802493    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-acd125-fb9c67dd5-4nrls"]
Oct 30 13:39:09.811924 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:09.811891    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:09.912681 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:09.912643    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h"]
Oct 30 13:39:09.955532 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:09.955489    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m"]
Oct 30 13:39:10.092712 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:10.092669    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:10.215285 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:10.214372    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:11.295056 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:11.295016    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01fb99d1-307e-4917-bb84-b0276b6f78a7" path="/var/lib/kubelet/pods/01fb99d1-307e-4917-bb84-b0276b6f78a7/volumes"
Oct 30 13:39:11.584619 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:11.584587    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:12.328593 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:12.328534    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx" podUID="fb981ab3-c15c-48d3-868d-5d5d93616212" containerName="payload-processing" containerID="cri-o://56127a11104d74fee2e2b51d20306dc9198826a0132bcc9c92fa180264c09248" gracePeriod=25
Oct 30 13:39:12.363348 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:12.363306    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq" podUID="8013c75e-2665-43ee-879b-ec14f338cca5" containerName="payload-pre-processing" containerID="cri-o://94ed8da1a2b8104aeb2efa3559ca628109c55dcf45e01b0ee5dee61aba172e25" gracePeriod=25
Oct 30 13:39:12.587432 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:12.587349    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:13.018831 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:13.018729    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:13.163474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:13.163432    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:13.315638 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:13.315542    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk"]
Oct 30 13:39:13.315919 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:13.315838    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk" podUID="d4fc9668-7314-46ee-bfef-6092e5b58d61" containerName="maas-api" containerID="cri-o://b613d9d7edcf38bd9e93fbe10be08a613bc31eaf3a8f18f6df35d0883081c6cc" gracePeriod=30
Oct 30 13:39:13.403583 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:13.403545    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:13.621945 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:13.621916    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk"
Oct 30 13:39:13.631727 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:13.631696    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4fc9668-7314-46ee-bfef-6092e5b58d61-kube-api-access-sw7gb\" (UniqueName: \"kubernetes.io/projected/d4fc9668-7314-46ee-bfef-6092e5b58d61-kube-api-access-sw7gb\") pod \"d4fc9668-7314-46ee-bfef-6092e5b58d61\" (UID: \"d4fc9668-7314-46ee-bfef-6092e5b58d61\") "
Oct 30 13:39:13.631936 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:13.631806    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4fc9668-7314-46ee-bfef-6092e5b58d61-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4fc9668-7314-46ee-bfef-6092e5b58d61-metrics-certs\") pod \"d4fc9668-7314-46ee-bfef-6092e5b58d61\" (UID: \"d4fc9668-7314-46ee-bfef-6092e5b58d61\") "
Oct 30 13:39:13.632057 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:13.631982    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4fc9668-7314-46ee-bfef-6092e5b58d61-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4fc9668-7314-46ee-bfef-6092e5b58d61-maas-api-tls\") pod \"d4fc9668-7314-46ee-bfef-6092e5b58d61\" (UID: \"d4fc9668-7314-46ee-bfef-6092e5b58d61\") "
Oct 30 13:39:13.632057 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:13.632042    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4fc9668-7314-46ee-bfef-6092e5b58d61-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fc9668-7314-46ee-bfef-6092e5b58d61-openshift-service-ca\") pod \"d4fc9668-7314-46ee-bfef-6092e5b58d61\" (UID: \"d4fc9668-7314-46ee-bfef-6092e5b58d61\") "
Oct 30 13:39:13.633092 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:13.633059    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4fc9668-7314-46ee-bfef-6092e5b58d61-openshift-service-ca" pod "d4fc9668-7314-46ee-bfef-6092e5b58d61" (UID: "d4fc9668-7314-46ee-bfef-6092e5b58d61"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:13.634401 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:13.634379    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4fc9668-7314-46ee-bfef-6092e5b58d61-maas-api-tls" pod "d4fc9668-7314-46ee-bfef-6092e5b58d61" (UID: "d4fc9668-7314-46ee-bfef-6092e5b58d61"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:39:13.634655 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:13.634616    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4fc9668-7314-46ee-bfef-6092e5b58d61-metrics-certs" pod "d4fc9668-7314-46ee-bfef-6092e5b58d61" (UID: "d4fc9668-7314-46ee-bfef-6092e5b58d61"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:39:13.634798 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:13.634772    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4fc9668-7314-46ee-bfef-6092e5b58d61-kube-api-access-sw7gb" pod "d4fc9668-7314-46ee-bfef-6092e5b58d61" (UID: "d4fc9668-7314-46ee-bfef-6092e5b58d61"). InnerVolumeSpecName "kube-api-access-sw7gb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:13.667205 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:13.667168    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:13.734772 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:13.734738    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4fc9668-7314-46ee-bfef-6092e5b58d61-maas-api-tls\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:13.734772 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:13.734774    4544 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fc9668-7314-46ee-bfef-6092e5b58d61-openshift-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:13.735002 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:13.734796    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw7gb\" (UniqueName: \"kubernetes.io/projected/d4fc9668-7314-46ee-bfef-6092e5b58d61-kube-api-access-sw7gb\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:13.735002 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:13.734809    4544 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4fc9668-7314-46ee-bfef-6092e5b58d61-metrics-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:13.773986 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:13.773950    4544 generic.go:356] "Generic (PLEG): container finished" podID="d4fc9668-7314-46ee-bfef-6092e5b58d61" containerID="b613d9d7edcf38bd9e93fbe10be08a613bc31eaf3a8f18f6df35d0883081c6cc" exitCode=0
Oct 30 13:39:13.774168 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:13.773996    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk"
Oct 30 13:39:13.774168 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:13.774029    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk" event={"ID":"d4fc9668-7314-46ee-bfef-6092e5b58d61","Type":"ContainerDied","Data":"b613d9d7edcf38bd9e93fbe10be08a613bc31eaf3a8f18f6df35d0883081c6cc"}
Oct 30 13:39:13.774168 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:13.774093    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk" event={"ID":"d4fc9668-7314-46ee-bfef-6092e5b58d61","Type":"ContainerDied","Data":"b0532979304701f3ef4b0b386b684b36f1b5bf58037d67e68915b3609513f3a3"}
Oct 30 13:39:13.774168 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:13.774121    4544 scope.go:122] "RemoveContainer" containerID="b613d9d7edcf38bd9e93fbe10be08a613bc31eaf3a8f18f6df35d0883081c6cc"
Oct 30 13:39:13.786714 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:13.786697    4544 scope.go:122] "RemoveContainer" containerID="b613d9d7edcf38bd9e93fbe10be08a613bc31eaf3a8f18f6df35d0883081c6cc"
Oct 30 13:39:13.787052 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:39:13.787028    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b613d9d7edcf38bd9e93fbe10be08a613bc31eaf3a8f18f6df35d0883081c6cc\": container with ID starting with b613d9d7edcf38bd9e93fbe10be08a613bc31eaf3a8f18f6df35d0883081c6cc not found: ID does not exist" containerID="b613d9d7edcf38bd9e93fbe10be08a613bc31eaf3a8f18f6df35d0883081c6cc"
Oct 30 13:39:13.787127 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:13.787063    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b613d9d7edcf38bd9e93fbe10be08a613bc31eaf3a8f18f6df35d0883081c6cc"} err="failed to get container status \"b613d9d7edcf38bd9e93fbe10be08a613bc31eaf3a8f18f6df35d0883081c6cc\": rpc error: code = NotFound desc = could not find container \"b613d9d7edcf38bd9e93fbe10be08a613bc31eaf3a8f18f6df35d0883081c6cc\": container with ID starting with b613d9d7edcf38bd9e93fbe10be08a613bc31eaf3a8f18f6df35d0883081c6cc not found: ID does not exist"
Oct 30 13:39:13.794042 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:13.794016    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:13.833830 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:13.833795    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk"]
Oct 30 13:39:13.847336 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:13.847262    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c91909-6dc85d8559-9nrgk"]
Oct 30 13:39:13.983247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:13.978126    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:14.242560 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:14.242456    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:14.397450 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:14.397398    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:14.879548 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:14.879514    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:14.913271 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:14.913240    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h" podUID="329fbf95-10bc-4b2f-afc2-14c679d9edd8" containerName="payload-processing" containerID="cri-o://552f1749d883c7e3d623a150d03d8efc63459572b4d8bffe2bf42a057fab6afd" gracePeriod=25
Oct 30 13:39:14.955782 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:14.955749    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m" podUID="611e617c-64ab-4006-972d-fa789281cc38" containerName="payload-pre-processing" containerID="cri-o://526813699e40d766f1321b213aaceeb866a0070b92310b5f5662b449af998e01" gracePeriod=25
Oct 30 13:39:15.021936 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:15.021897    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:15.087910 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:15.087874    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4fc9668-7314-46ee-bfef-6092e5b58d61" path="/var/lib/kubelet/pods/d4fc9668-7314-46ee-bfef-6092e5b58d61/volumes"
Oct 30 13:39:15.528653 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:15.528609    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:16.316578 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:16.316533    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx" podUID="fb981ab3-c15c-48d3-868d-5d5d93616212" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:39:16.317016 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:16.316582    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq" podUID="8013c75e-2665-43ee-879b-ec14f338cca5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:39:16.591441 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:16.591402    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h" podUID="329fbf95-10bc-4b2f-afc2-14c679d9edd8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:39:16.591600 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:16.591417    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m" podUID="611e617c-64ab-4006-972d-fa789281cc38" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:39:16.732998 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:16.732963    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:17.285778 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:17.285735    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:17.431402 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:17.431368    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:17.576805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:17.576762    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:17.641254 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:17.614951    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s"]
Oct 30 13:39:17.641254 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:17.615251    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s" podUID="3f3ee96f-30cf-4afe-a5a7-0fb69beaa873" containerName="maas-api" containerID="cri-o://d1c06a039a0f077cdaedeeff7456afd0d3b441a87950853445915b2ad18ff9a4" gracePeriod=30
Oct 30 13:39:17.776595 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:17.776538    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:17.815260 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:17.815184    4544 generic.go:356] "Generic (PLEG): container finished" podID="3f3ee96f-30cf-4afe-a5a7-0fb69beaa873" containerID="d1c06a039a0f077cdaedeeff7456afd0d3b441a87950853445915b2ad18ff9a4" exitCode=0
Oct 30 13:39:17.815555 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:17.815527    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s" event={"ID":"3f3ee96f-30cf-4afe-a5a7-0fb69beaa873","Type":"ContainerDied","Data":"d1c06a039a0f077cdaedeeff7456afd0d3b441a87950853445915b2ad18ff9a4"}
Oct 30 13:39:17.911234 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:17.911155    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:18.005188 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:18.005131    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s"
Oct 30 13:39:18.033798 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:18.033761    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:18.048750 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:18.048719    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-09db16-kmnhr"]
Oct 30 13:39:18.055782 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:18.055748    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f3ee96f-30cf-4afe-a5a7-0fb69beaa873-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f3ee96f-30cf-4afe-a5a7-0fb69beaa873-openshift-service-ca\") pod \"3f3ee96f-30cf-4afe-a5a7-0fb69beaa873\" (UID: \"3f3ee96f-30cf-4afe-a5a7-0fb69beaa873\") "
Oct 30 13:39:18.055987 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:18.055812    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f3ee96f-30cf-4afe-a5a7-0fb69beaa873-kube-api-access-4s7wc\" (UniqueName: \"kubernetes.io/projected/3f3ee96f-30cf-4afe-a5a7-0fb69beaa873-kube-api-access-4s7wc\") pod \"3f3ee96f-30cf-4afe-a5a7-0fb69beaa873\" (UID: \"3f3ee96f-30cf-4afe-a5a7-0fb69beaa873\") "
Oct 30 13:39:18.055987 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:18.055950    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f3ee96f-30cf-4afe-a5a7-0fb69beaa873-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f3ee96f-30cf-4afe-a5a7-0fb69beaa873-metrics-certs\") pod \"3f3ee96f-30cf-4afe-a5a7-0fb69beaa873\" (UID: \"3f3ee96f-30cf-4afe-a5a7-0fb69beaa873\") "
Oct 30 13:39:18.056155 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:18.056050    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f3ee96f-30cf-4afe-a5a7-0fb69beaa873-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f3ee96f-30cf-4afe-a5a7-0fb69beaa873-maas-api-tls\") pod \"3f3ee96f-30cf-4afe-a5a7-0fb69beaa873\" (UID: \"3f3ee96f-30cf-4afe-a5a7-0fb69beaa873\") "
Oct 30 13:39:18.056155 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:18.056129    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f3ee96f-30cf-4afe-a5a7-0fb69beaa873-openshift-service-ca" pod "3f3ee96f-30cf-4afe-a5a7-0fb69beaa873" (UID: "3f3ee96f-30cf-4afe-a5a7-0fb69beaa873"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:18.056882 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:18.056601    4544 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f3ee96f-30cf-4afe-a5a7-0fb69beaa873-openshift-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:18.058526 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:18.058495    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f3ee96f-30cf-4afe-a5a7-0fb69beaa873-metrics-certs" pod "3f3ee96f-30cf-4afe-a5a7-0fb69beaa873" (UID: "3f3ee96f-30cf-4afe-a5a7-0fb69beaa873"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:39:18.058720 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:18.058698    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f3ee96f-30cf-4afe-a5a7-0fb69beaa873-kube-api-access-4s7wc" pod "3f3ee96f-30cf-4afe-a5a7-0fb69beaa873" (UID: "3f3ee96f-30cf-4afe-a5a7-0fb69beaa873"). InnerVolumeSpecName "kube-api-access-4s7wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:18.059572 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:18.059545    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f3ee96f-30cf-4afe-a5a7-0fb69beaa873-maas-api-tls" pod "3f3ee96f-30cf-4afe-a5a7-0fb69beaa873" (UID: "3f3ee96f-30cf-4afe-a5a7-0fb69beaa873"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:39:18.062828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:18.062804    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-09db16-kmnhr"]
Oct 30 13:39:18.157786 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:18.157639    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f3ee96f-30cf-4afe-a5a7-0fb69beaa873-maas-api-tls\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:18.157786 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:18.157673    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s7wc\" (UniqueName: \"kubernetes.io/projected/3f3ee96f-30cf-4afe-a5a7-0fb69beaa873-kube-api-access-4s7wc\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:18.157786 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:18.157687    4544 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f3ee96f-30cf-4afe-a5a7-0fb69beaa873-metrics-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:18.158323 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:18.158290    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:18.401162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:18.401120    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:18.560063 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:18.559967    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:18.729425 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:18.729384    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:18.844063 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:18.844014    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s" event={"ID":"3f3ee96f-30cf-4afe-a5a7-0fb69beaa873","Type":"ContainerDied","Data":"c3f69de9ac5f6d416a5de34bc8c6f05ff2a4bb4a3b2c29d2feb78a89a9b41d6f"}
Oct 30 13:39:18.844063 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:18.844058    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s"
Oct 30 13:39:18.844354 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:18.844115    4544 scope.go:122] "RemoveContainer" containerID="d1c06a039a0f077cdaedeeff7456afd0d3b441a87950853445915b2ad18ff9a4"
Oct 30 13:39:18.847209 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:18.847172    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:18.904779 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:18.904735    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s"]
Oct 30 13:39:18.908525 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:18.908491    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-96ab73-6bf4699f9-ctm9s"]
Oct 30 13:39:19.087625 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:19.087590    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f3ee96f-30cf-4afe-a5a7-0fb69beaa873" path="/var/lib/kubelet/pods/3f3ee96f-30cf-4afe-a5a7-0fb69beaa873/volumes"
Oct 30 13:39:19.088114 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:19.088098    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94261b46-b158-4895-bfd1-21365792a055" path="/var/lib/kubelet/pods/94261b46-b158-4895-bfd1-21365792a055/volumes"
Oct 30 13:39:19.263305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:19.263170    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:20.807887 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:20.807834    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:22.548416 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:22.548362    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:22.668398 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:22.668362    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:22.846985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:22.845335    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:22.876510 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:22.876475    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"]
Oct 30 13:39:22.876719 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:22.876692    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz" podUID="4400fc6b-5d1c-4f3f-93be-7ef8540e8d79" containerName="istio-proxy" containerID="cri-o://2cb39dca7182c76aa1d05a555653c8aa056776ba7487c90250ed1b74ff785902" gracePeriod=30
Oct 30 13:39:23.016445 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.011456    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:23.171872 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.166255    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"]
Oct 30 13:39:23.171872 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.166622    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q" podUID="75391895-2b05-426d-98ad-87ad85d2e9c3" containerName="istio-proxy" containerID="cri-o://afa41dd1dabc3afefd67cdaf51460fce13ca39abb7b80869946180bd527ddedb" gracePeriod=30
Oct 30 13:39:23.174271 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.172810    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:23.692747 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.692692    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a71ca-jjq6c"]
Oct 30 13:39:23.693529 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693508    4544 cpu_manager.go:414] "containerMap: removing container" podUID="01e24b49-04cb-4af7-a9cc-a0b6fa608451" containerName="revoke-keys"
Oct 30 13:39:23.693581 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693530    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01e24b49-04cb-4af7-a9cc-a0b6fa608451" containerName="revoke-keys"
Oct 30 13:39:23.693581 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693548    4544 cpu_manager.go:414] "containerMap: removing container" podUID="8b6f4e05-5105-4596-b248-bc018f59ce81" containerName="istio-proxy"
Oct 30 13:39:23.693581 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693554    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b6f4e05-5105-4596-b248-bc018f59ce81" containerName="istio-proxy"
Oct 30 13:39:23.693581 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693561    4544 cpu_manager.go:414] "containerMap: removing container" podUID="128497c2-9f64-4902-a913-5acb1489b88b" containerName="istio-proxy"
Oct 30 13:39:23.693581 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693567    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="128497c2-9f64-4902-a913-5acb1489b88b" containerName="istio-proxy"
Oct 30 13:39:23.693581 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693578    4544 cpu_manager.go:414] "containerMap: removing container" podUID="62db3dba-ac93-4845-9e85-6d83a6fb0cb1" containerName="istio-proxy"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693583    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62db3dba-ac93-4845-9e85-6d83a6fb0cb1" containerName="istio-proxy"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693594    4544 cpu_manager.go:414] "containerMap: removing container" podUID="ee5e270c-f424-4803-992f-a49fbc67c6dd" containerName="revoke-keys"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693599    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee5e270c-f424-4803-992f-a49fbc67c6dd" containerName="revoke-keys"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693607    4544 cpu_manager.go:414] "containerMap: removing container" podUID="eb8eebdc-5933-463b-a511-264945e917e9" containerName="revoke-keys"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693611    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb8eebdc-5933-463b-a511-264945e917e9" containerName="revoke-keys"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693620    4544 cpu_manager.go:414] "containerMap: removing container" podUID="01e24b49-04cb-4af7-a9cc-a0b6fa608451" containerName="revoke-keys"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693624    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01e24b49-04cb-4af7-a9cc-a0b6fa608451" containerName="revoke-keys"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693633    4544 cpu_manager.go:414] "containerMap: removing container" podUID="d4fc9668-7314-46ee-bfef-6092e5b58d61" containerName="maas-api"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693638    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4fc9668-7314-46ee-bfef-6092e5b58d61" containerName="maas-api"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693644    4544 cpu_manager.go:414] "containerMap: removing container" podUID="7ab2ca89-081f-411d-8ec4-20a8d399010f" containerName="payload-pre-processing"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693649    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ab2ca89-081f-411d-8ec4-20a8d399010f" containerName="payload-pre-processing"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693662    4544 cpu_manager.go:414] "containerMap: removing container" podUID="68fa1f90-25f8-442f-96ff-12e6a716db84" containerName="istio-proxy"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693667    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68fa1f90-25f8-442f-96ff-12e6a716db84" containerName="istio-proxy"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693675    4544 cpu_manager.go:414] "containerMap: removing container" podUID="f44f3385-bd49-41d7-9add-718b415008f2" containerName="istio-proxy"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693679    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f44f3385-bd49-41d7-9add-718b415008f2" containerName="istio-proxy"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693684    4544 cpu_manager.go:414] "containerMap: removing container" podUID="ff60abd5-8f12-4e67-99f4-e6baeaee47ce" containerName="istio-proxy"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693690    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff60abd5-8f12-4e67-99f4-e6baeaee47ce" containerName="istio-proxy"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693699    4544 cpu_manager.go:414] "containerMap: removing container" podUID="e6aed883-80e6-44a5-ac18-855437d32862" containerName="istio-proxy"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693705    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6aed883-80e6-44a5-ac18-855437d32862" containerName="istio-proxy"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693713    4544 cpu_manager.go:414] "containerMap: removing container" podUID="820333af-678e-40c7-a1b7-24cab9591ea2" containerName="istio-proxy"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693726    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="820333af-678e-40c7-a1b7-24cab9591ea2" containerName="istio-proxy"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693734    4544 cpu_manager.go:414] "containerMap: removing container" podUID="820333af-678e-40c7-a1b7-24cab9591ea2" containerName="istio-proxy"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693739    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="820333af-678e-40c7-a1b7-24cab9591ea2" containerName="istio-proxy"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693746    4544 cpu_manager.go:414] "containerMap: removing container" podUID="94261b46-b158-4895-bfd1-21365792a055" containerName="revoke-keys"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693752    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94261b46-b158-4895-bfd1-21365792a055" containerName="revoke-keys"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693760    4544 cpu_manager.go:414] "containerMap: removing container" podUID="80b08e28-171e-49cd-bafe-0fbf4efc09e3" containerName="istio-proxy"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693764    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80b08e28-171e-49cd-bafe-0fbf4efc09e3" containerName="istio-proxy"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693773    4544 cpu_manager.go:414] "containerMap: removing container" podUID="68fa1f90-25f8-442f-96ff-12e6a716db84" containerName="istio-proxy"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693778    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68fa1f90-25f8-442f-96ff-12e6a716db84" containerName="istio-proxy"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693784    4544 cpu_manager.go:414] "containerMap: removing container" podUID="01fb99d1-307e-4917-bb84-b0276b6f78a7" containerName="maas-api"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693788    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01fb99d1-307e-4917-bb84-b0276b6f78a7" containerName="maas-api"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693794    4544 cpu_manager.go:414] "containerMap: removing container" podUID="45260af2-724d-4065-b966-542af12813a6" containerName="payload-pre-processing"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693799    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45260af2-724d-4065-b966-542af12813a6" containerName="payload-pre-processing"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693807    4544 cpu_manager.go:414] "containerMap: removing container" podUID="f90bdf7e-ba66-4f37-9585-5173e85566fb" containerName="istio-proxy"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693812    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f90bdf7e-ba66-4f37-9585-5173e85566fb" containerName="istio-proxy"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693817    4544 cpu_manager.go:414] "containerMap: removing container" podUID="e3559421-6424-4480-8d49-5ed4a11deb3d" containerName="istio-proxy"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693822    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3559421-6424-4480-8d49-5ed4a11deb3d" containerName="istio-proxy"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693828    4544 cpu_manager.go:414] "containerMap: removing container" podUID="7ef74cc1-eb94-4550-af70-38d4f4149963" containerName="istio-proxy"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693833    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ef74cc1-eb94-4550-af70-38d4f4149963" containerName="istio-proxy"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693845    4544 cpu_manager.go:414] "containerMap: removing container" podUID="8732b808-436c-4bf7-868e-1e934515e700" containerName="maas-api"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693850    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8732b808-436c-4bf7-868e-1e934515e700" containerName="maas-api"
Oct 30 13:39:23.693837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693856    4544 cpu_manager.go:414] "containerMap: removing container" podUID="347e2f12-0089-42aa-86b8-d42d02a56175" containerName="istio-proxy"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693861    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="347e2f12-0089-42aa-86b8-d42d02a56175" containerName="istio-proxy"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693868    4544 cpu_manager.go:414] "containerMap: removing container" podUID="fa3df092-b18c-47fb-8793-90f9869a1078" containerName="payload-pre-processing"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693872    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa3df092-b18c-47fb-8793-90f9869a1078" containerName="payload-pre-processing"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693880    4544 cpu_manager.go:414] "containerMap: removing container" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693884    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693891    4544 cpu_manager.go:414] "containerMap: removing container" podUID="5a60c9a2-4e23-4e4e-be07-09609b79f023" containerName="istio-proxy"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693895    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a60c9a2-4e23-4e4e-be07-09609b79f023" containerName="istio-proxy"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693903    4544 cpu_manager.go:414] "containerMap: removing container" podUID="aba97076-557c-4b61-9c1a-374341fbe152" containerName="payload-processing"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693907    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aba97076-557c-4b61-9c1a-374341fbe152" containerName="payload-processing"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693916    4544 cpu_manager.go:414] "containerMap: removing container" podUID="3f3ee96f-30cf-4afe-a5a7-0fb69beaa873" containerName="maas-api"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693921    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f3ee96f-30cf-4afe-a5a7-0fb69beaa873" containerName="maas-api"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693932    4544 cpu_manager.go:414] "containerMap: removing container" podUID="d232746e-9562-4578-b95f-09895eece465" containerName="payload-pre-processing"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693936    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d232746e-9562-4578-b95f-09895eece465" containerName="payload-pre-processing"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693947    4544 cpu_manager.go:414] "containerMap: removing container" podUID="eb8eebdc-5933-463b-a511-264945e917e9" containerName="revoke-keys"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.693951    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb8eebdc-5933-463b-a511-264945e917e9" containerName="revoke-keys"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.694052    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="01e24b49-04cb-4af7-a9cc-a0b6fa608451" containerName="revoke-keys"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.694061    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b6f4e05-5105-4596-b248-bc018f59ce81" containerName="istio-proxy"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.694070    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="128497c2-9f64-4902-a913-5acb1489b88b" containerName="istio-proxy"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.694076    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="62db3dba-ac93-4845-9e85-6d83a6fb0cb1" containerName="istio-proxy"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.694083    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee5e270c-f424-4803-992f-a49fbc67c6dd" containerName="revoke-keys"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.694089    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb8eebdc-5933-463b-a511-264945e917e9" containerName="revoke-keys"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.694094    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="01e24b49-04cb-4af7-a9cc-a0b6fa608451" containerName="revoke-keys"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.694100    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4fc9668-7314-46ee-bfef-6092e5b58d61" containerName="maas-api"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.694106    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ab2ca89-081f-411d-8ec4-20a8d399010f" containerName="payload-pre-processing"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.694112    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="68fa1f90-25f8-442f-96ff-12e6a716db84" containerName="istio-proxy"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.694118    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="f44f3385-bd49-41d7-9add-718b415008f2" containerName="istio-proxy"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.694123    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff60abd5-8f12-4e67-99f4-e6baeaee47ce" containerName="istio-proxy"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.694128    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6aed883-80e6-44a5-ac18-855437d32862" containerName="istio-proxy"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.694134    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="820333af-678e-40c7-a1b7-24cab9591ea2" containerName="istio-proxy"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.694139    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="820333af-678e-40c7-a1b7-24cab9591ea2" containerName="istio-proxy"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.694144    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="94261b46-b158-4895-bfd1-21365792a055" containerName="revoke-keys"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.694151    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="80b08e28-171e-49cd-bafe-0fbf4efc09e3" containerName="istio-proxy"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.694156    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="68fa1f90-25f8-442f-96ff-12e6a716db84" containerName="istio-proxy"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.694163    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="01fb99d1-307e-4917-bb84-b0276b6f78a7" containerName="maas-api"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.694169    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="45260af2-724d-4065-b966-542af12813a6" containerName="payload-pre-processing"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.694175    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="f90bdf7e-ba66-4f37-9585-5173e85566fb" containerName="istio-proxy"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.694179    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3559421-6424-4480-8d49-5ed4a11deb3d" containerName="istio-proxy"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.694184    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ef74cc1-eb94-4550-af70-38d4f4149963" containerName="istio-proxy"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.694190    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="8732b808-436c-4bf7-868e-1e934515e700" containerName="maas-api"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.694196    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="347e2f12-0089-42aa-86b8-d42d02a56175" containerName="istio-proxy"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.694202    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa3df092-b18c-47fb-8793-90f9869a1078" containerName="payload-pre-processing"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.694208    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="91c6b5c7-0c42-4a3b-95d3-ebd7fdac21e2" containerName="main"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.694214    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a60c9a2-4e23-4e4e-be07-09609b79f023" containerName="istio-proxy"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.694237    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="aba97076-557c-4b61-9c1a-374341fbe152" containerName="payload-processing"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.694250    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f3ee96f-30cf-4afe-a5a7-0fb69beaa873" containerName="maas-api"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.694261    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="d232746e-9562-4578-b95f-09895eece465" containerName="payload-pre-processing"
Oct 30 13:39:23.695110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.694270    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb8eebdc-5933-463b-a511-264945e917e9" containerName="revoke-keys"
Oct 30 13:39:23.699761 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.699729    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a71ca-jjq6c"
Oct 30 13:39:23.718967 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.718937    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a71ca-jjq6c"]
Oct 30 13:39:23.750146 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.750100    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng78h\" (UniqueName: \"kubernetes.io/projected/00cc4882-e29c-4ee7-8540-3ce2d8f34159-kube-api-access-ng78h\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-7a71ca-jjq6c\" (UID: \"00cc4882-e29c-4ee7-8540-3ce2d8f34159\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a71ca-jjq6c"
Oct 30 13:39:23.750418 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.750249    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00cc4882-e29c-4ee7-8540-3ce2d8f34159-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-7a71ca-jjq6c\" (UID: \"00cc4882-e29c-4ee7-8540-3ce2d8f34159\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a71ca-jjq6c"
Oct 30 13:39:23.750504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.750475    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00cc4882-e29c-4ee7-8540-3ce2d8f34159-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-7a71ca-jjq6c\" (UID: \"00cc4882-e29c-4ee7-8540-3ce2d8f34159\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a71ca-jjq6c"
Oct 30 13:39:23.803489 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.803449    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:39:23.803654 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.803534    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:39:23.851647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.851606    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00cc4882-e29c-4ee7-8540-3ce2d8f34159-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-7a71ca-jjq6c\" (UID: \"00cc4882-e29c-4ee7-8540-3ce2d8f34159\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a71ca-jjq6c"
Oct 30 13:39:23.851830 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.851754    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00cc4882-e29c-4ee7-8540-3ce2d8f34159-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-7a71ca-jjq6c\" (UID: \"00cc4882-e29c-4ee7-8540-3ce2d8f34159\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a71ca-jjq6c"
Oct 30 13:39:23.851891 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.851863    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng78h\" (UniqueName: \"kubernetes.io/projected/00cc4882-e29c-4ee7-8540-3ce2d8f34159-kube-api-access-ng78h\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-7a71ca-jjq6c\" (UID: \"00cc4882-e29c-4ee7-8540-3ce2d8f34159\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a71ca-jjq6c"
Oct 30 13:39:23.852453 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.852426    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00cc4882-e29c-4ee7-8540-3ce2d8f34159-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-7a71ca-jjq6c\" (UID: \"00cc4882-e29c-4ee7-8540-3ce2d8f34159\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a71ca-jjq6c"
Oct 30 13:39:23.854286 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.854252    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00cc4882-e29c-4ee7-8540-3ce2d8f34159-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-7a71ca-jjq6c\" (UID: \"00cc4882-e29c-4ee7-8540-3ce2d8f34159\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a71ca-jjq6c"
Oct 30 13:39:23.875110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:23.875077    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng78h\" (UniqueName: \"kubernetes.io/projected/00cc4882-e29c-4ee7-8540-3ce2d8f34159-kube-api-access-ng78h\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-7a71ca-jjq6c\" (UID: \"00cc4882-e29c-4ee7-8540-3ce2d8f34159\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a71ca-jjq6c"
Oct 30 13:39:24.015514 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:24.015419    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a71ca-jjq6c"
Oct 30 13:39:24.329301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:24.329270    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a71ca-jjq6c"]
Oct 30 13:39:24.331170 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:39:24.331142    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00cc4882_e29c_4ee7_8540_3ce2d8f34159.slice/crio-3604e0a06e46af3a19a9a7730f1badbac75060ecf405c82f51f4da4ac59d6356 WatchSource:0}: Error finding container 3604e0a06e46af3a19a9a7730f1badbac75060ecf405c82f51f4da4ac59d6356: Status 404 returned error can't find the container with id 3604e0a06e46af3a19a9a7730f1badbac75060ecf405c82f51f4da4ac59d6356
Oct 30 13:39:24.766610 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:24.766576    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:24.904563 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:24.904474    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a71ca-jjq6c" event={"ID":"00cc4882-e29c-4ee7-8540-3ce2d8f34159","Type":"ContainerStarted","Data":"5c2051dc7e4f8fd8e2770fdf391d6c4f9e61c3de44aa4d165b4dfb13ace5445d"}
Oct 30 13:39:24.904563 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:24.904528    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a71ca-jjq6c" event={"ID":"00cc4882-e29c-4ee7-8540-3ce2d8f34159","Type":"ContainerStarted","Data":"3604e0a06e46af3a19a9a7730f1badbac75060ecf405c82f51f4da4ac59d6356"}
Oct 30 13:39:24.949358 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:24.949299    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a71ca-jjq6c" podStartSLOduration=1.9492838300000002 podStartE2EDuration="1.94928383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:39:24.947746792 +0000 UTC m=+3259.994880745" watchObservedRunningTime="2026-10-30 13:39:24.94928383 +0000 UTC m=+3259.996417783"
Oct 30 13:39:25.913251 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:25.913194    4544 generic.go:356] "Generic (PLEG): container finished" podID="00cc4882-e29c-4ee7-8540-3ce2d8f34159" containerID="5c2051dc7e4f8fd8e2770fdf391d6c4f9e61c3de44aa4d165b4dfb13ace5445d" exitCode=0
Oct 30 13:39:25.913683 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:25.913258    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a71ca-jjq6c" event={"ID":"00cc4882-e29c-4ee7-8540-3ce2d8f34159","Type":"ContainerDied","Data":"5c2051dc7e4f8fd8e2770fdf391d6c4f9e61c3de44aa4d165b4dfb13ace5445d"}
Oct 30 13:39:26.317399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:26.317316    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq" podUID="8013c75e-2665-43ee-879b-ec14f338cca5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:39:26.317559 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:26.317433    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx" podUID="fb981ab3-c15c-48d3-868d-5d5d93616212" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:39:26.592323 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:26.592284    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m" podUID="611e617c-64ab-4006-972d-fa789281cc38" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:39:26.592636 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:26.592609    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h" podUID="329fbf95-10bc-4b2f-afc2-14c679d9edd8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:39:26.638506 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:26.638469    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:26.785404 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:26.785370    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:26.850453 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:26.850375    4544 patch_prober.go:29] interesting pod/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:39:26.850453 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:26.850434    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz" podUID="4400fc6b-5d1c-4f3f-93be-7ef8540e8d79" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:39:26.962419 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:26.956590    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"]
Oct 30 13:39:26.962419 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:26.956774    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd" podUID="c713af1f-1695-4ad5-a564-25420ba533a6" containerName="istio-proxy" containerID="cri-o://b2bafbd277d63f97ad6ce60e32d6f9749ab6f4bf470148aa97716768d9033831" gracePeriod=30
Oct 30 13:39:27.095197 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:27.095165    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:27.209165 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:27.209139    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a71ca-jjq6c"
Oct 30 13:39:27.342298 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:27.342262    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00cc4882-e29c-4ee7-8540-3ce2d8f34159-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00cc4882-e29c-4ee7-8540-3ce2d8f34159-maas-api-xks-ca\") pod \"00cc4882-e29c-4ee7-8540-3ce2d8f34159\" (UID: \"00cc4882-e29c-4ee7-8540-3ce2d8f34159\") "
Oct 30 13:39:27.342511 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:27.342382    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00cc4882-e29c-4ee7-8540-3ce2d8f34159-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00cc4882-e29c-4ee7-8540-3ce2d8f34159-maas-api-openshift-ca\") pod \"00cc4882-e29c-4ee7-8540-3ce2d8f34159\" (UID: \"00cc4882-e29c-4ee7-8540-3ce2d8f34159\") "
Oct 30 13:39:27.342511 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:27.342499    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00cc4882-e29c-4ee7-8540-3ce2d8f34159-kube-api-access-ng78h\" (UniqueName: \"kubernetes.io/projected/00cc4882-e29c-4ee7-8540-3ce2d8f34159-kube-api-access-ng78h\") pod \"00cc4882-e29c-4ee7-8540-3ce2d8f34159\" (UID: \"00cc4882-e29c-4ee7-8540-3ce2d8f34159\") "
Oct 30 13:39:27.342791 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:27.342762    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00cc4882-e29c-4ee7-8540-3ce2d8f34159-maas-api-openshift-ca" pod "00cc4882-e29c-4ee7-8540-3ce2d8f34159" (UID: "00cc4882-e29c-4ee7-8540-3ce2d8f34159"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:27.343097 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:27.343079    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00cc4882-e29c-4ee7-8540-3ce2d8f34159-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:27.344528 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:27.344501    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00cc4882-e29c-4ee7-8540-3ce2d8f34159-maas-api-xks-ca" pod "00cc4882-e29c-4ee7-8540-3ce2d8f34159" (UID: "00cc4882-e29c-4ee7-8540-3ce2d8f34159"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:39:27.344942 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:27.344913    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00cc4882-e29c-4ee7-8540-3ce2d8f34159-kube-api-access-ng78h" pod "00cc4882-e29c-4ee7-8540-3ce2d8f34159" (UID: "00cc4882-e29c-4ee7-8540-3ce2d8f34159"). InnerVolumeSpecName "kube-api-access-ng78h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:27.433777 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:27.433682    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:27.444456 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:27.444427    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00cc4882-e29c-4ee7-8540-3ce2d8f34159-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:27.444599 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:27.444459    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng78h\" (UniqueName: \"kubernetes.io/projected/00cc4882-e29c-4ee7-8540-3ce2d8f34159-kube-api-access-ng78h\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:27.624247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:27.615756    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:27.960008 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:27.959975    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a71ca-jjq6c" event={"ID":"00cc4882-e29c-4ee7-8540-3ce2d8f34159","Type":"ContainerDied","Data":"3604e0a06e46af3a19a9a7730f1badbac75060ecf405c82f51f4da4ac59d6356"}
Oct 30 13:39:27.960187 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:27.960015    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a71ca-jjq6c"
Oct 30 13:39:27.960187 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:27.960024    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3604e0a06e46af3a19a9a7730f1badbac75060ecf405c82f51f4da4ac59d6356"
Oct 30 13:39:28.150845 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.150819    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:39:28.255971 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.255940    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istiod-ca-cert\") pod \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") "
Oct 30 13:39:28.256183 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.255984    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istio-podinfo\") pod \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") "
Oct 30 13:39:28.256183 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.256128    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-kube-api-access-4nsbj\" (UniqueName: \"kubernetes.io/projected/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-kube-api-access-4nsbj\") pod \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") "
Oct 30 13:39:28.256183 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.256162    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-workload-certs\") pod \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") "
Oct 30 13:39:28.256423 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.256291    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istio-data\") pod \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") "
Oct 30 13:39:28.256423 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.256325    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-credential-socket\") pod \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") "
Oct 30 13:39:28.256423 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.256388    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istio-token\" (UniqueName: \"kubernetes.io/projected/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istio-token\") pod \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") "
Oct 30 13:39:28.256423 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.256392    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istiod-ca-cert" pod "4400fc6b-5d1c-4f3f-93be-7ef8540e8d79" (UID: "4400fc6b-5d1c-4f3f-93be-7ef8540e8d79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:28.256423 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.256422    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istio-envoy\") pod \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") "
Oct 30 13:39:28.256707 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.256526    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-workload-socket\") pod \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\" (UID: \"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79\") "
Oct 30 13:39:28.256775 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.256702    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istio-data" pod "4400fc6b-5d1c-4f3f-93be-7ef8540e8d79" (UID: "4400fc6b-5d1c-4f3f-93be-7ef8540e8d79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:28.256775 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.256747    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-workload-certs" pod "4400fc6b-5d1c-4f3f-93be-7ef8540e8d79" (UID: "4400fc6b-5d1c-4f3f-93be-7ef8540e8d79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:28.257395 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.256994    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-workload-socket" pod "4400fc6b-5d1c-4f3f-93be-7ef8540e8d79" (UID: "4400fc6b-5d1c-4f3f-93be-7ef8540e8d79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:28.257395 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.257309    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:28.257395 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.257330    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:28.257395 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.257346    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:28.257395 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.257359    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:28.258015 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.257989    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-credential-socket" pod "4400fc6b-5d1c-4f3f-93be-7ef8540e8d79" (UID: "4400fc6b-5d1c-4f3f-93be-7ef8540e8d79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:28.260402 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.260345    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istio-envoy" pod "4400fc6b-5d1c-4f3f-93be-7ef8540e8d79" (UID: "4400fc6b-5d1c-4f3f-93be-7ef8540e8d79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:28.260402 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.260363    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istio-podinfo" pod "4400fc6b-5d1c-4f3f-93be-7ef8540e8d79" (UID: "4400fc6b-5d1c-4f3f-93be-7ef8540e8d79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:39:28.260553 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.260404    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-kube-api-access-4nsbj" pod "4400fc6b-5d1c-4f3f-93be-7ef8540e8d79" (UID: "4400fc6b-5d1c-4f3f-93be-7ef8540e8d79"). InnerVolumeSpecName "kube-api-access-4nsbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:28.260822 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.260803    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istio-token" pod "4400fc6b-5d1c-4f3f-93be-7ef8540e8d79" (UID: "4400fc6b-5d1c-4f3f-93be-7ef8540e8d79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:28.360662 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.360024    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:28.360662 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.360069    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:28.360662 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.360084    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:28.360662 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.360096    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:28.360662 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.360109    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nsbj\" (UniqueName: \"kubernetes.io/projected/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79-kube-api-access-4nsbj\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:28.446530 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.446503    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:39:28.562187 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.562154    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75391895-2b05-426d-98ad-87ad85d2e9c3-kube-api-access-cx98c\" (UniqueName: \"kubernetes.io/projected/75391895-2b05-426d-98ad-87ad85d2e9c3-kube-api-access-cx98c\") pod \"75391895-2b05-426d-98ad-87ad85d2e9c3\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") "
Oct 30 13:39:28.562417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.562210    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-workload-certs\") pod \"75391895-2b05-426d-98ad-87ad85d2e9c3\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") "
Oct 30 13:39:28.562417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.562281    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-credential-socket\") pod \"75391895-2b05-426d-98ad-87ad85d2e9c3\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") "
Oct 30 13:39:28.562417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.562326    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-workload-socket\") pod \"75391895-2b05-426d-98ad-87ad85d2e9c3\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") "
Oct 30 13:39:28.562417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.562377    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75391895-2b05-426d-98ad-87ad85d2e9c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75391895-2b05-426d-98ad-87ad85d2e9c3-istio-podinfo\") pod \"75391895-2b05-426d-98ad-87ad85d2e9c3\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") "
Oct 30 13:39:28.562417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.562410    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75391895-2b05-426d-98ad-87ad85d2e9c3-istio-token\" (UniqueName: \"kubernetes.io/projected/75391895-2b05-426d-98ad-87ad85d2e9c3-istio-token\") pod \"75391895-2b05-426d-98ad-87ad85d2e9c3\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") "
Oct 30 13:39:28.562720 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.562554    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75391895-2b05-426d-98ad-87ad85d2e9c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75391895-2b05-426d-98ad-87ad85d2e9c3-istiod-ca-cert\") pod \"75391895-2b05-426d-98ad-87ad85d2e9c3\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") "
Oct 30 13:39:28.562720 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.562585    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-credential-socket" pod "75391895-2b05-426d-98ad-87ad85d2e9c3" (UID: "75391895-2b05-426d-98ad-87ad85d2e9c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:28.562720 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.562599    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-workload-certs" pod "75391895-2b05-426d-98ad-87ad85d2e9c3" (UID: "75391895-2b05-426d-98ad-87ad85d2e9c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:28.562720 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.562606    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-istio-envoy\") pod \"75391895-2b05-426d-98ad-87ad85d2e9c3\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") "
Oct 30 13:39:28.562720 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.562632    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-workload-socket" pod "75391895-2b05-426d-98ad-87ad85d2e9c3" (UID: "75391895-2b05-426d-98ad-87ad85d2e9c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:28.563025 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.562745    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-istio-data\") pod \"75391895-2b05-426d-98ad-87ad85d2e9c3\" (UID: \"75391895-2b05-426d-98ad-87ad85d2e9c3\") "
Oct 30 13:39:28.563025 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.562936    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75391895-2b05-426d-98ad-87ad85d2e9c3-istiod-ca-cert" pod "75391895-2b05-426d-98ad-87ad85d2e9c3" (UID: "75391895-2b05-426d-98ad-87ad85d2e9c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:28.563202 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.563168    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-istio-data" pod "75391895-2b05-426d-98ad-87ad85d2e9c3" (UID: "75391895-2b05-426d-98ad-87ad85d2e9c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:28.563536 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.563513    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:28.563631 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.563543    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:28.563631 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.563561    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:28.563631 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.563577    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75391895-2b05-426d-98ad-87ad85d2e9c3-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:28.563631 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.563591    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:28.564779 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.564745    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75391895-2b05-426d-98ad-87ad85d2e9c3-kube-api-access-cx98c" pod "75391895-2b05-426d-98ad-87ad85d2e9c3" (UID: "75391895-2b05-426d-98ad-87ad85d2e9c3"). InnerVolumeSpecName "kube-api-access-cx98c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:28.565059 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.565031    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-istio-envoy" pod "75391895-2b05-426d-98ad-87ad85d2e9c3" (UID: "75391895-2b05-426d-98ad-87ad85d2e9c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:28.565189 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.565103    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75391895-2b05-426d-98ad-87ad85d2e9c3-istio-token" pod "75391895-2b05-426d-98ad-87ad85d2e9c3" (UID: "75391895-2b05-426d-98ad-87ad85d2e9c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:28.565289 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.565265    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75391895-2b05-426d-98ad-87ad85d2e9c3-istio-podinfo" pod "75391895-2b05-426d-98ad-87ad85d2e9c3" (UID: "75391895-2b05-426d-98ad-87ad85d2e9c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:39:28.664442 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.664407    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75391895-2b05-426d-98ad-87ad85d2e9c3-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:28.664442 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.664437    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx98c\" (UniqueName: \"kubernetes.io/projected/75391895-2b05-426d-98ad-87ad85d2e9c3-kube-api-access-cx98c\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:28.664442 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.664449    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75391895-2b05-426d-98ad-87ad85d2e9c3-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:28.664795 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.664458    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75391895-2b05-426d-98ad-87ad85d2e9c3-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:28.683563 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.683477    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:28.823107 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.823070    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:28.970623 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.970523    4544 generic.go:356] "Generic (PLEG): container finished" podID="75391895-2b05-426d-98ad-87ad85d2e9c3" containerID="afa41dd1dabc3afefd67cdaf51460fce13ca39abb7b80869946180bd527ddedb" exitCode=0
Oct 30 13:39:28.970623 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.970582    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"
Oct 30 13:39:28.970623 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.970611    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q" event={"ID":"75391895-2b05-426d-98ad-87ad85d2e9c3","Type":"ContainerDied","Data":"afa41dd1dabc3afefd67cdaf51460fce13ca39abb7b80869946180bd527ddedb"}
Oct 30 13:39:28.970914 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.970694    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q" event={"ID":"75391895-2b05-426d-98ad-87ad85d2e9c3","Type":"ContainerDied","Data":"34f0c24244f7b06467b0d506e09e5ba197643c0b5c3994a964b1e9481a9078aa"}
Oct 30 13:39:28.970914 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.970728    4544 scope.go:122] "RemoveContainer" containerID="afa41dd1dabc3afefd67cdaf51460fce13ca39abb7b80869946180bd527ddedb"
Oct 30 13:39:28.972520 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.972496    4544 generic.go:356] "Generic (PLEG): container finished" podID="4400fc6b-5d1c-4f3f-93be-7ef8540e8d79" containerID="2cb39dca7182c76aa1d05a555653c8aa056776ba7487c90250ed1b74ff785902" exitCode=0
Oct 30 13:39:28.972661 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.972566    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"
Oct 30 13:39:28.972661 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.972578    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz" event={"ID":"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79","Type":"ContainerDied","Data":"2cb39dca7182c76aa1d05a555653c8aa056776ba7487c90250ed1b74ff785902"}
Oct 30 13:39:28.972661 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.972631    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz" event={"ID":"4400fc6b-5d1c-4f3f-93be-7ef8540e8d79","Type":"ContainerDied","Data":"74fe47361d6fb0176fbcd9d3f2931e1d4540b6ee551005f9538872d914e3d488"}
Oct 30 13:39:28.988387 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:28.985971    4544 scope.go:122] "RemoveContainer" containerID="4d2447981a522cbf7003cb6702442c0af55f422dd1521822112c515e9b06111c"
Oct 30 13:39:29.041618 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.036313    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"]
Oct 30 13:39:29.041618 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.036550    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt" podUID="4e66eb26-5727-4ad8-881f-64b6711fe440" containerName="istio-proxy" containerID="cri-o://b70ca727e64d4d8c689bfdf170699feeb415ee43f6b7336ed5fef90ec1d55285" gracePeriod=30
Oct 30 13:39:29.058406 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:39:29.043344    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:39:29.058406 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.043420    4544 scope.go:122] "RemoveContainer" containerID="afa41dd1dabc3afefd67cdaf51460fce13ca39abb7b80869946180bd527ddedb"
Oct 30 13:39:29.058406 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:39:29.055991    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afa41dd1dabc3afefd67cdaf51460fce13ca39abb7b80869946180bd527ddedb\": container with ID starting with afa41dd1dabc3afefd67cdaf51460fce13ca39abb7b80869946180bd527ddedb not found: ID does not exist" containerID="afa41dd1dabc3afefd67cdaf51460fce13ca39abb7b80869946180bd527ddedb"
Oct 30 13:39:29.058406 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.056020    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afa41dd1dabc3afefd67cdaf51460fce13ca39abb7b80869946180bd527ddedb"} err="failed to get container status \"afa41dd1dabc3afefd67cdaf51460fce13ca39abb7b80869946180bd527ddedb\": rpc error: code = NotFound desc = could not find container \"afa41dd1dabc3afefd67cdaf51460fce13ca39abb7b80869946180bd527ddedb\": container with ID starting with afa41dd1dabc3afefd67cdaf51460fce13ca39abb7b80869946180bd527ddedb not found: ID does not exist"
Oct 30 13:39:29.058406 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.056043    4544 scope.go:122] "RemoveContainer" containerID="4d2447981a522cbf7003cb6702442c0af55f422dd1521822112c515e9b06111c"
Oct 30 13:39:29.058406 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.056259    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:29.058406 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:39:29.056570    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d2447981a522cbf7003cb6702442c0af55f422dd1521822112c515e9b06111c\": container with ID starting with 4d2447981a522cbf7003cb6702442c0af55f422dd1521822112c515e9b06111c not found: ID does not exist" containerID="4d2447981a522cbf7003cb6702442c0af55f422dd1521822112c515e9b06111c"
Oct 30 13:39:29.058406 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.056608    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d2447981a522cbf7003cb6702442c0af55f422dd1521822112c515e9b06111c"} err="failed to get container status \"4d2447981a522cbf7003cb6702442c0af55f422dd1521822112c515e9b06111c\": rpc error: code = NotFound desc = could not find container \"4d2447981a522cbf7003cb6702442c0af55f422dd1521822112c515e9b06111c\": container with ID starting with 4d2447981a522cbf7003cb6702442c0af55f422dd1521822112c515e9b06111c not found: ID does not exist"
Oct 30 13:39:29.058406 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.056630    4544 scope.go:122] "RemoveContainer" containerID="2cb39dca7182c76aa1d05a555653c8aa056776ba7487c90250ed1b74ff785902"
Oct 30 13:39:29.072937 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.072912    4544 scope.go:122] "RemoveContainer" containerID="d6d4b0cad2e96d7bf655fc4d0c2d3d91862bd6ff7b91b1be789b66aeb860b672"
Oct 30 13:39:29.076457 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.074608    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"]
Oct 30 13:39:29.088395 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.088376    4544 scope.go:122] "RemoveContainer" containerID="2cb39dca7182c76aa1d05a555653c8aa056776ba7487c90250ed1b74ff785902"
Oct 30 13:39:29.088733 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:39:29.088710    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cb39dca7182c76aa1d05a555653c8aa056776ba7487c90250ed1b74ff785902\": container with ID starting with 2cb39dca7182c76aa1d05a555653c8aa056776ba7487c90250ed1b74ff785902 not found: ID does not exist" containerID="2cb39dca7182c76aa1d05a555653c8aa056776ba7487c90250ed1b74ff785902"
Oct 30 13:39:29.088835 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.088745    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cb39dca7182c76aa1d05a555653c8aa056776ba7487c90250ed1b74ff785902"} err="failed to get container status \"2cb39dca7182c76aa1d05a555653c8aa056776ba7487c90250ed1b74ff785902\": rpc error: code = NotFound desc = could not find container \"2cb39dca7182c76aa1d05a555653c8aa056776ba7487c90250ed1b74ff785902\": container with ID starting with 2cb39dca7182c76aa1d05a555653c8aa056776ba7487c90250ed1b74ff785902 not found: ID does not exist"
Oct 30 13:39:29.088835 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.088770    4544 scope.go:122] "RemoveContainer" containerID="d6d4b0cad2e96d7bf655fc4d0c2d3d91862bd6ff7b91b1be789b66aeb860b672"
Oct 30 13:39:29.089086 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:39:29.089062    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6d4b0cad2e96d7bf655fc4d0c2d3d91862bd6ff7b91b1be789b66aeb860b672\": container with ID starting with d6d4b0cad2e96d7bf655fc4d0c2d3d91862bd6ff7b91b1be789b66aeb860b672 not found: ID does not exist" containerID="d6d4b0cad2e96d7bf655fc4d0c2d3d91862bd6ff7b91b1be789b66aeb860b672"
Oct 30 13:39:29.089132 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.089095    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d4b0cad2e96d7bf655fc4d0c2d3d91862bd6ff7b91b1be789b66aeb860b672"} err="failed to get container status \"d6d4b0cad2e96d7bf655fc4d0c2d3d91862bd6ff7b91b1be789b66aeb860b672\": rpc error: code = NotFound desc = could not find container \"d6d4b0cad2e96d7bf655fc4d0c2d3d91862bd6ff7b91b1be789b66aeb860b672\": container with ID starting with d6d4b0cad2e96d7bf655fc4d0c2d3d91862bd6ff7b91b1be789b66aeb860b672 not found: ID does not exist"
Oct 30 13:39:29.091316 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.091297    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz"]
Oct 30 13:39:29.125966 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.117579    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"]
Oct 30 13:39:29.132203 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.130903    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q"]
Oct 30 13:39:29.193575 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.190800    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:29.341407 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.341373    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47"]
Oct 30 13:39:29.373402 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.373366    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn"]
Oct 30 13:39:29.492803 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.492738    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:29.592391 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.592292    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-8a5d5e-fck9q"]
Oct 30 13:39:29.593577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.593403    4544 cpu_manager.go:414] "containerMap: removing container" podUID="75391895-2b05-426d-98ad-87ad85d2e9c3" containerName="istio-proxy"
Oct 30 13:39:29.593577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.593429    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75391895-2b05-426d-98ad-87ad85d2e9c3" containerName="istio-proxy"
Oct 30 13:39:29.593577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.593448    4544 cpu_manager.go:414] "containerMap: removing container" podUID="01e24b49-04cb-4af7-a9cc-a0b6fa608451" containerName="revoke-keys"
Oct 30 13:39:29.593577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.593457    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01e24b49-04cb-4af7-a9cc-a0b6fa608451" containerName="revoke-keys"
Oct 30 13:39:29.593577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.593478    4544 cpu_manager.go:414] "containerMap: removing container" podUID="4400fc6b-5d1c-4f3f-93be-7ef8540e8d79" containerName="istio-proxy"
Oct 30 13:39:29.593577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.593522    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4400fc6b-5d1c-4f3f-93be-7ef8540e8d79" containerName="istio-proxy"
Oct 30 13:39:29.593577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.593546    4544 cpu_manager.go:414] "containerMap: removing container" podUID="4400fc6b-5d1c-4f3f-93be-7ef8540e8d79" containerName="istio-proxy"
Oct 30 13:39:29.593577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.593555    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4400fc6b-5d1c-4f3f-93be-7ef8540e8d79" containerName="istio-proxy"
Oct 30 13:39:29.593577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.593570    4544 cpu_manager.go:414] "containerMap: removing container" podUID="00cc4882-e29c-4ee7-8540-3ce2d8f34159" containerName="revoke-keys"
Oct 30 13:39:29.593577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.593577    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00cc4882-e29c-4ee7-8540-3ce2d8f34159" containerName="revoke-keys"
Oct 30 13:39:29.594587 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.593639    4544 cpu_manager.go:414] "containerMap: removing container" podUID="eb8eebdc-5933-463b-a511-264945e917e9" containerName="revoke-keys"
Oct 30 13:39:29.594587 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.593646    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb8eebdc-5933-463b-a511-264945e917e9" containerName="revoke-keys"
Oct 30 13:39:29.594587 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.593833    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="00cc4882-e29c-4ee7-8540-3ce2d8f34159" containerName="revoke-keys"
Oct 30 13:39:29.594587 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.593861    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb8eebdc-5933-463b-a511-264945e917e9" containerName="revoke-keys"
Oct 30 13:39:29.594587 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.593874    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="75391895-2b05-426d-98ad-87ad85d2e9c3" containerName="istio-proxy"
Oct 30 13:39:29.594587 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.593884    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="75391895-2b05-426d-98ad-87ad85d2e9c3" containerName="istio-proxy"
Oct 30 13:39:29.594587 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.593895    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="01e24b49-04cb-4af7-a9cc-a0b6fa608451" containerName="revoke-keys"
Oct 30 13:39:29.594587 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.593905    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="4400fc6b-5d1c-4f3f-93be-7ef8540e8d79" containerName="istio-proxy"
Oct 30 13:39:29.600318 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.600292    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-8a5d5e-fck9q"
Oct 30 13:39:29.612737 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.612707    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-8a5d5e-fck9q"]
Oct 30 13:39:29.615399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.615376    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:39:29.620303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.620269    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:29.620823 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.620794    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:39:29.693413 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.693376    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7cqk\" (UniqueName: \"kubernetes.io/projected/8dacf33b-a434-4657-bc05-71ebd72337dc-kube-api-access-h7cqk\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-8a5d5e-fck9q\" (UID: \"8dacf33b-a434-4657-bc05-71ebd72337dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-8a5d5e-fck9q"
Oct 30 13:39:29.693614 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.693471    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dacf33b-a434-4657-bc05-71ebd72337dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-8a5d5e-fck9q\" (UID: \"8dacf33b-a434-4657-bc05-71ebd72337dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-8a5d5e-fck9q"
Oct 30 13:39:29.693614 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.693544    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dacf33b-a434-4657-bc05-71ebd72337dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-8a5d5e-fck9q\" (UID: \"8dacf33b-a434-4657-bc05-71ebd72337dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-8a5d5e-fck9q"
Oct 30 13:39:29.741082 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.741046    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:39:29.741298 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.741110    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt" podUID="4e66eb26-5727-4ad8-881f-64b6711fe440" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:39:29.794740 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.794702    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dacf33b-a434-4657-bc05-71ebd72337dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-8a5d5e-fck9q\" (UID: \"8dacf33b-a434-4657-bc05-71ebd72337dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-8a5d5e-fck9q"
Oct 30 13:39:29.794964 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.794770    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dacf33b-a434-4657-bc05-71ebd72337dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-8a5d5e-fck9q\" (UID: \"8dacf33b-a434-4657-bc05-71ebd72337dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-8a5d5e-fck9q"
Oct 30 13:39:29.794964 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.794959    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7cqk\" (UniqueName: \"kubernetes.io/projected/8dacf33b-a434-4657-bc05-71ebd72337dc-kube-api-access-h7cqk\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-8a5d5e-fck9q\" (UID: \"8dacf33b-a434-4657-bc05-71ebd72337dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-8a5d5e-fck9q"
Oct 30 13:39:29.795502 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.795478    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dacf33b-a434-4657-bc05-71ebd72337dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-8a5d5e-fck9q\" (UID: \"8dacf33b-a434-4657-bc05-71ebd72337dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-8a5d5e-fck9q"
Oct 30 13:39:29.797213 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.797187    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dacf33b-a434-4657-bc05-71ebd72337dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-8a5d5e-fck9q\" (UID: \"8dacf33b-a434-4657-bc05-71ebd72337dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-8a5d5e-fck9q"
Oct 30 13:39:29.818397 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.818371    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7cqk\" (UniqueName: \"kubernetes.io/projected/8dacf33b-a434-4657-bc05-71ebd72337dc-kube-api-access-h7cqk\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-8a5d5e-fck9q\" (UID: \"8dacf33b-a434-4657-bc05-71ebd72337dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-8a5d5e-fck9q"
Oct 30 13:39:29.920589 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:29.920500    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-8a5d5e-fck9q"
Oct 30 13:39:30.253665 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:30.253561    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-8a5d5e-fck9q"]
Oct 30 13:39:30.256794 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:39:30.256764    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dacf33b_a434_4657_bc05_71ebd72337dc.slice/crio-649c685f4eabbace2eeef85e7245894ef68fb48ebeac510972bfe05fde16db7f WatchSource:0}: Error finding container 649c685f4eabbace2eeef85e7245894ef68fb48ebeac510972bfe05fde16db7f: Status 404 returned error can't find the container with id 649c685f4eabbace2eeef85e7245894ef68fb48ebeac510972bfe05fde16db7f
Oct 30 13:39:31.088967 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:31.088933    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4400fc6b-5d1c-4f3f-93be-7ef8540e8d79" path="/var/lib/kubelet/pods/4400fc6b-5d1c-4f3f-93be-7ef8540e8d79/volumes"
Oct 30 13:39:31.089504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:31.089488    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75391895-2b05-426d-98ad-87ad85d2e9c3" path="/var/lib/kubelet/pods/75391895-2b05-426d-98ad-87ad85d2e9c3/volumes"
Oct 30 13:39:31.089848 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:31.089830    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-8a5d5e-fck9q" event={"ID":"8dacf33b-a434-4657-bc05-71ebd72337dc","Type":"ContainerStarted","Data":"c280c7467610ee828ee7416e8c6536981431eadcb7606938c6c41253c020b8c5"}
Oct 30 13:39:31.089916 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:31.089875    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-8a5d5e-fck9q" event={"ID":"8dacf33b-a434-4657-bc05-71ebd72337dc","Type":"ContainerStarted","Data":"649c685f4eabbace2eeef85e7245894ef68fb48ebeac510972bfe05fde16db7f"}
Oct 30 13:39:31.140043 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:31.139985    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-8a5d5e-fck9q" podStartSLOduration=2.139967049 podStartE2EDuration="2.139967049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:39:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:39:31.132209967 +0000 UTC m=+3266.179343922" watchObservedRunningTime="2026-10-30 13:39:31.139967049 +0000 UTC m=+3266.187101011"
Oct 30 13:39:31.314640 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:31.314594    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:32.098369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.098335    4544 generic.go:356] "Generic (PLEG): container finished" podID="c713af1f-1695-4ad5-a564-25420ba533a6" containerID="b2bafbd277d63f97ad6ce60e32d6f9749ab6f4bf470148aa97716768d9033831" exitCode=0
Oct 30 13:39:32.098554 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.098416    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd" event={"ID":"c713af1f-1695-4ad5-a564-25420ba533a6","Type":"ContainerDied","Data":"b2bafbd277d63f97ad6ce60e32d6f9749ab6f4bf470148aa97716768d9033831"}
Oct 30 13:39:32.098554 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.098494    4544 scope.go:122] "RemoveContainer" containerID="89b11d4dfe81eb377fa0657c55a2f4f775b613b4b6f4a86b7e6a1f139cffb103"
Oct 30 13:39:32.100349 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.100326    4544 generic.go:356] "Generic (PLEG): container finished" podID="8dacf33b-a434-4657-bc05-71ebd72337dc" containerID="c280c7467610ee828ee7416e8c6536981431eadcb7606938c6c41253c020b8c5" exitCode=0
Oct 30 13:39:32.100494 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.100389    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-8a5d5e-fck9q" event={"ID":"8dacf33b-a434-4657-bc05-71ebd72337dc","Type":"ContainerDied","Data":"c280c7467610ee828ee7416e8c6536981431eadcb7606938c6c41253c020b8c5"}
Oct 30 13:39:32.245465 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.245438    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:39:32.342156 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.342116    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c713af1f-1695-4ad5-a564-25420ba533a6-istio-token\" (UniqueName: \"kubernetes.io/projected/c713af1f-1695-4ad5-a564-25420ba533a6-istio-token\") pod \"c713af1f-1695-4ad5-a564-25420ba533a6\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") "
Oct 30 13:39:32.342699 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.342183    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c713af1f-1695-4ad5-a564-25420ba533a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c713af1f-1695-4ad5-a564-25420ba533a6-istiod-ca-cert\") pod \"c713af1f-1695-4ad5-a564-25420ba533a6\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") "
Oct 30 13:39:32.342699 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.342347    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c713af1f-1695-4ad5-a564-25420ba533a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c713af1f-1695-4ad5-a564-25420ba533a6-istio-podinfo\") pod \"c713af1f-1695-4ad5-a564-25420ba533a6\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") "
Oct 30 13:39:32.342699 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.342419    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-istio-data\") pod \"c713af1f-1695-4ad5-a564-25420ba533a6\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") "
Oct 30 13:39:32.342699 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.342518    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-workload-socket\") pod \"c713af1f-1695-4ad5-a564-25420ba533a6\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") "
Oct 30 13:39:32.342699 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.342609    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c713af1f-1695-4ad5-a564-25420ba533a6-kube-api-access-w5zj6\" (UniqueName: \"kubernetes.io/projected/c713af1f-1695-4ad5-a564-25420ba533a6-kube-api-access-w5zj6\") pod \"c713af1f-1695-4ad5-a564-25420ba533a6\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") "
Oct 30 13:39:32.343017 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.342733    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-credential-socket\") pod \"c713af1f-1695-4ad5-a564-25420ba533a6\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") "
Oct 30 13:39:32.343017 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.342750    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c713af1f-1695-4ad5-a564-25420ba533a6-istiod-ca-cert" pod "c713af1f-1695-4ad5-a564-25420ba533a6" (UID: "c713af1f-1695-4ad5-a564-25420ba533a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:32.343017 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.342829    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-workload-certs\") pod \"c713af1f-1695-4ad5-a564-25420ba533a6\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") "
Oct 30 13:39:32.343017 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.342864    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-istio-envoy\") pod \"c713af1f-1695-4ad5-a564-25420ba533a6\" (UID: \"c713af1f-1695-4ad5-a564-25420ba533a6\") "
Oct 30 13:39:32.343274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.343043    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-workload-socket" pod "c713af1f-1695-4ad5-a564-25420ba533a6" (UID: "c713af1f-1695-4ad5-a564-25420ba533a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:32.343274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.343248    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-istio-data" pod "c713af1f-1695-4ad5-a564-25420ba533a6" (UID: "c713af1f-1695-4ad5-a564-25420ba533a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:32.343585 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.343562    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c713af1f-1695-4ad5-a564-25420ba533a6-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:32.343667 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.343591    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:32.343667 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.343606    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:32.343808 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.343774    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-workload-certs" pod "c713af1f-1695-4ad5-a564-25420ba533a6" (UID: "c713af1f-1695-4ad5-a564-25420ba533a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:32.343888 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.343862    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-credential-socket" pod "c713af1f-1695-4ad5-a564-25420ba533a6" (UID: "c713af1f-1695-4ad5-a564-25420ba533a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:32.349793 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.349361    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-istio-envoy" pod "c713af1f-1695-4ad5-a564-25420ba533a6" (UID: "c713af1f-1695-4ad5-a564-25420ba533a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:32.349793 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.349420    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c713af1f-1695-4ad5-a564-25420ba533a6-istio-token" pod "c713af1f-1695-4ad5-a564-25420ba533a6" (UID: "c713af1f-1695-4ad5-a564-25420ba533a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:32.349793 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.349455    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c713af1f-1695-4ad5-a564-25420ba533a6-kube-api-access-w5zj6" pod "c713af1f-1695-4ad5-a564-25420ba533a6" (UID: "c713af1f-1695-4ad5-a564-25420ba533a6"). InnerVolumeSpecName "kube-api-access-w5zj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:32.349793 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.349452    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c713af1f-1695-4ad5-a564-25420ba533a6-istio-podinfo" pod "c713af1f-1695-4ad5-a564-25420ba533a6" (UID: "c713af1f-1695-4ad5-a564-25420ba533a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:39:32.445541 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.445504    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c713af1f-1695-4ad5-a564-25420ba533a6-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:32.445541 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.445539    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5zj6\" (UniqueName: \"kubernetes.io/projected/c713af1f-1695-4ad5-a564-25420ba533a6-kube-api-access-w5zj6\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:32.445730 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.445549    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:32.445730 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.445559    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:32.445730 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.445567    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c713af1f-1695-4ad5-a564-25420ba533a6-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:32.445730 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.445575    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c713af1f-1695-4ad5-a564-25420ba533a6-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:32.626889 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.626856    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx"
Oct 30 13:39:32.649143 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.649107    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb981ab3-c15c-48d3-868d-5d5d93616212-kube-api-access-fcbvs\" (UniqueName: \"kubernetes.io/projected/fb981ab3-c15c-48d3-868d-5d5d93616212-kube-api-access-fcbvs\") pod \"fb981ab3-c15c-48d3-868d-5d5d93616212\" (UID: \"fb981ab3-c15c-48d3-868d-5d5d93616212\") "
Oct 30 13:39:32.649350 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.649164    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb981ab3-c15c-48d3-868d-5d5d93616212-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb981ab3-c15c-48d3-868d-5d5d93616212-plugins-config-volume\") pod \"fb981ab3-c15c-48d3-868d-5d5d93616212\" (UID: \"fb981ab3-c15c-48d3-868d-5d5d93616212\") "
Oct 30 13:39:32.649530 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.649508    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb981ab3-c15c-48d3-868d-5d5d93616212-plugins-config-volume" pod "fb981ab3-c15c-48d3-868d-5d5d93616212" (UID: "fb981ab3-c15c-48d3-868d-5d5d93616212"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:32.649789 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.649772    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb981ab3-c15c-48d3-868d-5d5d93616212-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:32.651559 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.651529    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb981ab3-c15c-48d3-868d-5d5d93616212-kube-api-access-fcbvs" pod "fb981ab3-c15c-48d3-868d-5d5d93616212" (UID: "fb981ab3-c15c-48d3-868d-5d5d93616212"). InnerVolumeSpecName "kube-api-access-fcbvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:32.675489 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.675463    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq"
Oct 30 13:39:32.751051 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.750950    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8013c75e-2665-43ee-879b-ec14f338cca5-kube-api-access-d4k24\" (UniqueName: \"kubernetes.io/projected/8013c75e-2665-43ee-879b-ec14f338cca5-kube-api-access-d4k24\") pod \"8013c75e-2665-43ee-879b-ec14f338cca5\" (UID: \"8013c75e-2665-43ee-879b-ec14f338cca5\") "
Oct 30 13:39:32.751270 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.751114    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8013c75e-2665-43ee-879b-ec14f338cca5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8013c75e-2665-43ee-879b-ec14f338cca5-plugins-config-volume\") pod \"8013c75e-2665-43ee-879b-ec14f338cca5\" (UID: \"8013c75e-2665-43ee-879b-ec14f338cca5\") "
Oct 30 13:39:32.751536 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.751508    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8013c75e-2665-43ee-879b-ec14f338cca5-plugins-config-volume" pod "8013c75e-2665-43ee-879b-ec14f338cca5" (UID: "8013c75e-2665-43ee-879b-ec14f338cca5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:32.751706 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.751678    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcbvs\" (UniqueName: \"kubernetes.io/projected/fb981ab3-c15c-48d3-868d-5d5d93616212-kube-api-access-fcbvs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:32.751706 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.751707    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8013c75e-2665-43ee-879b-ec14f338cca5-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:32.753736 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.753701    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8013c75e-2665-43ee-879b-ec14f338cca5-kube-api-access-d4k24" pod "8013c75e-2665-43ee-879b-ec14f338cca5" (UID: "8013c75e-2665-43ee-879b-ec14f338cca5"). InnerVolumeSpecName "kube-api-access-d4k24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:32.852978 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:32.852936    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4k24\" (UniqueName: \"kubernetes.io/projected/8013c75e-2665-43ee-879b-ec14f338cca5-kube-api-access-d4k24\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:33.109503 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:33.109480    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"
Oct 30 13:39:33.109651 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:33.109524    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd" event={"ID":"c713af1f-1695-4ad5-a564-25420ba533a6","Type":"ContainerDied","Data":"d46ca10012c4d4bb329228b670d7d9cfc03f6103aefeef2e06b797c58a8be510"}
Oct 30 13:39:33.109651 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:33.109594    4544 scope.go:122] "RemoveContainer" containerID="b2bafbd277d63f97ad6ce60e32d6f9749ab6f4bf470148aa97716768d9033831"
Oct 30 13:39:33.111250 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:33.111205    4544 generic.go:356] "Generic (PLEG): container finished" podID="fb981ab3-c15c-48d3-868d-5d5d93616212" containerID="56127a11104d74fee2e2b51d20306dc9198826a0132bcc9c92fa180264c09248" exitCode=0
Oct 30 13:39:33.111451 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:33.111275    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx"
Oct 30 13:39:33.111568 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:33.111279    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx" event={"ID":"fb981ab3-c15c-48d3-868d-5d5d93616212","Type":"ContainerDied","Data":"56127a11104d74fee2e2b51d20306dc9198826a0132bcc9c92fa180264c09248"}
Oct 30 13:39:33.113149 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:33.113129    4544 generic.go:356] "Generic (PLEG): container finished" podID="8013c75e-2665-43ee-879b-ec14f338cca5" containerID="94ed8da1a2b8104aeb2efa3559ca628109c55dcf45e01b0ee5dee61aba172e25" exitCode=0
Oct 30 13:39:33.113308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:33.113282    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq"
Oct 30 13:39:33.114869 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:33.114839    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx" event={"ID":"fb981ab3-c15c-48d3-868d-5d5d93616212","Type":"ContainerDied","Data":"ca17c04994dee5c5072f826613f8b19ab88afb6ee002290287db0c31535be803"}
Oct 30 13:39:33.115029 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:33.114885    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq" event={"ID":"8013c75e-2665-43ee-879b-ec14f338cca5","Type":"ContainerDied","Data":"94ed8da1a2b8104aeb2efa3559ca628109c55dcf45e01b0ee5dee61aba172e25"}
Oct 30 13:39:33.115029 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:33.114910    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq" event={"ID":"8013c75e-2665-43ee-879b-ec14f338cca5","Type":"ContainerDied","Data":"c16bc3a94bf86ed694c957d536f92719c07cfb44c85d963d3c0870abb90cad70"}
Oct 30 13:39:33.123850 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:33.123824    4544 scope.go:122] "RemoveContainer" containerID="56127a11104d74fee2e2b51d20306dc9198826a0132bcc9c92fa180264c09248"
Oct 30 13:39:33.134170 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:33.134147    4544 scope.go:122] "RemoveContainer" containerID="56127a11104d74fee2e2b51d20306dc9198826a0132bcc9c92fa180264c09248"
Oct 30 13:39:33.134561 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:39:33.134530    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56127a11104d74fee2e2b51d20306dc9198826a0132bcc9c92fa180264c09248\": container with ID starting with 56127a11104d74fee2e2b51d20306dc9198826a0132bcc9c92fa180264c09248 not found: ID does not exist" containerID="56127a11104d74fee2e2b51d20306dc9198826a0132bcc9c92fa180264c09248"
Oct 30 13:39:33.134644 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:33.134566    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56127a11104d74fee2e2b51d20306dc9198826a0132bcc9c92fa180264c09248"} err="failed to get container status \"56127a11104d74fee2e2b51d20306dc9198826a0132bcc9c92fa180264c09248\": rpc error: code = NotFound desc = could not find container \"56127a11104d74fee2e2b51d20306dc9198826a0132bcc9c92fa180264c09248\": container with ID starting with 56127a11104d74fee2e2b51d20306dc9198826a0132bcc9c92fa180264c09248 not found: ID does not exist"
Oct 30 13:39:33.134644 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:33.134590    4544 scope.go:122] "RemoveContainer" containerID="94ed8da1a2b8104aeb2efa3559ca628109c55dcf45e01b0ee5dee61aba172e25"
Oct 30 13:39:33.175401 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:33.170524    4544 scope.go:122] "RemoveContainer" containerID="94ed8da1a2b8104aeb2efa3559ca628109c55dcf45e01b0ee5dee61aba172e25"
Oct 30 13:39:33.182347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:33.182278    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx"]
Oct 30 13:39:33.187287 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:33.187215    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx"]
Oct 30 13:39:33.187639 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:39:33.187602    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94ed8da1a2b8104aeb2efa3559ca628109c55dcf45e01b0ee5dee61aba172e25\": container with ID starting with 94ed8da1a2b8104aeb2efa3559ca628109c55dcf45e01b0ee5dee61aba172e25 not found: ID does not exist" containerID="94ed8da1a2b8104aeb2efa3559ca628109c55dcf45e01b0ee5dee61aba172e25"
Oct 30 13:39:33.187796 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:33.187649    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94ed8da1a2b8104aeb2efa3559ca628109c55dcf45e01b0ee5dee61aba172e25"} err="failed to get container status \"94ed8da1a2b8104aeb2efa3559ca628109c55dcf45e01b0ee5dee61aba172e25\": rpc error: code = NotFound desc = could not find container \"94ed8da1a2b8104aeb2efa3559ca628109c55dcf45e01b0ee5dee61aba172e25\": container with ID starting with 94ed8da1a2b8104aeb2efa3559ca628109c55dcf45e01b0ee5dee61aba172e25 not found: ID does not exist"
Oct 30 13:39:33.249993 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:33.249956    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"]
Oct 30 13:39:33.269574 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:33.269527    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd"]
Oct 30 13:39:33.280475 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:33.280439    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:33.311001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:33.310971    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq"]
Oct 30 13:39:33.321058 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:33.321031    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-8a5d5e-fck9q"
Oct 30 13:39:33.322142 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:33.322118    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq"]
Oct 30 13:39:33.367013 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:33.366207    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dacf33b-a434-4657-bc05-71ebd72337dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dacf33b-a434-4657-bc05-71ebd72337dc-maas-api-openshift-ca\") pod \"8dacf33b-a434-4657-bc05-71ebd72337dc\" (UID: \"8dacf33b-a434-4657-bc05-71ebd72337dc\") "
Oct 30 13:39:33.367013 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:33.366339    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8dacf33b-a434-4657-bc05-71ebd72337dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dacf33b-a434-4657-bc05-71ebd72337dc-maas-api-xks-ca\") pod \"8dacf33b-a434-4657-bc05-71ebd72337dc\" (UID: \"8dacf33b-a434-4657-bc05-71ebd72337dc\") "
Oct 30 13:39:33.367013 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:33.366377    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dacf33b-a434-4657-bc05-71ebd72337dc-kube-api-access-h7cqk\" (UniqueName: \"kubernetes.io/projected/8dacf33b-a434-4657-bc05-71ebd72337dc-kube-api-access-h7cqk\") pod \"8dacf33b-a434-4657-bc05-71ebd72337dc\" (UID: \"8dacf33b-a434-4657-bc05-71ebd72337dc\") "
Oct 30 13:39:33.367657 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:33.367322    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dacf33b-a434-4657-bc05-71ebd72337dc-maas-api-openshift-ca" pod "8dacf33b-a434-4657-bc05-71ebd72337dc" (UID: "8dacf33b-a434-4657-bc05-71ebd72337dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:33.369502 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:33.369463    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dacf33b-a434-4657-bc05-71ebd72337dc-kube-api-access-h7cqk" pod "8dacf33b-a434-4657-bc05-71ebd72337dc" (UID: "8dacf33b-a434-4657-bc05-71ebd72337dc"). InnerVolumeSpecName "kube-api-access-h7cqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:33.369642 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:33.369628    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dacf33b-a434-4657-bc05-71ebd72337dc-maas-api-xks-ca" pod "8dacf33b-a434-4657-bc05-71ebd72337dc" (UID: "8dacf33b-a434-4657-bc05-71ebd72337dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:39:33.468065 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:33.468030    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dacf33b-a434-4657-bc05-71ebd72337dc-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:33.468065 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:33.468060    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dacf33b-a434-4657-bc05-71ebd72337dc-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:33.468065 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:33.468072    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7cqk\" (UniqueName: \"kubernetes.io/projected/8dacf33b-a434-4657-bc05-71ebd72337dc-kube-api-access-h7cqk\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:34.046084 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.046051    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb"]
Oct 30 13:39:34.127193 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.127160    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-8a5d5e-fck9q" event={"ID":"8dacf33b-a434-4657-bc05-71ebd72337dc","Type":"ContainerDied","Data":"649c685f4eabbace2eeef85e7245894ef68fb48ebeac510972bfe05fde16db7f"}
Oct 30 13:39:34.127412 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.127208    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="649c685f4eabbace2eeef85e7245894ef68fb48ebeac510972bfe05fde16db7f"
Oct 30 13:39:34.127412 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.127283    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-8a5d5e-fck9q"
Oct 30 13:39:34.219721 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.219687    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:34.342065 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.342010    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47" podUID="eb920234-9adb-4f7f-a5e4-27faa3603144" containerName="payload-processing" containerID="cri-o://ae684698bd369cffd22ee43a4f483c486f6b6a8c187dcac89bc133c35ade3ec9" gracePeriod=25
Oct 30 13:39:34.344470 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.344249    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:34.372609 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.372582    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:39:34.374273 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.374245    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn" podUID="f6d9c074-fc98-4507-822d-7ca79b3770c5" containerName="payload-pre-processing" containerID="cri-o://9a974fb0f58ac4c61fd08b97b7b563886ccc90cc6d32bb856b4c41192bf34794" gracePeriod=25
Oct 30 13:39:34.482347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.482309    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e66eb26-5727-4ad8-881f-64b6711fe440-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e66eb26-5727-4ad8-881f-64b6711fe440-istio-podinfo\") pod \"4e66eb26-5727-4ad8-881f-64b6711fe440\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") "
Oct 30 13:39:34.482563 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.482363    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-istio-envoy\") pod \"4e66eb26-5727-4ad8-881f-64b6711fe440\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") "
Oct 30 13:39:34.482563 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.482448    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e66eb26-5727-4ad8-881f-64b6711fe440-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e66eb26-5727-4ad8-881f-64b6711fe440-istiod-ca-cert\") pod \"4e66eb26-5727-4ad8-881f-64b6711fe440\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") "
Oct 30 13:39:34.482563 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.482474    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-workload-certs\") pod \"4e66eb26-5727-4ad8-881f-64b6711fe440\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") "
Oct 30 13:39:34.482563 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.482536    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-istio-data\") pod \"4e66eb26-5727-4ad8-881f-64b6711fe440\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") "
Oct 30 13:39:34.482817 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.482575    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-credential-socket\") pod \"4e66eb26-5727-4ad8-881f-64b6711fe440\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") "
Oct 30 13:39:34.482817 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.482667    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e66eb26-5727-4ad8-881f-64b6711fe440-kube-api-access-m7cg4\" (UniqueName: \"kubernetes.io/projected/4e66eb26-5727-4ad8-881f-64b6711fe440-kube-api-access-m7cg4\") pod \"4e66eb26-5727-4ad8-881f-64b6711fe440\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") "
Oct 30 13:39:34.482817 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.482723    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-workload-socket\") pod \"4e66eb26-5727-4ad8-881f-64b6711fe440\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") "
Oct 30 13:39:34.482817 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.482796    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e66eb26-5727-4ad8-881f-64b6711fe440-istio-token\" (UniqueName: \"kubernetes.io/projected/4e66eb26-5727-4ad8-881f-64b6711fe440-istio-token\") pod \"4e66eb26-5727-4ad8-881f-64b6711fe440\" (UID: \"4e66eb26-5727-4ad8-881f-64b6711fe440\") "
Oct 30 13:39:34.483053 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.482854    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-workload-certs" pod "4e66eb26-5727-4ad8-881f-64b6711fe440" (UID: "4e66eb26-5727-4ad8-881f-64b6711fe440"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:34.483053 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.482851    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-istio-data" pod "4e66eb26-5727-4ad8-881f-64b6711fe440" (UID: "4e66eb26-5727-4ad8-881f-64b6711fe440"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:34.483053 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.482885    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-credential-socket" pod "4e66eb26-5727-4ad8-881f-64b6711fe440" (UID: "4e66eb26-5727-4ad8-881f-64b6711fe440"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:34.483245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.483044    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-workload-socket" pod "4e66eb26-5727-4ad8-881f-64b6711fe440" (UID: "4e66eb26-5727-4ad8-881f-64b6711fe440"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:34.483321 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.483239    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e66eb26-5727-4ad8-881f-64b6711fe440-istiod-ca-cert" pod "4e66eb26-5727-4ad8-881f-64b6711fe440" (UID: "4e66eb26-5727-4ad8-881f-64b6711fe440"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:34.483537 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.483515    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:34.483626 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.483540    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e66eb26-5727-4ad8-881f-64b6711fe440-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:34.483626 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.483555    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:34.483626 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.483569    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:34.483626 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.483581    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:34.484783 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.484754    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e66eb26-5727-4ad8-881f-64b6711fe440-istio-podinfo" pod "4e66eb26-5727-4ad8-881f-64b6711fe440" (UID: "4e66eb26-5727-4ad8-881f-64b6711fe440"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:39:34.484942 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.484919    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-istio-envoy" pod "4e66eb26-5727-4ad8-881f-64b6711fe440" (UID: "4e66eb26-5727-4ad8-881f-64b6711fe440"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:34.485012 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.484955    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e66eb26-5727-4ad8-881f-64b6711fe440-kube-api-access-m7cg4" pod "4e66eb26-5727-4ad8-881f-64b6711fe440" (UID: "4e66eb26-5727-4ad8-881f-64b6711fe440"). InnerVolumeSpecName "kube-api-access-m7cg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:34.485055 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.485004    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e66eb26-5727-4ad8-881f-64b6711fe440-istio-token" pod "4e66eb26-5727-4ad8-881f-64b6711fe440" (UID: "4e66eb26-5727-4ad8-881f-64b6711fe440"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:34.585100 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.585064    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7cg4\" (UniqueName: \"kubernetes.io/projected/4e66eb26-5727-4ad8-881f-64b6711fe440-kube-api-access-m7cg4\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:34.585100 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.585095    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e66eb26-5727-4ad8-881f-64b6711fe440-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:34.585100 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.585105    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e66eb26-5727-4ad8-881f-64b6711fe440-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:34.585368 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:34.585113    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e66eb26-5727-4ad8-881f-64b6711fe440-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:35.026646 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.026601    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:35.109363 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.109325    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8013c75e-2665-43ee-879b-ec14f338cca5" path="/var/lib/kubelet/pods/8013c75e-2665-43ee-879b-ec14f338cca5/volumes"
Oct 30 13:39:35.109848 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.109831    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c713af1f-1695-4ad5-a564-25420ba533a6" path="/var/lib/kubelet/pods/c713af1f-1695-4ad5-a564-25420ba533a6/volumes"
Oct 30 13:39:35.110250 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.110236    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb981ab3-c15c-48d3-868d-5d5d93616212" path="/var/lib/kubelet/pods/fb981ab3-c15c-48d3-868d-5d5d93616212/volumes"
Oct 30 13:39:35.139332 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.138884    4544 generic.go:356] "Generic (PLEG): container finished" podID="4e66eb26-5727-4ad8-881f-64b6711fe440" containerID="b70ca727e64d4d8c689bfdf170699feeb415ee43f6b7336ed5fef90ec1d55285" exitCode=0
Oct 30 13:39:35.139332 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.138960    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt" event={"ID":"4e66eb26-5727-4ad8-881f-64b6711fe440","Type":"ContainerDied","Data":"b70ca727e64d4d8c689bfdf170699feeb415ee43f6b7336ed5fef90ec1d55285"}
Oct 30 13:39:35.139332 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.139024    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt" event={"ID":"4e66eb26-5727-4ad8-881f-64b6711fe440","Type":"ContainerDied","Data":"a275c8e06afd8d6ff095545a5abb025b7fb6f079593934867ab7bc6fe57c73c1"}
Oct 30 13:39:35.139332 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.139057    4544 scope.go:122] "RemoveContainer" containerID="b70ca727e64d4d8c689bfdf170699feeb415ee43f6b7336ed5fef90ec1d55285"
Oct 30 13:39:35.139872 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.139850    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"
Oct 30 13:39:35.142711 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.142492    4544 generic.go:356] "Generic (PLEG): container finished" podID="611e617c-64ab-4006-972d-fa789281cc38" containerID="526813699e40d766f1321b213aaceeb866a0070b92310b5f5662b449af998e01" exitCode=0
Oct 30 13:39:35.142711 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.142615    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m" event={"ID":"611e617c-64ab-4006-972d-fa789281cc38","Type":"ContainerDied","Data":"526813699e40d766f1321b213aaceeb866a0070b92310b5f5662b449af998e01"}
Oct 30 13:39:35.189204 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.189116    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:35.191074 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.191042    4544 generic.go:356] "Generic (PLEG): container finished" podID="329fbf95-10bc-4b2f-afc2-14c679d9edd8" containerID="552f1749d883c7e3d623a150d03d8efc63459572b4d8bffe2bf42a057fab6afd" exitCode=0
Oct 30 13:39:35.191212 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.191118    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h" event={"ID":"329fbf95-10bc-4b2f-afc2-14c679d9edd8","Type":"ContainerDied","Data":"552f1749d883c7e3d623a150d03d8efc63459572b4d8bffe2bf42a057fab6afd"}
Oct 30 13:39:35.209674 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.209498    4544 scope.go:122] "RemoveContainer" containerID="af461ba3c237d09537277a0321974f745885b1401311091e4ff9e15a5550f79a"
Oct 30 13:39:35.217649 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.217600    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"]
Oct 30 13:39:35.223453 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.223427    4544 scope.go:122] "RemoveContainer" containerID="b70ca727e64d4d8c689bfdf170699feeb415ee43f6b7336ed5fef90ec1d55285"
Oct 30 13:39:35.223789 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:39:35.223768    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b70ca727e64d4d8c689bfdf170699feeb415ee43f6b7336ed5fef90ec1d55285\": container with ID starting with b70ca727e64d4d8c689bfdf170699feeb415ee43f6b7336ed5fef90ec1d55285 not found: ID does not exist" containerID="b70ca727e64d4d8c689bfdf170699feeb415ee43f6b7336ed5fef90ec1d55285"
Oct 30 13:39:35.223862 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.223809    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b70ca727e64d4d8c689bfdf170699feeb415ee43f6b7336ed5fef90ec1d55285"} err="failed to get container status \"b70ca727e64d4d8c689bfdf170699feeb415ee43f6b7336ed5fef90ec1d55285\": rpc error: code = NotFound desc = could not find container \"b70ca727e64d4d8c689bfdf170699feeb415ee43f6b7336ed5fef90ec1d55285\": container with ID starting with b70ca727e64d4d8c689bfdf170699feeb415ee43f6b7336ed5fef90ec1d55285 not found: ID does not exist"
Oct 30 13:39:35.223862 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.223828    4544 scope.go:122] "RemoveContainer" containerID="af461ba3c237d09537277a0321974f745885b1401311091e4ff9e15a5550f79a"
Oct 30 13:39:35.224090 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:39:35.224074    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af461ba3c237d09537277a0321974f745885b1401311091e4ff9e15a5550f79a\": container with ID starting with af461ba3c237d09537277a0321974f745885b1401311091e4ff9e15a5550f79a not found: ID does not exist" containerID="af461ba3c237d09537277a0321974f745885b1401311091e4ff9e15a5550f79a"
Oct 30 13:39:35.224146 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.224100    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af461ba3c237d09537277a0321974f745885b1401311091e4ff9e15a5550f79a"} err="failed to get container status \"af461ba3c237d09537277a0321974f745885b1401311091e4ff9e15a5550f79a\": rpc error: code = NotFound desc = could not find container \"af461ba3c237d09537277a0321974f745885b1401311091e4ff9e15a5550f79a\": container with ID starting with af461ba3c237d09537277a0321974f745885b1401311091e4ff9e15a5550f79a not found: ID does not exist"
Oct 30 13:39:35.230422 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.230065    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt"]
Oct 30 13:39:35.303513 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.303474    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h"
Oct 30 13:39:35.346558 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.346531    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m"
Oct 30 13:39:35.405172 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.405142    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/329fbf95-10bc-4b2f-afc2-14c679d9edd8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/329fbf95-10bc-4b2f-afc2-14c679d9edd8-plugins-config-volume\") pod \"329fbf95-10bc-4b2f-afc2-14c679d9edd8\" (UID: \"329fbf95-10bc-4b2f-afc2-14c679d9edd8\") "
Oct 30 13:39:35.405670 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.405382    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/329fbf95-10bc-4b2f-afc2-14c679d9edd8-kube-api-access-sgp6l\" (UniqueName: \"kubernetes.io/projected/329fbf95-10bc-4b2f-afc2-14c679d9edd8-kube-api-access-sgp6l\") pod \"329fbf95-10bc-4b2f-afc2-14c679d9edd8\" (UID: \"329fbf95-10bc-4b2f-afc2-14c679d9edd8\") "
Oct 30 13:39:35.405670 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.405422    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/611e617c-64ab-4006-972d-fa789281cc38-kube-api-access-4xt6x\" (UniqueName: \"kubernetes.io/projected/611e617c-64ab-4006-972d-fa789281cc38-kube-api-access-4xt6x\") pod \"611e617c-64ab-4006-972d-fa789281cc38\" (UID: \"611e617c-64ab-4006-972d-fa789281cc38\") "
Oct 30 13:39:35.405670 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.405524    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/611e617c-64ab-4006-972d-fa789281cc38-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/611e617c-64ab-4006-972d-fa789281cc38-plugins-config-volume\") pod \"611e617c-64ab-4006-972d-fa789281cc38\" (UID: \"611e617c-64ab-4006-972d-fa789281cc38\") "
Oct 30 13:39:35.405670 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.405581    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/329fbf95-10bc-4b2f-afc2-14c679d9edd8-plugins-config-volume" pod "329fbf95-10bc-4b2f-afc2-14c679d9edd8" (UID: "329fbf95-10bc-4b2f-afc2-14c679d9edd8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:35.406034 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.406010    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/329fbf95-10bc-4b2f-afc2-14c679d9edd8-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:35.406168 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.406028    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/611e617c-64ab-4006-972d-fa789281cc38-plugins-config-volume" pod "611e617c-64ab-4006-972d-fa789281cc38" (UID: "611e617c-64ab-4006-972d-fa789281cc38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:35.407662 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.407638    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/611e617c-64ab-4006-972d-fa789281cc38-kube-api-access-4xt6x" pod "611e617c-64ab-4006-972d-fa789281cc38" (UID: "611e617c-64ab-4006-972d-fa789281cc38"). InnerVolumeSpecName "kube-api-access-4xt6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:35.407747 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.407661    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/329fbf95-10bc-4b2f-afc2-14c679d9edd8-kube-api-access-sgp6l" pod "329fbf95-10bc-4b2f-afc2-14c679d9edd8" (UID: "329fbf95-10bc-4b2f-afc2-14c679d9edd8"). InnerVolumeSpecName "kube-api-access-sgp6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:35.507347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.507245    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgp6l\" (UniqueName: \"kubernetes.io/projected/329fbf95-10bc-4b2f-afc2-14c679d9edd8-kube-api-access-sgp6l\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:35.507347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.507283    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xt6x\" (UniqueName: \"kubernetes.io/projected/611e617c-64ab-4006-972d-fa789281cc38-kube-api-access-4xt6x\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:35.507347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:35.507299    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/611e617c-64ab-4006-972d-fa789281cc38-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:36.201136 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:36.201104    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m"
Oct 30 13:39:36.201372 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:36.201133    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m" event={"ID":"611e617c-64ab-4006-972d-fa789281cc38","Type":"ContainerDied","Data":"14bcd1d0c1c39d110888b39465badad2ce57d8c8ca40eb24d049ebf8089093bd"}
Oct 30 13:39:36.201372 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:36.201210    4544 scope.go:122] "RemoveContainer" containerID="526813699e40d766f1321b213aaceeb866a0070b92310b5f5662b449af998e01"
Oct 30 13:39:36.202641 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:36.202626    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h"
Oct 30 13:39:36.202760 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:36.202674    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h" event={"ID":"329fbf95-10bc-4b2f-afc2-14c679d9edd8","Type":"ContainerDied","Data":"b6aaffbba92aa7ff5fd0671241d717770317bf76fbc92231c1a03ca631c8a730"}
Oct 30 13:39:36.212150 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:36.211980    4544 scope.go:122] "RemoveContainer" containerID="552f1749d883c7e3d623a150d03d8efc63459572b4d8bffe2bf42a057fab6afd"
Oct 30 13:39:36.254735 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:36.254704    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h"]
Oct 30 13:39:36.266696 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:36.266664    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h"]
Oct 30 13:39:36.294758 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:36.294725    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m"]
Oct 30 13:39:36.300675 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:36.300645    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m"]
Oct 30 13:39:36.890979 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:36.890944    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:37.088895 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:37.088853    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="329fbf95-10bc-4b2f-afc2-14c679d9edd8" path="/var/lib/kubelet/pods/329fbf95-10bc-4b2f-afc2-14c679d9edd8/volumes"
Oct 30 13:39:37.089389 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:37.089374    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e66eb26-5727-4ad8-881f-64b6711fe440" path="/var/lib/kubelet/pods/4e66eb26-5727-4ad8-881f-64b6711fe440/volumes"
Oct 30 13:39:37.089787 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:37.089772    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="611e617c-64ab-4006-972d-fa789281cc38" path="/var/lib/kubelet/pods/611e617c-64ab-4006-972d-fa789281cc38/volumes"
Oct 30 13:39:38.021939 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:38.021899    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn" podUID="f6d9c074-fc98-4507-822d-7ca79b3770c5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:39:38.022379 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:38.021909    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47" podUID="eb920234-9adb-4f7f-a5e4-27faa3603144" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:39:38.193458 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:38.193423    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p"]
Oct 30 13:39:38.230633 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:38.230603    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7"]
Oct 30 13:39:38.335595 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:38.335556    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:38.460832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:38.460791    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:38.851374 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:38.851336    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:39.046505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:39.046468    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb" podUID="f673c81d-7be1-4c6c-bd5c-904136739702" containerName="payload-processing" containerID="cri-o://c434a75ca1235f1fb4b4e289578aa4aeb4f96d3713cb50ed2a49ada207c025c6" gracePeriod=25
Oct 30 13:39:40.046498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:40.046463    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss"]
Oct 30 13:39:40.197549 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:40.197510    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565"]
Oct 30 13:39:40.344559 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:40.344510    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:40.482334 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:40.482297    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:40.821505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:40.821416    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:42.539696 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:42.539659    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:43.194214 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:43.194166    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p" podUID="276d8bab-6921-4d6d-a482-258fbcbca9b7" containerName="payload-processing" containerID="cri-o://507152f2d87a669374635d0ea2c82180f38d4aff04ada125aff000ca05732f05" gracePeriod=25
Oct 30 13:39:43.231653 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:43.231614    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7" podUID="2beb5b6e-8222-48ce-b46b-b90e478277ed" containerName="payload-pre-processing" containerID="cri-o://56b51e3a0198e871c6f7c6774b900ad7c42b56a5e29064554ce173f8dbecc2ca" gracePeriod=25
Oct 30 13:39:44.417557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:44.417513    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:45.047485 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:45.047426    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss" podUID="6df66313-8135-4c91-a387-e7f654c305b3" containerName="payload-processing" containerID="cri-o://6127dbb1dca4c9019a739a59b3933d00ee6b4c3b39138042277c4e7cbed1c58f" gracePeriod=25
Oct 30 13:39:45.198178 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:45.198125    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565" podUID="e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6" containerName="payload-pre-processing" containerID="cri-o://cda9cfa5cedb6520e5c106b6c5dc7b1c48f0a02707aa8af26d1fdf7c2394d80e" gracePeriod=25
Oct 30 13:39:45.275939 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:45.275904    4544 generic.go:356] "Generic (PLEG): container finished" podID="6df66313-8135-4c91-a387-e7f654c305b3" containerID="6127dbb1dca4c9019a739a59b3933d00ee6b4c3b39138042277c4e7cbed1c58f" exitCode=0
Oct 30 13:39:45.276138 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:45.275976    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss" event={"ID":"6df66313-8135-4c91-a387-e7f654c305b3","Type":"ContainerDied","Data":"6127dbb1dca4c9019a739a59b3933d00ee6b4c3b39138042277c4e7cbed1c58f"}
Oct 30 13:39:45.331208 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:45.331180    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss"
Oct 30 13:39:45.448985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:45.448946    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6df66313-8135-4c91-a387-e7f654c305b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df66313-8135-4c91-a387-e7f654c305b3-plugins-config-volume\") pod \"6df66313-8135-4c91-a387-e7f654c305b3\" (UID: \"6df66313-8135-4c91-a387-e7f654c305b3\") "
Oct 30 13:39:45.449484 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:45.449122    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6df66313-8135-4c91-a387-e7f654c305b3-kube-api-access-hlh69\" (UniqueName: \"kubernetes.io/projected/6df66313-8135-4c91-a387-e7f654c305b3-kube-api-access-hlh69\") pod \"6df66313-8135-4c91-a387-e7f654c305b3\" (UID: \"6df66313-8135-4c91-a387-e7f654c305b3\") "
Oct 30 13:39:45.449484 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:45.449368    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df66313-8135-4c91-a387-e7f654c305b3-plugins-config-volume" pod "6df66313-8135-4c91-a387-e7f654c305b3" (UID: "6df66313-8135-4c91-a387-e7f654c305b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:45.449613 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:45.449574    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6df66313-8135-4c91-a387-e7f654c305b3-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:45.451466 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:45.451442    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df66313-8135-4c91-a387-e7f654c305b3-kube-api-access-hlh69" pod "6df66313-8135-4c91-a387-e7f654c305b3" (UID: "6df66313-8135-4c91-a387-e7f654c305b3"). InnerVolumeSpecName "kube-api-access-hlh69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:45.550973 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:45.550891    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlh69\" (UniqueName: \"kubernetes.io/projected/6df66313-8135-4c91-a387-e7f654c305b3-kube-api-access-hlh69\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:46.242568 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:46.242529    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:46.316558 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:46.316527    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss"
Oct 30 13:39:46.316752 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:46.316553    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss" event={"ID":"6df66313-8135-4c91-a387-e7f654c305b3","Type":"ContainerDied","Data":"abd5b09a1a47f441279ae55c32f94902579b5922c779ad403310bbe6a6f98be6"}
Oct 30 13:39:46.316752 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:46.316628    4544 scope.go:122] "RemoveContainer" containerID="6127dbb1dca4c9019a739a59b3933d00ee6b4c3b39138042277c4e7cbed1c58f"
Oct 30 13:39:46.369893 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:46.369860    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss"]
Oct 30 13:39:46.381446 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:46.381412    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss"]
Oct 30 13:39:46.940899 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:46.940808    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565" podUID="e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:39:47.089250 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:47.089182    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6df66313-8135-4c91-a387-e7f654c305b3" path="/var/lib/kubelet/pods/6df66313-8135-4c91-a387-e7f654c305b3/volumes"
Oct 30 13:39:47.561621 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:47.561585    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:47.758504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:47.758464    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:47.883164 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:47.883127    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7" podUID="2beb5b6e-8222-48ce-b46b-b90e478277ed" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:39:48.013904 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:48.013863    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb" podUID="f673c81d-7be1-4c6c-bd5c-904136739702" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:39:48.021534 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:48.021492    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn" podUID="f6d9c074-fc98-4507-822d-7ca79b3770c5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:39:48.021712 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:48.021545    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47" podUID="eb920234-9adb-4f7f-a5e4-27faa3603144" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:39:48.502836 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:48.502802    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:48.944204 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:48.944161    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p" podUID="276d8bab-6921-4d6d-a482-258fbcbca9b7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:39:48.977149 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:48.977110    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:49.108251 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:49.108203    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:49.240929 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:49.240837    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:49.749969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:49.749911    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:49.873367 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:49.873332    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:50.001409 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:50.001305    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:50.441897 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:50.441852    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:51.872524 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:51.872481    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:53.334556 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:53.334506    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:53.492392 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:53.492349    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:53.906427 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:53.906390    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:54.022187 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:54.022153    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:54.050515 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:54.050483    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a71ca-jjq6c"]
Oct 30 13:39:54.062402 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:54.062368    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a71ca-jjq6c"]
Oct 30 13:39:54.144129 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:54.144088    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:54.387181 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:54.387147    4544 generic.go:356] "Generic (PLEG): container finished" podID="eb920234-9adb-4f7f-a5e4-27faa3603144" containerID="ae684698bd369cffd22ee43a4f483c486f6b6a8c187dcac89bc133c35ade3ec9" exitCode=0
Oct 30 13:39:54.387605 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:54.387234    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47" event={"ID":"eb920234-9adb-4f7f-a5e4-27faa3603144","Type":"ContainerDied","Data":"ae684698bd369cffd22ee43a4f483c486f6b6a8c187dcac89bc133c35ade3ec9"}
Oct 30 13:39:54.534874 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:54.534531    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47"
Oct 30 13:39:54.594023 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:54.593987    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:54.652105 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:54.652081    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn"
Oct 30 13:39:54.678578 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:54.678539    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb920234-9adb-4f7f-a5e4-27faa3603144-kube-api-access-ddlsr\" (UniqueName: \"kubernetes.io/projected/eb920234-9adb-4f7f-a5e4-27faa3603144-kube-api-access-ddlsr\") pod \"eb920234-9adb-4f7f-a5e4-27faa3603144\" (UID: \"eb920234-9adb-4f7f-a5e4-27faa3603144\") "
Oct 30 13:39:54.678784 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:54.678676    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb920234-9adb-4f7f-a5e4-27faa3603144-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb920234-9adb-4f7f-a5e4-27faa3603144-plugins-config-volume\") pod \"eb920234-9adb-4f7f-a5e4-27faa3603144\" (UID: \"eb920234-9adb-4f7f-a5e4-27faa3603144\") "
Oct 30 13:39:54.679133 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:54.679104    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb920234-9adb-4f7f-a5e4-27faa3603144-plugins-config-volume" pod "eb920234-9adb-4f7f-a5e4-27faa3603144" (UID: "eb920234-9adb-4f7f-a5e4-27faa3603144"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:54.679421 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:54.679403    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb920234-9adb-4f7f-a5e4-27faa3603144-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:54.680859 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:54.680835    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb920234-9adb-4f7f-a5e4-27faa3603144-kube-api-access-ddlsr" pod "eb920234-9adb-4f7f-a5e4-27faa3603144" (UID: "eb920234-9adb-4f7f-a5e4-27faa3603144"). InnerVolumeSpecName "kube-api-access-ddlsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:54.780001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:54.779966    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6d9c074-fc98-4507-822d-7ca79b3770c5-kube-api-access-blc6p\" (UniqueName: \"kubernetes.io/projected/f6d9c074-fc98-4507-822d-7ca79b3770c5-kube-api-access-blc6p\") pod \"f6d9c074-fc98-4507-822d-7ca79b3770c5\" (UID: \"f6d9c074-fc98-4507-822d-7ca79b3770c5\") "
Oct 30 13:39:54.780180 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:54.780169    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6d9c074-fc98-4507-822d-7ca79b3770c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6d9c074-fc98-4507-822d-7ca79b3770c5-plugins-config-volume\") pod \"f6d9c074-fc98-4507-822d-7ca79b3770c5\" (UID: \"f6d9c074-fc98-4507-822d-7ca79b3770c5\") "
Oct 30 13:39:54.780565 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:54.780534    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6d9c074-fc98-4507-822d-7ca79b3770c5-plugins-config-volume" pod "f6d9c074-fc98-4507-822d-7ca79b3770c5" (UID: "f6d9c074-fc98-4507-822d-7ca79b3770c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:54.780694 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:54.780592    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddlsr\" (UniqueName: \"kubernetes.io/projected/eb920234-9adb-4f7f-a5e4-27faa3603144-kube-api-access-ddlsr\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:54.782210 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:54.782184    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6d9c074-fc98-4507-822d-7ca79b3770c5-kube-api-access-blc6p" pod "f6d9c074-fc98-4507-822d-7ca79b3770c5" (UID: "f6d9c074-fc98-4507-822d-7ca79b3770c5"). InnerVolumeSpecName "kube-api-access-blc6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:54.882618 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:54.882529    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6d9c074-fc98-4507-822d-7ca79b3770c5-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:54.882618 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:54.882562    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blc6p\" (UniqueName: \"kubernetes.io/projected/f6d9c074-fc98-4507-822d-7ca79b3770c5-kube-api-access-blc6p\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:55.087787 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:55.087748    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00cc4882-e29c-4ee7-8540-3ce2d8f34159" path="/var/lib/kubelet/pods/00cc4882-e29c-4ee7-8540-3ce2d8f34159/volumes"
Oct 30 13:39:55.396717 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:55.396689    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47"
Oct 30 13:39:55.397216 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:55.396712    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47" event={"ID":"eb920234-9adb-4f7f-a5e4-27faa3603144","Type":"ContainerDied","Data":"0f7f0a73b20c944968985f36e6f0402d0451bd37e3d380b0b4c5f55587b0ac3d"}
Oct 30 13:39:55.397216 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:55.396810    4544 scope.go:122] "RemoveContainer" containerID="ae684698bd369cffd22ee43a4f483c486f6b6a8c187dcac89bc133c35ade3ec9"
Oct 30 13:39:55.398453 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:55.398427    4544 generic.go:356] "Generic (PLEG): container finished" podID="f6d9c074-fc98-4507-822d-7ca79b3770c5" containerID="9a974fb0f58ac4c61fd08b97b7b563886ccc90cc6d32bb856b4c41192bf34794" exitCode=0
Oct 30 13:39:55.398566 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:55.398492    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn"
Oct 30 13:39:55.398566 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:55.398519    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn" event={"ID":"f6d9c074-fc98-4507-822d-7ca79b3770c5","Type":"ContainerDied","Data":"9a974fb0f58ac4c61fd08b97b7b563886ccc90cc6d32bb856b4c41192bf34794"}
Oct 30 13:39:55.398653 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:55.398586    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn" event={"ID":"f6d9c074-fc98-4507-822d-7ca79b3770c5","Type":"ContainerDied","Data":"293950139e6a0b3706e938d8468598c17fb0c1fdd8718280c5985691db979f5c"}
Oct 30 13:39:55.407762 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:55.407739    4544 scope.go:122] "RemoveContainer" containerID="9a974fb0f58ac4c61fd08b97b7b563886ccc90cc6d32bb856b4c41192bf34794"
Oct 30 13:39:55.418784 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:55.418760    4544 scope.go:122] "RemoveContainer" containerID="9a974fb0f58ac4c61fd08b97b7b563886ccc90cc6d32bb856b4c41192bf34794"
Oct 30 13:39:55.419137 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:39:55.419114    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a974fb0f58ac4c61fd08b97b7b563886ccc90cc6d32bb856b4c41192bf34794\": container with ID starting with 9a974fb0f58ac4c61fd08b97b7b563886ccc90cc6d32bb856b4c41192bf34794 not found: ID does not exist" containerID="9a974fb0f58ac4c61fd08b97b7b563886ccc90cc6d32bb856b4c41192bf34794"
Oct 30 13:39:55.419187 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:55.419149    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a974fb0f58ac4c61fd08b97b7b563886ccc90cc6d32bb856b4c41192bf34794"} err="failed to get container status \"9a974fb0f58ac4c61fd08b97b7b563886ccc90cc6d32bb856b4c41192bf34794\": rpc error: code = NotFound desc = could not find container \"9a974fb0f58ac4c61fd08b97b7b563886ccc90cc6d32bb856b4c41192bf34794\": container with ID starting with 9a974fb0f58ac4c61fd08b97b7b563886ccc90cc6d32bb856b4c41192bf34794 not found: ID does not exist"
Oct 30 13:39:55.451189 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:55.451148    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47"]
Oct 30 13:39:55.462593 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:55.462559    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47"]
Oct 30 13:39:55.494760 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:55.494727    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn"]
Oct 30 13:39:55.498284 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:55.498249    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn"]
Oct 30 13:39:55.802442 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:55.802359    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:56.291046 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:56.291003    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb"]
Oct 30 13:39:56.317469 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:56.317398    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb" podUID="0824f178-4f12-4b6c-8609-4c953e28d039" containerName="maas-api" containerID="cri-o://a29d9f05976fecb3a4dd4c22e4962a0fae470b1850b962433bd4adcdbaf52428" gracePeriod=30
Oct 30 13:39:56.379140 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:56.379107    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:56.501881 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:56.501844    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:56.621475 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:56.621438    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:56.643598 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:56.643558    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-8a5d5e-fck9q"]
Oct 30 13:39:56.646816 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:56.646793    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb"
Oct 30 13:39:56.647056 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:56.647035    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-8a5d5e-fck9q"]
Oct 30 13:39:56.723290 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:56.722765    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0824f178-4f12-4b6c-8609-4c953e28d039-kube-api-access-vptbb\" (UniqueName: \"kubernetes.io/projected/0824f178-4f12-4b6c-8609-4c953e28d039-kube-api-access-vptbb\") pod \"0824f178-4f12-4b6c-8609-4c953e28d039\" (UID: \"0824f178-4f12-4b6c-8609-4c953e28d039\") "
Oct 30 13:39:56.723290 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:56.722853    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0824f178-4f12-4b6c-8609-4c953e28d039-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0824f178-4f12-4b6c-8609-4c953e28d039-maas-api-tls\") pod \"0824f178-4f12-4b6c-8609-4c953e28d039\" (UID: \"0824f178-4f12-4b6c-8609-4c953e28d039\") "
Oct 30 13:39:56.723290 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:56.722898    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0824f178-4f12-4b6c-8609-4c953e28d039-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0824f178-4f12-4b6c-8609-4c953e28d039-openshift-service-ca\") pod \"0824f178-4f12-4b6c-8609-4c953e28d039\" (UID: \"0824f178-4f12-4b6c-8609-4c953e28d039\") "
Oct 30 13:39:56.723290 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:56.723082    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0824f178-4f12-4b6c-8609-4c953e28d039-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0824f178-4f12-4b6c-8609-4c953e28d039-metrics-certs\") pod \"0824f178-4f12-4b6c-8609-4c953e28d039\" (UID: \"0824f178-4f12-4b6c-8609-4c953e28d039\") "
Oct 30 13:39:56.740144 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:56.740078    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0824f178-4f12-4b6c-8609-4c953e28d039-openshift-service-ca" pod "0824f178-4f12-4b6c-8609-4c953e28d039" (UID: "0824f178-4f12-4b6c-8609-4c953e28d039"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:56.797436 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:56.797371    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0824f178-4f12-4b6c-8609-4c953e28d039-kube-api-access-vptbb" pod "0824f178-4f12-4b6c-8609-4c953e28d039" (UID: "0824f178-4f12-4b6c-8609-4c953e28d039"). InnerVolumeSpecName "kube-api-access-vptbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:56.799825 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:56.799793    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0824f178-4f12-4b6c-8609-4c953e28d039-metrics-certs" pod "0824f178-4f12-4b6c-8609-4c953e28d039" (UID: "0824f178-4f12-4b6c-8609-4c953e28d039"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:39:56.807289 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:56.802322    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0824f178-4f12-4b6c-8609-4c953e28d039-maas-api-tls" pod "0824f178-4f12-4b6c-8609-4c953e28d039" (UID: "0824f178-4f12-4b6c-8609-4c953e28d039"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:39:56.826282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:56.826242    4544 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0824f178-4f12-4b6c-8609-4c953e28d039-metrics-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:56.826573 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:56.826551    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vptbb\" (UniqueName: \"kubernetes.io/projected/0824f178-4f12-4b6c-8609-4c953e28d039-kube-api-access-vptbb\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:56.826732 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:56.826716    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0824f178-4f12-4b6c-8609-4c953e28d039-maas-api-tls\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:56.826867 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:56.826853    4544 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0824f178-4f12-4b6c-8609-4c953e28d039-openshift-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:56.914159 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:56.914060    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:56.940470 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:56.940433    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565" podUID="e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:39:57.038784 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:57.038743    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b1a6c9a-jmx47"]
Oct 30 13:39:57.046806 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:57.046773    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b1a6c9a-jmx47"]
Oct 30 13:39:57.088042 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:57.087714    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dacf33b-a434-4657-bc05-71ebd72337dc" path="/var/lib/kubelet/pods/8dacf33b-a434-4657-bc05-71ebd72337dc/volumes"
Oct 30 13:39:57.088410 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:57.088388    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb920234-9adb-4f7f-a5e4-27faa3603144" path="/var/lib/kubelet/pods/eb920234-9adb-4f7f-a5e4-27faa3603144/volumes"
Oct 30 13:39:57.088829 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:57.088810    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6d9c074-fc98-4507-822d-7ca79b3770c5" path="/var/lib/kubelet/pods/f6d9c074-fc98-4507-822d-7ca79b3770c5/volumes"
Oct 30 13:39:57.089583 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:57.089561    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fef0affe-c298-4c61-9740-fe7218c39162" path="/var/lib/kubelet/pods/fef0affe-c298-4c61-9740-fe7218c39162/volumes"
Oct 30 13:39:57.177783 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:57.177689    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:57.285480 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:57.285435    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:57.416643 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:57.416602    4544 generic.go:356] "Generic (PLEG): container finished" podID="0824f178-4f12-4b6c-8609-4c953e28d039" containerID="a29d9f05976fecb3a4dd4c22e4962a0fae470b1850b962433bd4adcdbaf52428" exitCode=0
Oct 30 13:39:57.416826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:57.416657    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb"
Oct 30 13:39:57.416826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:57.416688    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb" event={"ID":"0824f178-4f12-4b6c-8609-4c953e28d039","Type":"ContainerDied","Data":"a29d9f05976fecb3a4dd4c22e4962a0fae470b1850b962433bd4adcdbaf52428"}
Oct 30 13:39:57.416826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:57.416741    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb" event={"ID":"0824f178-4f12-4b6c-8609-4c953e28d039","Type":"ContainerDied","Data":"011b16e1fd6bc71611b3cc4b8bcb0cf327fda5ef7dabfe73839f66c7b2377472"}
Oct 30 13:39:57.416826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:57.416768    4544 scope.go:122] "RemoveContainer" containerID="a29d9f05976fecb3a4dd4c22e4962a0fae470b1850b962433bd4adcdbaf52428"
Oct 30 13:39:57.427427 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:57.427404    4544 scope.go:122] "RemoveContainer" containerID="a29d9f05976fecb3a4dd4c22e4962a0fae470b1850b962433bd4adcdbaf52428"
Oct 30 13:39:57.427729 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:39:57.427709    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a29d9f05976fecb3a4dd4c22e4962a0fae470b1850b962433bd4adcdbaf52428\": container with ID starting with a29d9f05976fecb3a4dd4c22e4962a0fae470b1850b962433bd4adcdbaf52428 not found: ID does not exist" containerID="a29d9f05976fecb3a4dd4c22e4962a0fae470b1850b962433bd4adcdbaf52428"
Oct 30 13:39:57.427776 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:57.427741    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a29d9f05976fecb3a4dd4c22e4962a0fae470b1850b962433bd4adcdbaf52428"} err="failed to get container status \"a29d9f05976fecb3a4dd4c22e4962a0fae470b1850b962433bd4adcdbaf52428\": rpc error: code = NotFound desc = could not find container \"a29d9f05976fecb3a4dd4c22e4962a0fae470b1850b962433bd4adcdbaf52428\": container with ID starting with a29d9f05976fecb3a4dd4c22e4962a0fae470b1850b962433bd4adcdbaf52428 not found: ID does not exist"
Oct 30 13:39:57.435856 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:57.435787    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:57.478174 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:57.478137    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb"]
Oct 30 13:39:57.482033 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:57.482003    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-bcd76c-6df4b6864f-mnhnb"]
Oct 30 13:39:57.583994 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:57.583957    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:57.733930 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:57.733847    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:57.853989 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:57.853952    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:57.882312 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:57.882273    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7" podUID="2beb5b6e-8222-48ce-b46b-b90e478277ed" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:39:57.971297 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:57.971253    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"]
Oct 30 13:39:57.971504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:57.971481    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg" podUID="65e7c2e0-231b-4455-b017-995e6691cf31" containerName="istio-proxy" containerID="cri-o://ceb3727fa65a96d13a78c790ea362893510032fc0a7dc9625cba29c0564297be" gracePeriod=30
Oct 30 13:39:57.981412 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:57.981383    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:58.000787 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:58.000717    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb" podUID="f673c81d-7be1-4c6c-bd5c-904136739702" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:39:58.039793 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:58.039752    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b30a2b9b-hhhkp"]
Oct 30 13:39:58.045865 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:58.045838    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b30a2b9b-hhhkp"]
Oct 30 13:39:58.309726 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:58.309631    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:58.436438 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:58.436405    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:58.814716 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:58.814675    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:58.937888 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:58.937842    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:58.944125 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:58.944091    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p" podUID="276d8bab-6921-4d6d-a482-258fbcbca9b7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:39:59.008114 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:39:59.008076    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:39:59.086859 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.086824    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0824f178-4f12-4b6c-8609-4c953e28d039" path="/var/lib/kubelet/pods/0824f178-4f12-4b6c-8609-4c953e28d039/volumes"
Oct 30 13:39:59.087329 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.087314    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="677b185b-bc7d-4d32-87e7-a5b47de80407" path="/var/lib/kubelet/pods/677b185b-bc7d-4d32-87e7-a5b47de80407/volumes"
Oct 30 13:39:59.149795 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.149743    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"]
Oct 30 13:39:59.150060 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.150028    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj" podUID="02d79e94-7094-44da-a04a-824f9328125e" containerName="istio-proxy" containerID="cri-o://f184a1e7b07b784c4ba5b07513f457f06e9cee0f7fe058a5cf7c524173db7cac" gracePeriod=30
Oct 30 13:39:59.179775 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.179735    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:59.398207 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.398175    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb"
Oct 30 13:39:59.435164 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.435133    4544 generic.go:356] "Generic (PLEG): container finished" podID="f673c81d-7be1-4c6c-bd5c-904136739702" containerID="c434a75ca1235f1fb4b4e289578aa4aeb4f96d3713cb50ed2a49ada207c025c6" exitCode=0
Oct 30 13:39:59.435407 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.435188    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb"
Oct 30 13:39:59.435407 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.435258    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb" event={"ID":"f673c81d-7be1-4c6c-bd5c-904136739702","Type":"ContainerDied","Data":"c434a75ca1235f1fb4b4e289578aa4aeb4f96d3713cb50ed2a49ada207c025c6"}
Oct 30 13:39:59.435407 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.435330    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb" event={"ID":"f673c81d-7be1-4c6c-bd5c-904136739702","Type":"ContainerDied","Data":"4922763cb77301c035fc53d4672378889758481c771b48d919fc632297c6da21"}
Oct 30 13:39:59.435407 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.435360    4544 scope.go:122] "RemoveContainer" containerID="c434a75ca1235f1fb4b4e289578aa4aeb4f96d3713cb50ed2a49ada207c025c6"
Oct 30 13:39:59.447607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.447588    4544 scope.go:122] "RemoveContainer" containerID="c434a75ca1235f1fb4b4e289578aa4aeb4f96d3713cb50ed2a49ada207c025c6"
Oct 30 13:39:59.447935 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:39:59.447916    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c434a75ca1235f1fb4b4e289578aa4aeb4f96d3713cb50ed2a49ada207c025c6\": container with ID starting with c434a75ca1235f1fb4b4e289578aa4aeb4f96d3713cb50ed2a49ada207c025c6 not found: ID does not exist" containerID="c434a75ca1235f1fb4b4e289578aa4aeb4f96d3713cb50ed2a49ada207c025c6"
Oct 30 13:39:59.447995 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.447946    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c434a75ca1235f1fb4b4e289578aa4aeb4f96d3713cb50ed2a49ada207c025c6"} err="failed to get container status \"c434a75ca1235f1fb4b4e289578aa4aeb4f96d3713cb50ed2a49ada207c025c6\": rpc error: code = NotFound desc = could not find container \"c434a75ca1235f1fb4b4e289578aa4aeb4f96d3713cb50ed2a49ada207c025c6\": container with ID starting with c434a75ca1235f1fb4b4e289578aa4aeb4f96d3713cb50ed2a49ada207c025c6 not found: ID does not exist"
Oct 30 13:39:59.559353 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.559309    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:59.565757 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.565710    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f673c81d-7be1-4c6c-bd5c-904136739702-kube-api-access-fb8dl\" (UniqueName: \"kubernetes.io/projected/f673c81d-7be1-4c6c-bd5c-904136739702-kube-api-access-fb8dl\") pod \"f673c81d-7be1-4c6c-bd5c-904136739702\" (UID: \"f673c81d-7be1-4c6c-bd5c-904136739702\") "
Oct 30 13:39:59.565944 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.565768    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f673c81d-7be1-4c6c-bd5c-904136739702-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f673c81d-7be1-4c6c-bd5c-904136739702-plugins-config-volume\") pod \"f673c81d-7be1-4c6c-bd5c-904136739702\" (UID: \"f673c81d-7be1-4c6c-bd5c-904136739702\") "
Oct 30 13:39:59.566270 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.566213    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f673c81d-7be1-4c6c-bd5c-904136739702-plugins-config-volume" pod "f673c81d-7be1-4c6c-bd5c-904136739702" (UID: "f673c81d-7be1-4c6c-bd5c-904136739702"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:59.568166 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.568125    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f673c81d-7be1-4c6c-bd5c-904136739702-kube-api-access-fb8dl" pod "f673c81d-7be1-4c6c-bd5c-904136739702" (UID: "f673c81d-7be1-4c6c-bd5c-904136739702"). InnerVolumeSpecName "kube-api-access-fb8dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:59.667725 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.667628    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb8dl\" (UniqueName: \"kubernetes.io/projected/f673c81d-7be1-4c6c-bd5c-904136739702-kube-api-access-fb8dl\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:59.667725 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.667664    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f673c81d-7be1-4c6c-bd5c-904136739702-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:39:59.732691 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.732653    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:39:59.784771 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.784736    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb"]
Oct 30 13:39:59.791905 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.791874    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb"]
Oct 30 13:39:59.835775 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.835746    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c2ce75-qwbbm"]
Oct 30 13:39:59.836434 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836408    4544 cpu_manager.go:414] "containerMap: removing container" podUID="0824f178-4f12-4b6c-8609-4c953e28d039" containerName="maas-api"
Oct 30 13:39:59.836434 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836429    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0824f178-4f12-4b6c-8609-4c953e28d039" containerName="maas-api"
Oct 30 13:39:59.836434 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836440    4544 cpu_manager.go:414] "containerMap: removing container" podUID="fb981ab3-c15c-48d3-868d-5d5d93616212" containerName="payload-processing"
Oct 30 13:39:59.836832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836447    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb981ab3-c15c-48d3-868d-5d5d93616212" containerName="payload-processing"
Oct 30 13:39:59.836832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836463    4544 cpu_manager.go:414] "containerMap: removing container" podUID="8dacf33b-a434-4657-bc05-71ebd72337dc" containerName="revoke-keys"
Oct 30 13:39:59.836832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836471    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dacf33b-a434-4657-bc05-71ebd72337dc" containerName="revoke-keys"
Oct 30 13:39:59.836832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836485    4544 cpu_manager.go:414] "containerMap: removing container" podUID="c713af1f-1695-4ad5-a564-25420ba533a6" containerName="istio-proxy"
Oct 30 13:39:59.836832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836492    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c713af1f-1695-4ad5-a564-25420ba533a6" containerName="istio-proxy"
Oct 30 13:39:59.836832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836503    4544 cpu_manager.go:414] "containerMap: removing container" podUID="6df66313-8135-4c91-a387-e7f654c305b3" containerName="payload-processing"
Oct 30 13:39:59.836832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836509    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df66313-8135-4c91-a387-e7f654c305b3" containerName="payload-processing"
Oct 30 13:39:59.836832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836529    4544 cpu_manager.go:414] "containerMap: removing container" podUID="f673c81d-7be1-4c6c-bd5c-904136739702" containerName="payload-processing"
Oct 30 13:39:59.836832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836536    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f673c81d-7be1-4c6c-bd5c-904136739702" containerName="payload-processing"
Oct 30 13:39:59.836832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836547    4544 cpu_manager.go:414] "containerMap: removing container" podUID="611e617c-64ab-4006-972d-fa789281cc38" containerName="payload-pre-processing"
Oct 30 13:39:59.836832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836554    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="611e617c-64ab-4006-972d-fa789281cc38" containerName="payload-pre-processing"
Oct 30 13:39:59.836832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836570    4544 cpu_manager.go:414] "containerMap: removing container" podUID="eb920234-9adb-4f7f-a5e4-27faa3603144" containerName="payload-processing"
Oct 30 13:39:59.836832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836577    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb920234-9adb-4f7f-a5e4-27faa3603144" containerName="payload-processing"
Oct 30 13:39:59.836832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836586    4544 cpu_manager.go:414] "containerMap: removing container" podUID="4e66eb26-5727-4ad8-881f-64b6711fe440" containerName="istio-proxy"
Oct 30 13:39:59.836832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836594    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e66eb26-5727-4ad8-881f-64b6711fe440" containerName="istio-proxy"
Oct 30 13:39:59.836832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836605    4544 cpu_manager.go:414] "containerMap: removing container" podUID="4e66eb26-5727-4ad8-881f-64b6711fe440" containerName="istio-proxy"
Oct 30 13:39:59.836832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836611    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e66eb26-5727-4ad8-881f-64b6711fe440" containerName="istio-proxy"
Oct 30 13:39:59.836832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836643    4544 cpu_manager.go:414] "containerMap: removing container" podUID="8013c75e-2665-43ee-879b-ec14f338cca5" containerName="payload-pre-processing"
Oct 30 13:39:59.836832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836649    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8013c75e-2665-43ee-879b-ec14f338cca5" containerName="payload-pre-processing"
Oct 30 13:39:59.836832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836667    4544 cpu_manager.go:414] "containerMap: removing container" podUID="f6d9c074-fc98-4507-822d-7ca79b3770c5" containerName="payload-pre-processing"
Oct 30 13:39:59.836832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836674    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6d9c074-fc98-4507-822d-7ca79b3770c5" containerName="payload-pre-processing"
Oct 30 13:39:59.836832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836685    4544 cpu_manager.go:414] "containerMap: removing container" podUID="75391895-2b05-426d-98ad-87ad85d2e9c3" containerName="istio-proxy"
Oct 30 13:39:59.836832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836691    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75391895-2b05-426d-98ad-87ad85d2e9c3" containerName="istio-proxy"
Oct 30 13:39:59.836832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836710    4544 cpu_manager.go:414] "containerMap: removing container" podUID="329fbf95-10bc-4b2f-afc2-14c679d9edd8" containerName="payload-processing"
Oct 30 13:39:59.836832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836715    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="329fbf95-10bc-4b2f-afc2-14c679d9edd8" containerName="payload-processing"
Oct 30 13:39:59.837952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836858    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="8013c75e-2665-43ee-879b-ec14f338cca5" containerName="payload-pre-processing"
Oct 30 13:39:59.837952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836874    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6d9c074-fc98-4507-822d-7ca79b3770c5" containerName="payload-pre-processing"
Oct 30 13:39:59.837952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836886    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="4400fc6b-5d1c-4f3f-93be-7ef8540e8d79" containerName="istio-proxy"
Oct 30 13:39:59.837952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836898    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="329fbf95-10bc-4b2f-afc2-14c679d9edd8" containerName="payload-processing"
Oct 30 13:39:59.837952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836907    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="0824f178-4f12-4b6c-8609-4c953e28d039" containerName="maas-api"
Oct 30 13:39:59.837952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836915    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb981ab3-c15c-48d3-868d-5d5d93616212" containerName="payload-processing"
Oct 30 13:39:59.837952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836925    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dacf33b-a434-4657-bc05-71ebd72337dc" containerName="revoke-keys"
Oct 30 13:39:59.837952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836934    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="c713af1f-1695-4ad5-a564-25420ba533a6" containerName="istio-proxy"
Oct 30 13:39:59.837952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836941    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df66313-8135-4c91-a387-e7f654c305b3" containerName="payload-processing"
Oct 30 13:39:59.837952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836952    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="f673c81d-7be1-4c6c-bd5c-904136739702" containerName="payload-processing"
Oct 30 13:39:59.837952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836960    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="611e617c-64ab-4006-972d-fa789281cc38" containerName="payload-pre-processing"
Oct 30 13:39:59.837952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836970    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb920234-9adb-4f7f-a5e4-27faa3603144" containerName="payload-processing"
Oct 30 13:39:59.837952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836977    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e66eb26-5727-4ad8-881f-64b6711fe440" containerName="istio-proxy"
Oct 30 13:39:59.837952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.836985    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e66eb26-5727-4ad8-881f-64b6711fe440" containerName="istio-proxy"
Oct 30 13:39:59.841677 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.841660    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c2ce75-qwbbm"
Oct 30 13:39:59.867654 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.867549    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c2ce75-qwbbm"]
Oct 30 13:39:59.913991 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.913938    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:39:59.914336 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.914016    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:39:59.971658 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.971561    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfbps\" (UniqueName: \"kubernetes.io/projected/68a65539-9527-425b-a39c-d1d536e41df1-kube-api-access-wfbps\") pod \"maas-api-revoke-keys-e2e-worker-w2-c2ce75-qwbbm\" (UID: \"68a65539-9527-425b-a39c-d1d536e41df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c2ce75-qwbbm"
Oct 30 13:39:59.971828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.971692    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68a65539-9527-425b-a39c-d1d536e41df1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c2ce75-qwbbm\" (UID: \"68a65539-9527-425b-a39c-d1d536e41df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c2ce75-qwbbm"
Oct 30 13:39:59.971877 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:39:59.971858    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68a65539-9527-425b-a39c-d1d536e41df1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c2ce75-qwbbm\" (UID: \"68a65539-9527-425b-a39c-d1d536e41df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c2ce75-qwbbm"
Oct 30 13:40:00.073124 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:00.073084    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68a65539-9527-425b-a39c-d1d536e41df1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c2ce75-qwbbm\" (UID: \"68a65539-9527-425b-a39c-d1d536e41df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c2ce75-qwbbm"
Oct 30 13:40:00.073367 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:00.073261    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68a65539-9527-425b-a39c-d1d536e41df1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c2ce75-qwbbm\" (UID: \"68a65539-9527-425b-a39c-d1d536e41df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c2ce75-qwbbm"
Oct 30 13:40:00.073367 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:00.073299    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfbps\" (UniqueName: \"kubernetes.io/projected/68a65539-9527-425b-a39c-d1d536e41df1-kube-api-access-wfbps\") pod \"maas-api-revoke-keys-e2e-worker-w2-c2ce75-qwbbm\" (UID: \"68a65539-9527-425b-a39c-d1d536e41df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c2ce75-qwbbm"
Oct 30 13:40:00.074092 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:00.074066    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68a65539-9527-425b-a39c-d1d536e41df1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c2ce75-qwbbm\" (UID: \"68a65539-9527-425b-a39c-d1d536e41df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c2ce75-qwbbm"
Oct 30 13:40:00.075581 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:00.075558    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68a65539-9527-425b-a39c-d1d536e41df1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c2ce75-qwbbm\" (UID: \"68a65539-9527-425b-a39c-d1d536e41df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c2ce75-qwbbm"
Oct 30 13:40:00.100301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:00.100267    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfbps\" (UniqueName: \"kubernetes.io/projected/68a65539-9527-425b-a39c-d1d536e41df1-kube-api-access-wfbps\") pod \"maas-api-revoke-keys-e2e-worker-w2-c2ce75-qwbbm\" (UID: \"68a65539-9527-425b-a39c-d1d536e41df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c2ce75-qwbbm"
Oct 30 13:40:00.155777 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:00.155738    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c2ce75-qwbbm"
Oct 30 13:40:00.469737 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:40:00.469701    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68a65539_9527_425b_a39c_d1d536e41df1.slice/crio-406e8978336a00caf9836d3f6a3700e40cc38000ef7fa6af4844fcceae120a80 WatchSource:0}: Error finding container 406e8978336a00caf9836d3f6a3700e40cc38000ef7fa6af4844fcceae120a80: Status 404 returned error can't find the container with id 406e8978336a00caf9836d3f6a3700e40cc38000ef7fa6af4844fcceae120a80
Oct 30 13:40:00.469737 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:00.469708    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c2ce75-qwbbm"]
Oct 30 13:40:00.481602 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:00.481566    4544 patch_prober.go:29] interesting pod/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:40:00.481788 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:00.481629    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg" podUID="65e7c2e0-231b-4455-b017-995e6691cf31" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:40:01.112255 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:01.106764    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f673c81d-7be1-4c6c-bd5c-904136739702" path="/var/lib/kubelet/pods/f673c81d-7be1-4c6c-bd5c-904136739702/volumes"
Oct 30 13:40:01.454023 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:01.453984    4544 generic.go:356] "Generic (PLEG): container finished" podID="68a65539-9527-425b-a39c-d1d536e41df1" containerID="3f2a690746eb91a6c4ead430def72b3300465fef87a56025ff2ee8c6a77f0af4" exitCode=0
Oct 30 13:40:01.454264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:01.454067    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c2ce75-qwbbm" event={"ID":"68a65539-9527-425b-a39c-d1d536e41df1","Type":"ContainerDied","Data":"3f2a690746eb91a6c4ead430def72b3300465fef87a56025ff2ee8c6a77f0af4"}
Oct 30 13:40:01.454264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:01.454137    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c2ce75-qwbbm" event={"ID":"68a65539-9527-425b-a39c-d1d536e41df1","Type":"ContainerStarted","Data":"406e8978336a00caf9836d3f6a3700e40cc38000ef7fa6af4844fcceae120a80"}
Oct 30 13:40:01.738867 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:01.738757    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:02.625620 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:02.625584    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c2ce75-qwbbm"
Oct 30 13:40:02.725458 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:02.725362    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68a65539-9527-425b-a39c-d1d536e41df1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68a65539-9527-425b-a39c-d1d536e41df1-maas-api-xks-ca\") pod \"68a65539-9527-425b-a39c-d1d536e41df1\" (UID: \"68a65539-9527-425b-a39c-d1d536e41df1\") "
Oct 30 13:40:02.725629 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:02.725561    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68a65539-9527-425b-a39c-d1d536e41df1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68a65539-9527-425b-a39c-d1d536e41df1-maas-api-openshift-ca\") pod \"68a65539-9527-425b-a39c-d1d536e41df1\" (UID: \"68a65539-9527-425b-a39c-d1d536e41df1\") "
Oct 30 13:40:02.725720 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:02.725702    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68a65539-9527-425b-a39c-d1d536e41df1-kube-api-access-wfbps\" (UniqueName: \"kubernetes.io/projected/68a65539-9527-425b-a39c-d1d536e41df1-kube-api-access-wfbps\") pod \"68a65539-9527-425b-a39c-d1d536e41df1\" (UID: \"68a65539-9527-425b-a39c-d1d536e41df1\") "
Oct 30 13:40:02.725967 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:02.725934    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68a65539-9527-425b-a39c-d1d536e41df1-maas-api-openshift-ca" pod "68a65539-9527-425b-a39c-d1d536e41df1" (UID: "68a65539-9527-425b-a39c-d1d536e41df1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:02.726403 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:02.726375    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68a65539-9527-425b-a39c-d1d536e41df1-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:02.727809 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:02.727782    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68a65539-9527-425b-a39c-d1d536e41df1-maas-api-xks-ca" pod "68a65539-9527-425b-a39c-d1d536e41df1" (UID: "68a65539-9527-425b-a39c-d1d536e41df1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:40:02.727961 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:02.727944    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68a65539-9527-425b-a39c-d1d536e41df1-kube-api-access-wfbps" pod "68a65539-9527-425b-a39c-d1d536e41df1" (UID: "68a65539-9527-425b-a39c-d1d536e41df1"). InnerVolumeSpecName "kube-api-access-wfbps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:02.827746 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:02.827701    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfbps\" (UniqueName: \"kubernetes.io/projected/68a65539-9527-425b-a39c-d1d536e41df1-kube-api-access-wfbps\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:02.827746 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:02.827738    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68a65539-9527-425b-a39c-d1d536e41df1-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:02.955008 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:02.954970    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:03.079730 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.079659    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:03.346682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.346653    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:40:03.351615 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.351586    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:03.491274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.490492    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65e7c2e0-231b-4455-b017-995e6691cf31-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65e7c2e0-231b-4455-b017-995e6691cf31-istiod-ca-cert\") pod \"65e7c2e0-231b-4455-b017-995e6691cf31\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") "
Oct 30 13:40:03.491274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.490627    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65e7c2e0-231b-4455-b017-995e6691cf31-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65e7c2e0-231b-4455-b017-995e6691cf31-istio-podinfo\") pod \"65e7c2e0-231b-4455-b017-995e6691cf31\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") "
Oct 30 13:40:03.491274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.490670    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65e7c2e0-231b-4455-b017-995e6691cf31-kube-api-access-g9fgs\" (UniqueName: \"kubernetes.io/projected/65e7c2e0-231b-4455-b017-995e6691cf31-kube-api-access-g9fgs\") pod \"65e7c2e0-231b-4455-b017-995e6691cf31\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") "
Oct 30 13:40:03.491274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.490714    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65e7c2e0-231b-4455-b017-995e6691cf31-istio-token\" (UniqueName: \"kubernetes.io/projected/65e7c2e0-231b-4455-b017-995e6691cf31-istio-token\") pod \"65e7c2e0-231b-4455-b017-995e6691cf31\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") "
Oct 30 13:40:03.491274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.490764    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-workload-socket\") pod \"65e7c2e0-231b-4455-b017-995e6691cf31\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") "
Oct 30 13:40:03.491274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.490853    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-istio-envoy\") pod \"65e7c2e0-231b-4455-b017-995e6691cf31\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") "
Oct 30 13:40:03.491274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.490898    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-workload-certs\") pod \"65e7c2e0-231b-4455-b017-995e6691cf31\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") "
Oct 30 13:40:03.491274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.491073    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-istio-data\") pod \"65e7c2e0-231b-4455-b017-995e6691cf31\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") "
Oct 30 13:40:03.491274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.491116    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-credential-socket\") pod \"65e7c2e0-231b-4455-b017-995e6691cf31\" (UID: \"65e7c2e0-231b-4455-b017-995e6691cf31\") "
Oct 30 13:40:03.496453 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.495155    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-credential-socket" pod "65e7c2e0-231b-4455-b017-995e6691cf31" (UID: "65e7c2e0-231b-4455-b017-995e6691cf31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:03.502831 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.502692    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65e7c2e0-231b-4455-b017-995e6691cf31-istiod-ca-cert" pod "65e7c2e0-231b-4455-b017-995e6691cf31" (UID: "65e7c2e0-231b-4455-b017-995e6691cf31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:03.503082 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.503012    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-workload-socket" pod "65e7c2e0-231b-4455-b017-995e6691cf31" (UID: "65e7c2e0-231b-4455-b017-995e6691cf31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:03.503613 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.503256    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-workload-certs" pod "65e7c2e0-231b-4455-b017-995e6691cf31" (UID: "65e7c2e0-231b-4455-b017-995e6691cf31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:03.503613 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.503444    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-istio-data" pod "65e7c2e0-231b-4455-b017-995e6691cf31" (UID: "65e7c2e0-231b-4455-b017-995e6691cf31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:03.507425 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.507379    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65e7c2e0-231b-4455-b017-995e6691cf31-istio-podinfo" pod "65e7c2e0-231b-4455-b017-995e6691cf31" (UID: "65e7c2e0-231b-4455-b017-995e6691cf31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:40:03.513897 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.513819    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e7c2e0-231b-4455-b017-995e6691cf31-kube-api-access-g9fgs" pod "65e7c2e0-231b-4455-b017-995e6691cf31" (UID: "65e7c2e0-231b-4455-b017-995e6691cf31"). InnerVolumeSpecName "kube-api-access-g9fgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:03.517543 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.516436    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-istio-envoy" pod "65e7c2e0-231b-4455-b017-995e6691cf31" (UID: "65e7c2e0-231b-4455-b017-995e6691cf31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:03.522790 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.519704    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e7c2e0-231b-4455-b017-995e6691cf31-istio-token" pod "65e7c2e0-231b-4455-b017-995e6691cf31" (UID: "65e7c2e0-231b-4455-b017-995e6691cf31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:03.522790 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.519783    4544 generic.go:356] "Generic (PLEG): container finished" podID="65e7c2e0-231b-4455-b017-995e6691cf31" containerID="ceb3727fa65a96d13a78c790ea362893510032fc0a7dc9625cba29c0564297be" exitCode=0
Oct 30 13:40:03.522790 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.519962    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"
Oct 30 13:40:03.522790 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.520489    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg" event={"ID":"65e7c2e0-231b-4455-b017-995e6691cf31","Type":"ContainerDied","Data":"ceb3727fa65a96d13a78c790ea362893510032fc0a7dc9625cba29c0564297be"}
Oct 30 13:40:03.522790 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.520560    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg" event={"ID":"65e7c2e0-231b-4455-b017-995e6691cf31","Type":"ContainerDied","Data":"4565b4874c12002cf54b4d2a97d33cf51e0fa6771ad5a10826526339e9b9698e"}
Oct 30 13:40:03.522790 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.520594    4544 scope.go:122] "RemoveContainer" containerID="ceb3727fa65a96d13a78c790ea362893510032fc0a7dc9625cba29c0564297be"
Oct 30 13:40:03.531121 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.531080    4544 generic.go:356] "Generic (PLEG): container finished" podID="276d8bab-6921-4d6d-a482-258fbcbca9b7" containerID="507152f2d87a669374635d0ea2c82180f38d4aff04ada125aff000ca05732f05" exitCode=0
Oct 30 13:40:03.531327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.531283    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p" event={"ID":"276d8bab-6921-4d6d-a482-258fbcbca9b7","Type":"ContainerDied","Data":"507152f2d87a669374635d0ea2c82180f38d4aff04ada125aff000ca05732f05"}
Oct 30 13:40:03.535441 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.533931    4544 generic.go:356] "Generic (PLEG): container finished" podID="2beb5b6e-8222-48ce-b46b-b90e478277ed" containerID="56b51e3a0198e871c6f7c6774b900ad7c42b56a5e29064554ce173f8dbecc2ca" exitCode=0
Oct 30 13:40:03.535441 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.534100    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7" event={"ID":"2beb5b6e-8222-48ce-b46b-b90e478277ed","Type":"ContainerDied","Data":"56b51e3a0198e871c6f7c6774b900ad7c42b56a5e29064554ce173f8dbecc2ca"}
Oct 30 13:40:03.537934 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.536940    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c2ce75-qwbbm"
Oct 30 13:40:03.537934 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.536967    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c2ce75-qwbbm" event={"ID":"68a65539-9527-425b-a39c-d1d536e41df1","Type":"ContainerDied","Data":"406e8978336a00caf9836d3f6a3700e40cc38000ef7fa6af4844fcceae120a80"}
Oct 30 13:40:03.537934 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.537024    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="406e8978336a00caf9836d3f6a3700e40cc38000ef7fa6af4844fcceae120a80"
Oct 30 13:40:03.549946 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.549923    4544 scope.go:122] "RemoveContainer" containerID="0ed787d6496b5ec272bc824ffd4aad4e76a5fe8227a8cf4b830090292cbdd1a2"
Oct 30 13:40:03.574028 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.570459    4544 scope.go:122] "RemoveContainer" containerID="ceb3727fa65a96d13a78c790ea362893510032fc0a7dc9625cba29c0564297be"
Oct 30 13:40:03.574028 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:40:03.570920    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceb3727fa65a96d13a78c790ea362893510032fc0a7dc9625cba29c0564297be\": container with ID starting with ceb3727fa65a96d13a78c790ea362893510032fc0a7dc9625cba29c0564297be not found: ID does not exist" containerID="ceb3727fa65a96d13a78c790ea362893510032fc0a7dc9625cba29c0564297be"
Oct 30 13:40:03.574028 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.570965    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceb3727fa65a96d13a78c790ea362893510032fc0a7dc9625cba29c0564297be"} err="failed to get container status \"ceb3727fa65a96d13a78c790ea362893510032fc0a7dc9625cba29c0564297be\": rpc error: code = NotFound desc = could not find container \"ceb3727fa65a96d13a78c790ea362893510032fc0a7dc9625cba29c0564297be\": container with ID starting with ceb3727fa65a96d13a78c790ea362893510032fc0a7dc9625cba29c0564297be not found: ID does not exist"
Oct 30 13:40:03.574028 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.570994    4544 scope.go:122] "RemoveContainer" containerID="0ed787d6496b5ec272bc824ffd4aad4e76a5fe8227a8cf4b830090292cbdd1a2"
Oct 30 13:40:03.574028 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:40:03.571280    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ed787d6496b5ec272bc824ffd4aad4e76a5fe8227a8cf4b830090292cbdd1a2\": container with ID starting with 0ed787d6496b5ec272bc824ffd4aad4e76a5fe8227a8cf4b830090292cbdd1a2 not found: ID does not exist" containerID="0ed787d6496b5ec272bc824ffd4aad4e76a5fe8227a8cf4b830090292cbdd1a2"
Oct 30 13:40:03.574028 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.571307    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ed787d6496b5ec272bc824ffd4aad4e76a5fe8227a8cf4b830090292cbdd1a2"} err="failed to get container status \"0ed787d6496b5ec272bc824ffd4aad4e76a5fe8227a8cf4b830090292cbdd1a2\": rpc error: code = NotFound desc = could not find container \"0ed787d6496b5ec272bc824ffd4aad4e76a5fe8227a8cf4b830090292cbdd1a2\": container with ID starting with 0ed787d6496b5ec272bc824ffd4aad4e76a5fe8227a8cf4b830090292cbdd1a2 not found: ID does not exist"
Oct 30 13:40:03.583500 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.582296    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"]
Oct 30 13:40:03.589975 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.589886    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg"]
Oct 30 13:40:03.593415 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.593379    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:03.593415 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.593414    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:03.593672 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.593423    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:03.593672 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.593434    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65e7c2e0-231b-4455-b017-995e6691cf31-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:03.593672 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.593444    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65e7c2e0-231b-4455-b017-995e6691cf31-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:03.593672 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.593452    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9fgs\" (UniqueName: \"kubernetes.io/projected/65e7c2e0-231b-4455-b017-995e6691cf31-kube-api-access-g9fgs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:03.593672 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.593463    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65e7c2e0-231b-4455-b017-995e6691cf31-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:03.593672 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.593472    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:03.593672 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.593482    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65e7c2e0-231b-4455-b017-995e6691cf31-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:03.644098 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.644064    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p"
Oct 30 13:40:03.673143 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.673117    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7"
Oct 30 13:40:03.795200 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.795161    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/276d8bab-6921-4d6d-a482-258fbcbca9b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/276d8bab-6921-4d6d-a482-258fbcbca9b7-plugins-config-volume\") pod \"276d8bab-6921-4d6d-a482-258fbcbca9b7\" (UID: \"276d8bab-6921-4d6d-a482-258fbcbca9b7\") "
Oct 30 13:40:03.795432 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.795319    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2beb5b6e-8222-48ce-b46b-b90e478277ed-kube-api-access-qttj7\" (UniqueName: \"kubernetes.io/projected/2beb5b6e-8222-48ce-b46b-b90e478277ed-kube-api-access-qttj7\") pod \"2beb5b6e-8222-48ce-b46b-b90e478277ed\" (UID: \"2beb5b6e-8222-48ce-b46b-b90e478277ed\") "
Oct 30 13:40:03.795432 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.795412    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2beb5b6e-8222-48ce-b46b-b90e478277ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2beb5b6e-8222-48ce-b46b-b90e478277ed-plugins-config-volume\") pod \"2beb5b6e-8222-48ce-b46b-b90e478277ed\" (UID: \"2beb5b6e-8222-48ce-b46b-b90e478277ed\") "
Oct 30 13:40:03.795534 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.795442    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/276d8bab-6921-4d6d-a482-258fbcbca9b7-kube-api-access-bn9hc\" (UniqueName: \"kubernetes.io/projected/276d8bab-6921-4d6d-a482-258fbcbca9b7-kube-api-access-bn9hc\") pod \"276d8bab-6921-4d6d-a482-258fbcbca9b7\" (UID: \"276d8bab-6921-4d6d-a482-258fbcbca9b7\") "
Oct 30 13:40:03.795604 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.795577    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/276d8bab-6921-4d6d-a482-258fbcbca9b7-plugins-config-volume" pod "276d8bab-6921-4d6d-a482-258fbcbca9b7" (UID: "276d8bab-6921-4d6d-a482-258fbcbca9b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:03.795846 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.795819    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2beb5b6e-8222-48ce-b46b-b90e478277ed-plugins-config-volume" pod "2beb5b6e-8222-48ce-b46b-b90e478277ed" (UID: "2beb5b6e-8222-48ce-b46b-b90e478277ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:03.795969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.795925    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/276d8bab-6921-4d6d-a482-258fbcbca9b7-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:03.795969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.795939    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2beb5b6e-8222-48ce-b46b-b90e478277ed-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:03.797613 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.797587    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/276d8bab-6921-4d6d-a482-258fbcbca9b7-kube-api-access-bn9hc" pod "276d8bab-6921-4d6d-a482-258fbcbca9b7" (UID: "276d8bab-6921-4d6d-a482-258fbcbca9b7"). InnerVolumeSpecName "kube-api-access-bn9hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:03.797794 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.797773    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2beb5b6e-8222-48ce-b46b-b90e478277ed-kube-api-access-qttj7" pod "2beb5b6e-8222-48ce-b46b-b90e478277ed" (UID: "2beb5b6e-8222-48ce-b46b-b90e478277ed"). InnerVolumeSpecName "kube-api-access-qttj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:03.798648 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.798619    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:03.897500 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.897415    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qttj7\" (UniqueName: \"kubernetes.io/projected/2beb5b6e-8222-48ce-b46b-b90e478277ed-kube-api-access-qttj7\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:03.897500 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.897451    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn9hc\" (UniqueName: \"kubernetes.io/projected/276d8bab-6921-4d6d-a482-258fbcbca9b7-kube-api-access-bn9hc\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:03.959074 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.959032    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b46c9-zkzr9"]
Oct 30 13:40:03.960007 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.959980    4544 cpu_manager.go:414] "containerMap: removing container" podUID="c713af1f-1695-4ad5-a564-25420ba533a6" containerName="istio-proxy"
Oct 30 13:40:03.960057 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.960013    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c713af1f-1695-4ad5-a564-25420ba533a6" containerName="istio-proxy"
Oct 30 13:40:03.960057 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.960052    4544 cpu_manager.go:414] "containerMap: removing container" podUID="276d8bab-6921-4d6d-a482-258fbcbca9b7" containerName="payload-processing"
Oct 30 13:40:03.960131 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.960060    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="276d8bab-6921-4d6d-a482-258fbcbca9b7" containerName="payload-processing"
Oct 30 13:40:03.960131 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.960093    4544 cpu_manager.go:414] "containerMap: removing container" podUID="65e7c2e0-231b-4455-b017-995e6691cf31" containerName="istio-proxy"
Oct 30 13:40:03.960131 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.960101    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65e7c2e0-231b-4455-b017-995e6691cf31" containerName="istio-proxy"
Oct 30 13:40:03.960131 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.960110    4544 cpu_manager.go:414] "containerMap: removing container" podUID="65e7c2e0-231b-4455-b017-995e6691cf31" containerName="istio-proxy"
Oct 30 13:40:03.960131 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.960117    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65e7c2e0-231b-4455-b017-995e6691cf31" containerName="istio-proxy"
Oct 30 13:40:03.960305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.960136    4544 cpu_manager.go:414] "containerMap: removing container" podUID="2beb5b6e-8222-48ce-b46b-b90e478277ed" containerName="payload-pre-processing"
Oct 30 13:40:03.960305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.960144    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2beb5b6e-8222-48ce-b46b-b90e478277ed" containerName="payload-pre-processing"
Oct 30 13:40:03.960305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.960172    4544 cpu_manager.go:414] "containerMap: removing container" podUID="68a65539-9527-425b-a39c-d1d536e41df1" containerName="revoke-keys"
Oct 30 13:40:03.960305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.960180    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68a65539-9527-425b-a39c-d1d536e41df1" containerName="revoke-keys"
Oct 30 13:40:03.960423 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.960342    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="68a65539-9527-425b-a39c-d1d536e41df1" containerName="revoke-keys"
Oct 30 13:40:03.960423 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.960360    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="c713af1f-1695-4ad5-a564-25420ba533a6" containerName="istio-proxy"
Oct 30 13:40:03.960423 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.960376    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="276d8bab-6921-4d6d-a482-258fbcbca9b7" containerName="payload-processing"
Oct 30 13:40:03.960423 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.960396    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="65e7c2e0-231b-4455-b017-995e6691cf31" containerName="istio-proxy"
Oct 30 13:40:03.960423 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.960404    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="65e7c2e0-231b-4455-b017-995e6691cf31" containerName="istio-proxy"
Oct 30 13:40:03.960423 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.960417    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="2beb5b6e-8222-48ce-b46b-b90e478277ed" containerName="payload-pre-processing"
Oct 30 13:40:03.964984 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.964954    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b46c9-zkzr9"
Oct 30 13:40:03.985880 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.985847    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b46c9-zkzr9"]
Oct 30 13:40:03.995105 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.992678    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:40:03.995105 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.992925    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:40:03.998812 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.998777    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tqqf\" (UniqueName: \"kubernetes.io/projected/c7368bce-df71-4576-90e7-923724db392c-kube-api-access-5tqqf\") pod \"maas-api-revoke-keys-e2e-worker-w1-8b46c9-zkzr9\" (UID: \"c7368bce-df71-4576-90e7-923724db392c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b46c9-zkzr9"
Oct 30 13:40:03.998989 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.998887    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7368bce-df71-4576-90e7-923724db392c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-8b46c9-zkzr9\" (UID: \"c7368bce-df71-4576-90e7-923724db392c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b46c9-zkzr9"
Oct 30 13:40:03.998989 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:03.998961    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7368bce-df71-4576-90e7-923724db392c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-8b46c9-zkzr9\" (UID: \"c7368bce-df71-4576-90e7-923724db392c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b46c9-zkzr9"
Oct 30 13:40:04.000626 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.000597    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:04.100446 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.100406    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tqqf\" (UniqueName: \"kubernetes.io/projected/c7368bce-df71-4576-90e7-923724db392c-kube-api-access-5tqqf\") pod \"maas-api-revoke-keys-e2e-worker-w1-8b46c9-zkzr9\" (UID: \"c7368bce-df71-4576-90e7-923724db392c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b46c9-zkzr9"
Oct 30 13:40:04.100638 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.100472    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7368bce-df71-4576-90e7-923724db392c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-8b46c9-zkzr9\" (UID: \"c7368bce-df71-4576-90e7-923724db392c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b46c9-zkzr9"
Oct 30 13:40:04.100638 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.100548    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7368bce-df71-4576-90e7-923724db392c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-8b46c9-zkzr9\" (UID: \"c7368bce-df71-4576-90e7-923724db392c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b46c9-zkzr9"
Oct 30 13:40:04.101312 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.101281    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7368bce-df71-4576-90e7-923724db392c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-8b46c9-zkzr9\" (UID: \"c7368bce-df71-4576-90e7-923724db392c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b46c9-zkzr9"
Oct 30 13:40:04.103162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.103138    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7368bce-df71-4576-90e7-923724db392c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-8b46c9-zkzr9\" (UID: \"c7368bce-df71-4576-90e7-923724db392c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b46c9-zkzr9"
Oct 30 13:40:04.126061 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.126028    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tqqf\" (UniqueName: \"kubernetes.io/projected/c7368bce-df71-4576-90e7-923724db392c-kube-api-access-5tqqf\") pod \"maas-api-revoke-keys-e2e-worker-w1-8b46c9-zkzr9\" (UID: \"c7368bce-df71-4576-90e7-923724db392c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b46c9-zkzr9"
Oct 30 13:40:04.280557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.280514    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b46c9-zkzr9"
Oct 30 13:40:04.432566 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.432532    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:40:04.503878 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.503844    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02d79e94-7094-44da-a04a-824f9328125e-kube-api-access-xdhgk\" (UniqueName: \"kubernetes.io/projected/02d79e94-7094-44da-a04a-824f9328125e-kube-api-access-xdhgk\") pod \"02d79e94-7094-44da-a04a-824f9328125e\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") "
Oct 30 13:40:04.504100 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.503916    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-workload-socket\") pod \"02d79e94-7094-44da-a04a-824f9328125e\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") "
Oct 30 13:40:04.504100 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.503960    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02d79e94-7094-44da-a04a-824f9328125e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d79e94-7094-44da-a04a-824f9328125e-istio-podinfo\") pod \"02d79e94-7094-44da-a04a-824f9328125e\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") "
Oct 30 13:40:04.504100 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.504045    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-workload-certs\") pod \"02d79e94-7094-44da-a04a-824f9328125e\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") "
Oct 30 13:40:04.504378 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.504124    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-credential-socket\") pod \"02d79e94-7094-44da-a04a-824f9328125e\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") "
Oct 30 13:40:04.504378 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.504190    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02d79e94-7094-44da-a04a-824f9328125e-istio-token\" (UniqueName: \"kubernetes.io/projected/02d79e94-7094-44da-a04a-824f9328125e-istio-token\") pod \"02d79e94-7094-44da-a04a-824f9328125e\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") "
Oct 30 13:40:04.504378 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.504241    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-istio-data\") pod \"02d79e94-7094-44da-a04a-824f9328125e\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") "
Oct 30 13:40:04.504378 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.504285    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02d79e94-7094-44da-a04a-824f9328125e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d79e94-7094-44da-a04a-824f9328125e-istiod-ca-cert\") pod \"02d79e94-7094-44da-a04a-824f9328125e\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") "
Oct 30 13:40:04.504378 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.504333    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-istio-envoy\") pod \"02d79e94-7094-44da-a04a-824f9328125e\" (UID: \"02d79e94-7094-44da-a04a-824f9328125e\") "
Oct 30 13:40:04.504643 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.504535    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-workload-certs" pod "02d79e94-7094-44da-a04a-824f9328125e" (UID: "02d79e94-7094-44da-a04a-824f9328125e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:04.504643 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.504548    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-workload-socket" pod "02d79e94-7094-44da-a04a-824f9328125e" (UID: "02d79e94-7094-44da-a04a-824f9328125e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:04.505240 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.504753    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-istio-data" pod "02d79e94-7094-44da-a04a-824f9328125e" (UID: "02d79e94-7094-44da-a04a-824f9328125e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:04.505240 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.504850    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02d79e94-7094-44da-a04a-824f9328125e-istiod-ca-cert" pod "02d79e94-7094-44da-a04a-824f9328125e" (UID: "02d79e94-7094-44da-a04a-824f9328125e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:04.505240 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.505067    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-credential-socket" pod "02d79e94-7094-44da-a04a-824f9328125e" (UID: "02d79e94-7094-44da-a04a-824f9328125e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:04.505240 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.505204    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:04.505498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.505261    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:04.505498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.505278    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:04.505498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.505292    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:04.505498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.505307    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d79e94-7094-44da-a04a-824f9328125e-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:04.507133 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.507102    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02d79e94-7094-44da-a04a-824f9328125e-kube-api-access-xdhgk" pod "02d79e94-7094-44da-a04a-824f9328125e" (UID: "02d79e94-7094-44da-a04a-824f9328125e"). InnerVolumeSpecName "kube-api-access-xdhgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:04.507273 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.507150    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02d79e94-7094-44da-a04a-824f9328125e-istio-podinfo" pod "02d79e94-7094-44da-a04a-824f9328125e" (UID: "02d79e94-7094-44da-a04a-824f9328125e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:40:04.507273 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.507199    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-istio-envoy" pod "02d79e94-7094-44da-a04a-824f9328125e" (UID: "02d79e94-7094-44da-a04a-824f9328125e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:04.507884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.507861    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02d79e94-7094-44da-a04a-824f9328125e-istio-token" pod "02d79e94-7094-44da-a04a-824f9328125e" (UID: "02d79e94-7094-44da-a04a-824f9328125e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:04.550119 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.550086    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p"
Oct 30 13:40:04.550119 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.550108    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p" event={"ID":"276d8bab-6921-4d6d-a482-258fbcbca9b7","Type":"ContainerDied","Data":"254fdc065f12370202da7902544c5c8fbb22d08a504bec4190ed929d98d4c591"}
Oct 30 13:40:04.550406 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.550169    4544 scope.go:122] "RemoveContainer" containerID="507152f2d87a669374635d0ea2c82180f38d4aff04ada125aff000ca05732f05"
Oct 30 13:40:04.551693 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.551673    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7"
Oct 30 13:40:04.551815 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.551672    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7" event={"ID":"2beb5b6e-8222-48ce-b46b-b90e478277ed","Type":"ContainerDied","Data":"4ffb28eccab1288450a99416b7b68710b35f33b0eacaa377be9b9e8e1c2d0f7b"}
Oct 30 13:40:04.553872 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.553846    4544 generic.go:356] "Generic (PLEG): container finished" podID="02d79e94-7094-44da-a04a-824f9328125e" containerID="f184a1e7b07b784c4ba5b07513f457f06e9cee0f7fe058a5cf7c524173db7cac" exitCode=0
Oct 30 13:40:04.554009 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.553895    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"
Oct 30 13:40:04.554009 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.553932    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj" event={"ID":"02d79e94-7094-44da-a04a-824f9328125e","Type":"ContainerDied","Data":"f184a1e7b07b784c4ba5b07513f457f06e9cee0f7fe058a5cf7c524173db7cac"}
Oct 30 13:40:04.554009 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.553995    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj" event={"ID":"02d79e94-7094-44da-a04a-824f9328125e","Type":"ContainerDied","Data":"12beb64402e22d718bd6336218c16253c884fb4b0d7d0645cea8925637a6677b"}
Oct 30 13:40:04.561447 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.561422    4544 scope.go:122] "RemoveContainer" containerID="56b51e3a0198e871c6f7c6774b900ad7c42b56a5e29064554ce173f8dbecc2ca"
Oct 30 13:40:04.573835 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.573811    4544 scope.go:122] "RemoveContainer" containerID="f184a1e7b07b784c4ba5b07513f457f06e9cee0f7fe058a5cf7c524173db7cac"
Oct 30 13:40:04.584548 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.584520    4544 scope.go:122] "RemoveContainer" containerID="f8dcec6e0c46f9f66b78c1fd2a67d5a55c7b87e3a7423e251b6da5ef84d7fdec"
Oct 30 13:40:04.601458 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.601000    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b46c9-zkzr9"]
Oct 30 13:40:04.604318 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:40:04.604290    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7368bce_df71_4576_90e7_923724db392c.slice/crio-ce09e0c664ea7adf7426d2cf71555f7c59b428135b61f084733c891d77163704 WatchSource:0}: Error finding container ce09e0c664ea7adf7426d2cf71555f7c59b428135b61f084733c891d77163704: Status 404 returned error can't find the container with id ce09e0c664ea7adf7426d2cf71555f7c59b428135b61f084733c891d77163704
Oct 30 13:40:04.606055 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.606035    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02d79e94-7094-44da-a04a-824f9328125e-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:04.606140 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.606058    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d79e94-7094-44da-a04a-824f9328125e-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:04.606140 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.606069    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdhgk\" (UniqueName: \"kubernetes.io/projected/02d79e94-7094-44da-a04a-824f9328125e-kube-api-access-xdhgk\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:04.606140 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.606081    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d79e94-7094-44da-a04a-824f9328125e-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:04.611187 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.611160    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p"]
Oct 30 13:40:04.617926 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.617893    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p"]
Oct 30 13:40:04.626708 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.626675    4544 scope.go:122] "RemoveContainer" containerID="f184a1e7b07b784c4ba5b07513f457f06e9cee0f7fe058a5cf7c524173db7cac"
Oct 30 13:40:04.627046 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:40:04.627015    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f184a1e7b07b784c4ba5b07513f457f06e9cee0f7fe058a5cf7c524173db7cac\": container with ID starting with f184a1e7b07b784c4ba5b07513f457f06e9cee0f7fe058a5cf7c524173db7cac not found: ID does not exist" containerID="f184a1e7b07b784c4ba5b07513f457f06e9cee0f7fe058a5cf7c524173db7cac"
Oct 30 13:40:04.627123 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.627058    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f184a1e7b07b784c4ba5b07513f457f06e9cee0f7fe058a5cf7c524173db7cac"} err="failed to get container status \"f184a1e7b07b784c4ba5b07513f457f06e9cee0f7fe058a5cf7c524173db7cac\": rpc error: code = NotFound desc = could not find container \"f184a1e7b07b784c4ba5b07513f457f06e9cee0f7fe058a5cf7c524173db7cac\": container with ID starting with f184a1e7b07b784c4ba5b07513f457f06e9cee0f7fe058a5cf7c524173db7cac not found: ID does not exist"
Oct 30 13:40:04.627123 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.627086    4544 scope.go:122] "RemoveContainer" containerID="f8dcec6e0c46f9f66b78c1fd2a67d5a55c7b87e3a7423e251b6da5ef84d7fdec"
Oct 30 13:40:04.627469 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:40:04.627444    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8dcec6e0c46f9f66b78c1fd2a67d5a55c7b87e3a7423e251b6da5ef84d7fdec\": container with ID starting with f8dcec6e0c46f9f66b78c1fd2a67d5a55c7b87e3a7423e251b6da5ef84d7fdec not found: ID does not exist" containerID="f8dcec6e0c46f9f66b78c1fd2a67d5a55c7b87e3a7423e251b6da5ef84d7fdec"
Oct 30 13:40:04.627540 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.627476    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8dcec6e0c46f9f66b78c1fd2a67d5a55c7b87e3a7423e251b6da5ef84d7fdec"} err="failed to get container status \"f8dcec6e0c46f9f66b78c1fd2a67d5a55c7b87e3a7423e251b6da5ef84d7fdec\": rpc error: code = NotFound desc = could not find container \"f8dcec6e0c46f9f66b78c1fd2a67d5a55c7b87e3a7423e251b6da5ef84d7fdec\": container with ID starting with f8dcec6e0c46f9f66b78c1fd2a67d5a55c7b87e3a7423e251b6da5ef84d7fdec not found: ID does not exist"
Oct 30 13:40:04.632541 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.632514    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"]
Oct 30 13:40:04.666802 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.666770    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7"]
Oct 30 13:40:04.671063 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.670955    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7"]
Oct 30 13:40:04.707115 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.707081    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"]
Oct 30 13:40:04.711017 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.710979    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj"]
Oct 30 13:40:04.804616 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:04.804580    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:05.104027 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.103984    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02d79e94-7094-44da-a04a-824f9328125e" path="/var/lib/kubelet/pods/02d79e94-7094-44da-a04a-824f9328125e/volumes"
Oct 30 13:40:05.104601 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.104584    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="276d8bab-6921-4d6d-a482-258fbcbca9b7" path="/var/lib/kubelet/pods/276d8bab-6921-4d6d-a482-258fbcbca9b7/volumes"
Oct 30 13:40:05.105069 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.105054    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2beb5b6e-8222-48ce-b46b-b90e478277ed" path="/var/lib/kubelet/pods/2beb5b6e-8222-48ce-b46b-b90e478277ed/volumes"
Oct 30 13:40:05.106088 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.106069    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65e7c2e0-231b-4455-b017-995e6691cf31" path="/var/lib/kubelet/pods/65e7c2e0-231b-4455-b017-995e6691cf31/volumes"
Oct 30 13:40:05.219750 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.219721    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:05.453178 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.453115    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565"
Oct 30 13:40:05.517427 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.517392    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6-plugins-config-volume\") pod \"e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6\" (UID: \"e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6\") "
Oct 30 13:40:05.517676 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.517501    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6-kube-api-access-g8c2v\" (UniqueName: \"kubernetes.io/projected/e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6-kube-api-access-g8c2v\") pod \"e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6\" (UID: \"e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6\") "
Oct 30 13:40:05.517938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.517903    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6-plugins-config-volume" pod "e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6" (UID: "e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:05.520927 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.520884    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6-kube-api-access-g8c2v" pod "e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6" (UID: "e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6"). InnerVolumeSpecName "kube-api-access-g8c2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:05.521124 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.521060    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8c2v\" (UniqueName: \"kubernetes.io/projected/e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6-kube-api-access-g8c2v\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:05.521124 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.521086    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:05.567481 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.567448    4544 generic.go:356] "Generic (PLEG): container finished" podID="c7368bce-df71-4576-90e7-923724db392c" containerID="7eadaf6403d31da67a65ca693ff1942e6fccea406a06711467f2f9131088e3d3" exitCode=0
Oct 30 13:40:05.567655 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.567536    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b46c9-zkzr9" event={"ID":"c7368bce-df71-4576-90e7-923724db392c","Type":"ContainerDied","Data":"7eadaf6403d31da67a65ca693ff1942e6fccea406a06711467f2f9131088e3d3"}
Oct 30 13:40:05.567655 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.567603    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b46c9-zkzr9" event={"ID":"c7368bce-df71-4576-90e7-923724db392c","Type":"ContainerStarted","Data":"ce09e0c664ea7adf7426d2cf71555f7c59b428135b61f084733c891d77163704"}
Oct 30 13:40:05.568965 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.568917    4544 generic.go:356] "Generic (PLEG): container finished" podID="e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6" containerID="cda9cfa5cedb6520e5c106b6c5dc7b1c48f0a02707aa8af26d1fdf7c2394d80e" exitCode=0
Oct 30 13:40:05.568965 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.568956    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565"
Oct 30 13:40:05.569175 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.569003    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565" event={"ID":"e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6","Type":"ContainerDied","Data":"cda9cfa5cedb6520e5c106b6c5dc7b1c48f0a02707aa8af26d1fdf7c2394d80e"}
Oct 30 13:40:05.569175 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.569050    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565" event={"ID":"e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6","Type":"ContainerDied","Data":"1a724a64be5632fbab3512548e3435303327a1ebc6b123a21dfeb6cb707ffec6"}
Oct 30 13:40:05.569175 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.569080    4544 scope.go:122] "RemoveContainer" containerID="cda9cfa5cedb6520e5c106b6c5dc7b1c48f0a02707aa8af26d1fdf7c2394d80e"
Oct 30 13:40:05.581234 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.581197    4544 scope.go:122] "RemoveContainer" containerID="cda9cfa5cedb6520e5c106b6c5dc7b1c48f0a02707aa8af26d1fdf7c2394d80e"
Oct 30 13:40:05.581667 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:40:05.581641    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cda9cfa5cedb6520e5c106b6c5dc7b1c48f0a02707aa8af26d1fdf7c2394d80e\": container with ID starting with cda9cfa5cedb6520e5c106b6c5dc7b1c48f0a02707aa8af26d1fdf7c2394d80e not found: ID does not exist" containerID="cda9cfa5cedb6520e5c106b6c5dc7b1c48f0a02707aa8af26d1fdf7c2394d80e"
Oct 30 13:40:05.581764 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.581681    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cda9cfa5cedb6520e5c106b6c5dc7b1c48f0a02707aa8af26d1fdf7c2394d80e"} err="failed to get container status \"cda9cfa5cedb6520e5c106b6c5dc7b1c48f0a02707aa8af26d1fdf7c2394d80e\": rpc error: code = NotFound desc = could not find container \"cda9cfa5cedb6520e5c106b6c5dc7b1c48f0a02707aa8af26d1fdf7c2394d80e\": container with ID starting with cda9cfa5cedb6520e5c106b6c5dc7b1c48f0a02707aa8af26d1fdf7c2394d80e not found: ID does not exist"
Oct 30 13:40:05.633668 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.633637    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"]
Oct 30 13:40:05.634385 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.634362    4544 cpu_manager.go:414] "containerMap: removing container" podUID="02d79e94-7094-44da-a04a-824f9328125e" containerName="istio-proxy"
Oct 30 13:40:05.634385 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.634381    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02d79e94-7094-44da-a04a-824f9328125e" containerName="istio-proxy"
Oct 30 13:40:05.634580 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.634411    4544 cpu_manager.go:414] "containerMap: removing container" podUID="e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6" containerName="payload-pre-processing"
Oct 30 13:40:05.634580 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.634421    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6" containerName="payload-pre-processing"
Oct 30 13:40:05.634580 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.634521    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="02d79e94-7094-44da-a04a-824f9328125e" containerName="istio-proxy"
Oct 30 13:40:05.634580 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.634533    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6" containerName="payload-pre-processing"
Oct 30 13:40:05.634580 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.634543    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="02d79e94-7094-44da-a04a-824f9328125e" containerName="istio-proxy"
Oct 30 13:40:05.639696 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.639671    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:05.648948 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.648914    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565"]
Oct 30 13:40:05.654260 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.653412    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565"]
Oct 30 13:40:05.655127 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.655098    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-87e742-openshift-default-dockercfg-g2smd\""
Oct 30 13:40:05.672105 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.672067    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"]
Oct 30 13:40:05.676650 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.676613    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:05.723941 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.723851    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-workload-socket\") pod \"e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") " pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:05.723941 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.723907    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istio-token\") pod \"e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") " pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:05.724141 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.724001    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istiod-ca-cert\") pod \"e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") " pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:05.724141 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.724093    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istio-data\") pod \"e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") " pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:05.724141 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.724122    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-workload-certs\") pod \"e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") " pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:05.724260 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.724235    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istio-envoy\") pod \"e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") " pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:05.724298 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.724284    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-credential-socket\") pod \"e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") " pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:05.724415 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.724394    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istio-podinfo\") pod \"e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") " pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:05.724503 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.724486    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlb75\" (UniqueName: \"kubernetes.io/projected/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-kube-api-access-wlb75\") pod \"e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") " pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:05.838272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.822703    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:05.838272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.825396    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-workload-socket\") pod \"e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") " pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:05.838272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.825436    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istio-token\") pod \"e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") " pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:05.838272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.825466    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istiod-ca-cert\") pod \"e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") " pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:05.838272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.825542    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istio-data\") pod \"e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") " pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:05.838272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.825577    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-workload-certs\") pod \"e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") " pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:05.838272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.825663    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istio-envoy\") pod \"e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") " pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:05.838272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.825727    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-credential-socket\") pod \"e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") " pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:05.838272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.825788    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istio-podinfo\") pod \"e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") " pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:05.838272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.825849    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlb75\" (UniqueName: \"kubernetes.io/projected/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-kube-api-access-wlb75\") pod \"e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") " pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:05.838272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.826616    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-workload-socket\") pod \"e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") " pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:05.838272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.827531    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istiod-ca-cert\") pod \"e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") " pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:05.838272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.827785    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istio-data\") pod \"e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") " pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:05.838272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.828013    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-workload-certs\") pod \"e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") " pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:05.838272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.832676    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-credential-socket\") pod \"e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") " pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:05.838272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.836338    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istio-envoy\") pod \"e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") " pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:05.839553 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.839523    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istio-podinfo\") pod \"e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") " pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:05.856000 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.855965    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istio-token\") pod \"e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") " pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:05.856183 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.856111    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlb75\" (UniqueName: \"kubernetes.io/projected/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-kube-api-access-wlb75\") pod \"e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") " pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:05.949957 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.949920    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:05.979891 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.979798    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:05.998274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.997072    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"]
Oct 30 13:40:05.998274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:05.997305    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz" podUID="09f03ae5-629b-4a35-8698-0043f221428d" containerName="istio-proxy" containerID="cri-o://731b2617b6db797a5b6f2dd2610ba69d6340c307926fee8f2c2c00599bf2cecc" gracePeriod=30
Oct 30 13:40:06.069336 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:06.069295    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f48014e-ffspr"]
Oct 30 13:40:06.075067 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:06.075032    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f48014e-ffspr"]
Oct 30 13:40:06.118281 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:06.107031    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:06.243928 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:06.243846    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:06.322854 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:06.322824    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"]
Oct 30 13:40:06.324639 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:40:06.324610    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode44b0460_7cc5_4fd3_9771_e5a6d53e0cf4.slice/crio-a82ced9aae19235bcbd98888999e8875cea756e621dc3cec81c1886d664515a1 WatchSource:0}: Error finding container a82ced9aae19235bcbd98888999e8875cea756e621dc3cec81c1886d664515a1: Status 404 returned error can't find the container with id a82ced9aae19235bcbd98888999e8875cea756e621dc3cec81c1886d664515a1
Oct 30 13:40:06.327584 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:06.327546    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:40:06.327685 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:06.327620    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:40:06.327685 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:06.327653    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:40:06.377551 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:06.377512    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:06.526037 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:06.525950    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:06.588913 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:06.588878    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"]
Oct 30 13:40:06.589148 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:06.589121    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs" podUID="e308ff03-37ae-4b7f-b27e-05b2f46ce66c" containerName="istio-proxy" containerID="cri-o://6aa0b9aac2f00d985e160d270a25eedf47c770451d610e36f60c1829e38b0023" gracePeriod=30
Oct 30 13:40:06.592380 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:06.592345    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r" event={"ID":"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4","Type":"ContainerStarted","Data":"0257c01bd914e8974dfe7a6529ab9f0f275cf73f72c147e567b92ed8d92683d9"}
Oct 30 13:40:06.592526 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:06.592409    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r" event={"ID":"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4","Type":"ContainerStarted","Data":"a82ced9aae19235bcbd98888999e8875cea756e621dc3cec81c1886d664515a1"}
Oct 30 13:40:06.753258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:06.751884    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:06.755792 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:06.754347    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r" podStartSLOduration=1.754325285 podStartE2EDuration="1.754325285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:06.718319602 +0000 UTC m=+3301.765453555" watchObservedRunningTime="2026-10-30 13:40:06.754325285 +0000 UTC m=+3301.801459238"
Oct 30 13:40:06.881880 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:06.881844    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b46c9-zkzr9"
Oct 30 13:40:06.980230 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:06.980168    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:07.040189 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.040143    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4334d165-nlzsw"]
Oct 30 13:40:07.043639 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.043610    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4334d165-nlzsw"]
Oct 30 13:40:07.047813 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.047787    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7368bce-df71-4576-90e7-923724db392c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7368bce-df71-4576-90e7-923724db392c-maas-api-xks-ca\") pod \"c7368bce-df71-4576-90e7-923724db392c\" (UID: \"c7368bce-df71-4576-90e7-923724db392c\") "
Oct 30 13:40:07.047995 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.047880    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7368bce-df71-4576-90e7-923724db392c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7368bce-df71-4576-90e7-923724db392c-maas-api-openshift-ca\") pod \"c7368bce-df71-4576-90e7-923724db392c\" (UID: \"c7368bce-df71-4576-90e7-923724db392c\") "
Oct 30 13:40:07.047995 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.047914    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7368bce-df71-4576-90e7-923724db392c-kube-api-access-5tqqf\" (UniqueName: \"kubernetes.io/projected/c7368bce-df71-4576-90e7-923724db392c-kube-api-access-5tqqf\") pod \"c7368bce-df71-4576-90e7-923724db392c\" (UID: \"c7368bce-df71-4576-90e7-923724db392c\") "
Oct 30 13:40:07.048325 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.048299    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7368bce-df71-4576-90e7-923724db392c-maas-api-openshift-ca" pod "c7368bce-df71-4576-90e7-923724db392c" (UID: "c7368bce-df71-4576-90e7-923724db392c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:07.048479 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.048459    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7368bce-df71-4576-90e7-923724db392c-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:07.050092 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.050066    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7368bce-df71-4576-90e7-923724db392c-kube-api-access-5tqqf" pod "c7368bce-df71-4576-90e7-923724db392c" (UID: "c7368bce-df71-4576-90e7-923724db392c"). InnerVolumeSpecName "kube-api-access-5tqqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:07.050487 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.050464    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7368bce-df71-4576-90e7-923724db392c-maas-api-xks-ca" pod "c7368bce-df71-4576-90e7-923724db392c" (UID: "c7368bce-df71-4576-90e7-923724db392c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:40:07.088885 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.088838    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40943ed5-3bb8-4e6d-830a-b15671e1c0aa" path="/var/lib/kubelet/pods/40943ed5-3bb8-4e6d-830a-b15671e1c0aa/volumes"
Oct 30 13:40:07.089442 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.089424    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="606e1cf8-1e8b-49d9-bc90-6e2403bc59a7" path="/var/lib/kubelet/pods/606e1cf8-1e8b-49d9-bc90-6e2403bc59a7/volumes"
Oct 30 13:40:07.089816 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.089802    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6" path="/var/lib/kubelet/pods/e2c0f591-0ec0-47cd-b6cf-dc7cc295c9f6/volumes"
Oct 30 13:40:07.149593 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.149555    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7368bce-df71-4576-90e7-923724db392c-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:07.149593 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.149589    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tqqf\" (UniqueName: \"kubernetes.io/projected/c7368bce-df71-4576-90e7-923724db392c-kube-api-access-5tqqf\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:07.205620 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.205585    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0e874e-xwghg"]
Oct 30 13:40:07.206272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.206252    4544 cpu_manager.go:414] "containerMap: removing container" podUID="c7368bce-df71-4576-90e7-923724db392c" containerName="revoke-keys"
Oct 30 13:40:07.206272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.206270    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7368bce-df71-4576-90e7-923724db392c" containerName="revoke-keys"
Oct 30 13:40:07.206423 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.206289    4544 cpu_manager.go:414] "containerMap: removing container" podUID="02d79e94-7094-44da-a04a-824f9328125e" containerName="istio-proxy"
Oct 30 13:40:07.206423 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.206294    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02d79e94-7094-44da-a04a-824f9328125e" containerName="istio-proxy"
Oct 30 13:40:07.206423 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.206394    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7368bce-df71-4576-90e7-923724db392c" containerName="revoke-keys"
Oct 30 13:40:07.218983 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.218955    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0e874e-xwghg"
Oct 30 13:40:07.237162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.237120    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0e874e-xwghg"]
Oct 30 13:40:07.352360 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.352264    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9dkh\" (UniqueName: \"kubernetes.io/projected/a18b9841-be8b-49c2-9676-87f53f621855-kube-api-access-l9dkh\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0e874e-xwghg\" (UID: \"a18b9841-be8b-49c2-9676-87f53f621855\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0e874e-xwghg"
Oct 30 13:40:07.352520 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.352368    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a18b9841-be8b-49c2-9676-87f53f621855-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0e874e-xwghg\" (UID: \"a18b9841-be8b-49c2-9676-87f53f621855\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0e874e-xwghg"
Oct 30 13:40:07.352583 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.352565    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a18b9841-be8b-49c2-9676-87f53f621855-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0e874e-xwghg\" (UID: \"a18b9841-be8b-49c2-9676-87f53f621855\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0e874e-xwghg"
Oct 30 13:40:07.454306 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.454268    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9dkh\" (UniqueName: \"kubernetes.io/projected/a18b9841-be8b-49c2-9676-87f53f621855-kube-api-access-l9dkh\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0e874e-xwghg\" (UID: \"a18b9841-be8b-49c2-9676-87f53f621855\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0e874e-xwghg"
Oct 30 13:40:07.454505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.454329    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a18b9841-be8b-49c2-9676-87f53f621855-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0e874e-xwghg\" (UID: \"a18b9841-be8b-49c2-9676-87f53f621855\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0e874e-xwghg"
Oct 30 13:40:07.454558 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.454522    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a18b9841-be8b-49c2-9676-87f53f621855-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0e874e-xwghg\" (UID: \"a18b9841-be8b-49c2-9676-87f53f621855\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0e874e-xwghg"
Oct 30 13:40:07.455130 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.455100    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a18b9841-be8b-49c2-9676-87f53f621855-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0e874e-xwghg\" (UID: \"a18b9841-be8b-49c2-9676-87f53f621855\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0e874e-xwghg"
Oct 30 13:40:07.456941 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.456904    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a18b9841-be8b-49c2-9676-87f53f621855-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0e874e-xwghg\" (UID: \"a18b9841-be8b-49c2-9676-87f53f621855\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0e874e-xwghg"
Oct 30 13:40:07.478296 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.478268    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9dkh\" (UniqueName: \"kubernetes.io/projected/a18b9841-be8b-49c2-9676-87f53f621855-kube-api-access-l9dkh\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0e874e-xwghg\" (UID: \"a18b9841-be8b-49c2-9676-87f53f621855\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0e874e-xwghg"
Oct 30 13:40:07.536549 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.536511    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0e874e-xwghg"
Oct 30 13:40:07.612697 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.612065    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b46c9-zkzr9"
Oct 30 13:40:07.612697 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.612085    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b46c9-zkzr9" event={"ID":"c7368bce-df71-4576-90e7-923724db392c","Type":"ContainerDied","Data":"ce09e0c664ea7adf7426d2cf71555f7c59b428135b61f084733c891d77163704"}
Oct 30 13:40:07.612697 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.612145    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce09e0c664ea7adf7426d2cf71555f7c59b428135b61f084733c891d77163704"
Oct 30 13:40:07.853597 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.852943    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0e874e-xwghg"]
Oct 30 13:40:07.856398 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:40:07.856366    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18b9841_be8b_49c2_9676_87f53f621855.slice/crio-52ff4716645cff7e02a96dc5b6fcd7743cb774af95f93f94d9e4d4e3eb9f7a3b WatchSource:0}: Error finding container 52ff4716645cff7e02a96dc5b6fcd7743cb774af95f93f94d9e4d4e3eb9f7a3b: Status 404 returned error can't find the container with id 52ff4716645cff7e02a96dc5b6fcd7743cb774af95f93f94d9e4d4e3eb9f7a3b
Oct 30 13:40:07.986613 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:07.986581    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:08.367638 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:08.367603    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:08.622495 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:08.622452    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0e874e-xwghg" event={"ID":"a18b9841-be8b-49c2-9676-87f53f621855","Type":"ContainerStarted","Data":"4fdbb923ca53a5dae50a6c46d1958622057861b7907a7ec0305f0f0a93d12b7f"}
Oct 30 13:40:08.622664 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:08.622519    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0e874e-xwghg" event={"ID":"a18b9841-be8b-49c2-9676-87f53f621855","Type":"ContainerStarted","Data":"52ff4716645cff7e02a96dc5b6fcd7743cb774af95f93f94d9e4d4e3eb9f7a3b"}
Oct 30 13:40:08.623013 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:08.622993    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:08.623927 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:08.623906    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:08.672948 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:08.672898    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0e874e-xwghg" podStartSLOduration=1.672884569 podStartE2EDuration="1.672884569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:08.672156265 +0000 UTC m=+3303.719290219" watchObservedRunningTime="2026-10-30 13:40:08.672884569 +0000 UTC m=+3303.720018523"
Oct 30 13:40:08.769823 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:08.769775    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"]
Oct 30 13:40:08.965211 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:08.965123    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:09.265549 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:09.265452    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:09.419601 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:09.419562    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:09.632307 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:09.632270    4544 generic.go:356] "Generic (PLEG): container finished" podID="a18b9841-be8b-49c2-9676-87f53f621855" containerID="4fdbb923ca53a5dae50a6c46d1958622057861b7907a7ec0305f0f0a93d12b7f" exitCode=0
Oct 30 13:40:09.632550 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:09.632358    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0e874e-xwghg" event={"ID":"a18b9841-be8b-49c2-9676-87f53f621855","Type":"ContainerDied","Data":"4fdbb923ca53a5dae50a6c46d1958622057861b7907a7ec0305f0f0a93d12b7f"}
Oct 30 13:40:10.110876 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:10.110838    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw"]
Oct 30 13:40:10.133004 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:10.132959    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5"]
Oct 30 13:40:10.470068 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:10.469949    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:10.656342 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:10.656297    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:10.821877 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:10.821846    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0e874e-xwghg"
Oct 30 13:40:10.883361 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:10.883323    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:10.995040 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:10.995007    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a18b9841-be8b-49c2-9676-87f53f621855-kube-api-access-l9dkh\" (UniqueName: \"kubernetes.io/projected/a18b9841-be8b-49c2-9676-87f53f621855-kube-api-access-l9dkh\") pod \"a18b9841-be8b-49c2-9676-87f53f621855\" (UID: \"a18b9841-be8b-49c2-9676-87f53f621855\") "
Oct 30 13:40:10.995283 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:10.995083    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a18b9841-be8b-49c2-9676-87f53f621855-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a18b9841-be8b-49c2-9676-87f53f621855-maas-api-xks-ca\") pod \"a18b9841-be8b-49c2-9676-87f53f621855\" (UID: \"a18b9841-be8b-49c2-9676-87f53f621855\") "
Oct 30 13:40:10.995283 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:10.995270    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a18b9841-be8b-49c2-9676-87f53f621855-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a18b9841-be8b-49c2-9676-87f53f621855-maas-api-openshift-ca\") pod \"a18b9841-be8b-49c2-9676-87f53f621855\" (UID: \"a18b9841-be8b-49c2-9676-87f53f621855\") "
Oct 30 13:40:10.995682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:10.995641    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a18b9841-be8b-49c2-9676-87f53f621855-maas-api-openshift-ca" pod "a18b9841-be8b-49c2-9676-87f53f621855" (UID: "a18b9841-be8b-49c2-9676-87f53f621855"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:10.998184 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:10.998143    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a18b9841-be8b-49c2-9676-87f53f621855-maas-api-xks-ca" pod "a18b9841-be8b-49c2-9676-87f53f621855" (UID: "a18b9841-be8b-49c2-9676-87f53f621855"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:40:11.005535 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.005505    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a18b9841-be8b-49c2-9676-87f53f621855-kube-api-access-l9dkh" pod "a18b9841-be8b-49c2-9676-87f53f621855" (UID: "a18b9841-be8b-49c2-9676-87f53f621855"). InnerVolumeSpecName "kube-api-access-l9dkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:11.096434 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.096410    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a18b9841-be8b-49c2-9676-87f53f621855-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:11.096544 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.096437    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9dkh\" (UniqueName: \"kubernetes.io/projected/a18b9841-be8b-49c2-9676-87f53f621855-kube-api-access-l9dkh\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:11.096544 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.096447    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a18b9841-be8b-49c2-9676-87f53f621855-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:11.262648 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.262619    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:40:11.400383 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.400288    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-istio-envoy\") pod \"09f03ae5-629b-4a35-8698-0043f221428d\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") "
Oct 30 13:40:11.400383 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.400334    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-workload-socket\") pod \"09f03ae5-629b-4a35-8698-0043f221428d\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") "
Oct 30 13:40:11.400649 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.400402    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09f03ae5-629b-4a35-8698-0043f221428d-kube-api-access-mkbbv\" (UniqueName: \"kubernetes.io/projected/09f03ae5-629b-4a35-8698-0043f221428d-kube-api-access-mkbbv\") pod \"09f03ae5-629b-4a35-8698-0043f221428d\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") "
Oct 30 13:40:11.400649 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.400451    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09f03ae5-629b-4a35-8698-0043f221428d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f03ae5-629b-4a35-8698-0043f221428d-istiod-ca-cert\") pod \"09f03ae5-629b-4a35-8698-0043f221428d\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") "
Oct 30 13:40:11.400649 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.400501    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-workload-certs\") pod \"09f03ae5-629b-4a35-8698-0043f221428d\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") "
Oct 30 13:40:11.400649 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.400586    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-credential-socket\") pod \"09f03ae5-629b-4a35-8698-0043f221428d\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") "
Oct 30 13:40:11.400876 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.400656    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09f03ae5-629b-4a35-8698-0043f221428d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f03ae5-629b-4a35-8698-0043f221428d-istio-podinfo\") pod \"09f03ae5-629b-4a35-8698-0043f221428d\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") "
Oct 30 13:40:11.400876 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.400690    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-istio-data\") pod \"09f03ae5-629b-4a35-8698-0043f221428d\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") "
Oct 30 13:40:11.400876 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.400723    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09f03ae5-629b-4a35-8698-0043f221428d-istio-token\" (UniqueName: \"kubernetes.io/projected/09f03ae5-629b-4a35-8698-0043f221428d-istio-token\") pod \"09f03ae5-629b-4a35-8698-0043f221428d\" (UID: \"09f03ae5-629b-4a35-8698-0043f221428d\") "
Oct 30 13:40:11.400876 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.400719    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-workload-socket" pod "09f03ae5-629b-4a35-8698-0043f221428d" (UID: "09f03ae5-629b-4a35-8698-0043f221428d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:11.401117 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.400952    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-workload-certs" pod "09f03ae5-629b-4a35-8698-0043f221428d" (UID: "09f03ae5-629b-4a35-8698-0043f221428d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:11.401117 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.401033    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-credential-socket" pod "09f03ae5-629b-4a35-8698-0043f221428d" (UID: "09f03ae5-629b-4a35-8698-0043f221428d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:11.401290 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.401256    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-istio-data" pod "09f03ae5-629b-4a35-8698-0043f221428d" (UID: "09f03ae5-629b-4a35-8698-0043f221428d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:11.401428 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.401396    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:11.401510 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.401425    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:11.401510 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.401442    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:11.401510 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.401455    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:11.401686 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.401509    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09f03ae5-629b-4a35-8698-0043f221428d-istiod-ca-cert" pod "09f03ae5-629b-4a35-8698-0043f221428d" (UID: "09f03ae5-629b-4a35-8698-0043f221428d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:11.402866 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.402841    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-istio-envoy" pod "09f03ae5-629b-4a35-8698-0043f221428d" (UID: "09f03ae5-629b-4a35-8698-0043f221428d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:11.402955 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.402931    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09f03ae5-629b-4a35-8698-0043f221428d-istio-podinfo" pod "09f03ae5-629b-4a35-8698-0043f221428d" (UID: "09f03ae5-629b-4a35-8698-0043f221428d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:40:11.403163 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.403141    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f03ae5-629b-4a35-8698-0043f221428d-kube-api-access-mkbbv" pod "09f03ae5-629b-4a35-8698-0043f221428d" (UID: "09f03ae5-629b-4a35-8698-0043f221428d"). InnerVolumeSpecName "kube-api-access-mkbbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:11.403268 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.403177    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f03ae5-629b-4a35-8698-0043f221428d-istio-token" pod "09f03ae5-629b-4a35-8698-0043f221428d" (UID: "09f03ae5-629b-4a35-8698-0043f221428d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:11.480875 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.480836    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"]
Oct 30 13:40:11.502294 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.502259    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f03ae5-629b-4a35-8698-0043f221428d-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:11.502294 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.502286    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09f03ae5-629b-4a35-8698-0043f221428d-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:11.502294 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.502295    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f03ae5-629b-4a35-8698-0043f221428d-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:11.502541 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.502303    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkbbv\" (UniqueName: \"kubernetes.io/projected/09f03ae5-629b-4a35-8698-0043f221428d-kube-api-access-mkbbv\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:11.502541 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.502314    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f03ae5-629b-4a35-8698-0043f221428d-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:11.654091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.653988    4544 generic.go:356] "Generic (PLEG): container finished" podID="09f03ae5-629b-4a35-8698-0043f221428d" containerID="731b2617b6db797a5b6f2dd2610ba69d6340c307926fee8f2c2c00599bf2cecc" exitCode=0
Oct 30 13:40:11.654091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.654035    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"
Oct 30 13:40:11.654091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.654069    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz" event={"ID":"09f03ae5-629b-4a35-8698-0043f221428d","Type":"ContainerDied","Data":"731b2617b6db797a5b6f2dd2610ba69d6340c307926fee8f2c2c00599bf2cecc"}
Oct 30 13:40:11.654409 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.654128    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz" event={"ID":"09f03ae5-629b-4a35-8698-0043f221428d","Type":"ContainerDied","Data":"2600921f1c88052af48eb51fd648b4f2bd607ec2f88026e2b0fad98be656b4af"}
Oct 30 13:40:11.654409 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.654157    4544 scope.go:122] "RemoveContainer" containerID="731b2617b6db797a5b6f2dd2610ba69d6340c307926fee8f2c2c00599bf2cecc"
Oct 30 13:40:11.655911 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.655892    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0e874e-xwghg"
Oct 30 13:40:11.656004 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.655892    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0e874e-xwghg" event={"ID":"a18b9841-be8b-49c2-9676-87f53f621855","Type":"ContainerDied","Data":"52ff4716645cff7e02a96dc5b6fcd7743cb774af95f93f94d9e4d4e3eb9f7a3b"}
Oct 30 13:40:11.656060 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.656024    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52ff4716645cff7e02a96dc5b6fcd7743cb774af95f93f94d9e4d4e3eb9f7a3b"
Oct 30 13:40:11.686515 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.686478    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:11.726164 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.726143    4544 scope.go:122] "RemoveContainer" containerID="731b2617b6db797a5b6f2dd2610ba69d6340c307926fee8f2c2c00599bf2cecc"
Oct 30 13:40:11.726750 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:40:11.726664    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"731b2617b6db797a5b6f2dd2610ba69d6340c307926fee8f2c2c00599bf2cecc\": container with ID starting with 731b2617b6db797a5b6f2dd2610ba69d6340c307926fee8f2c2c00599bf2cecc not found: ID does not exist" containerID="731b2617b6db797a5b6f2dd2610ba69d6340c307926fee8f2c2c00599bf2cecc"
Oct 30 13:40:11.726750 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.726709    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"731b2617b6db797a5b6f2dd2610ba69d6340c307926fee8f2c2c00599bf2cecc"} err="failed to get container status \"731b2617b6db797a5b6f2dd2610ba69d6340c307926fee8f2c2c00599bf2cecc\": rpc error: code = NotFound desc = could not find container \"731b2617b6db797a5b6f2dd2610ba69d6340c307926fee8f2c2c00599bf2cecc\": container with ID starting with 731b2617b6db797a5b6f2dd2610ba69d6340c307926fee8f2c2c00599bf2cecc not found: ID does not exist"
Oct 30 13:40:11.750550 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.750515    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"]
Oct 30 13:40:11.757370 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.757337    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz"]
Oct 30 13:40:11.894659 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.894625    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:11.911210 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.911127    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:40:11.965061 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.965013    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6"]
Oct 30 13:40:11.965662 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.965645    4544 cpu_manager.go:414] "containerMap: removing container" podUID="09f03ae5-629b-4a35-8698-0043f221428d" containerName="istio-proxy"
Oct 30 13:40:11.965734 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.965663    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09f03ae5-629b-4a35-8698-0043f221428d" containerName="istio-proxy"
Oct 30 13:40:11.965734 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.965687    4544 cpu_manager.go:414] "containerMap: removing container" podUID="e308ff03-37ae-4b7f-b27e-05b2f46ce66c" containerName="istio-proxy"
Oct 30 13:40:11.965734 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.965695    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e308ff03-37ae-4b7f-b27e-05b2f46ce66c" containerName="istio-proxy"
Oct 30 13:40:11.965734 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.965716    4544 cpu_manager.go:414] "containerMap: removing container" podUID="a18b9841-be8b-49c2-9676-87f53f621855" containerName="revoke-keys"
Oct 30 13:40:11.965734 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.965722    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a18b9841-be8b-49c2-9676-87f53f621855" containerName="revoke-keys"
Oct 30 13:40:11.965908 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.965829    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="a18b9841-be8b-49c2-9676-87f53f621855" containerName="revoke-keys"
Oct 30 13:40:11.965908 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.965839    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="e308ff03-37ae-4b7f-b27e-05b2f46ce66c" containerName="istio-proxy"
Oct 30 13:40:11.965908 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.965849    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="09f03ae5-629b-4a35-8698-0043f221428d" containerName="istio-proxy"
Oct 30 13:40:11.969437 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:11.969410    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6"
Oct 30 13:40:12.002896 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.002858    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6"]
Oct 30 13:40:12.009192 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.009161    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istio-data\") pod \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") "
Oct 30 13:40:12.009397 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.009302    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-credential-socket\") pod \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") "
Oct 30 13:40:12.009397 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.009340    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-workload-socket\") pod \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") "
Oct 30 13:40:12.009526 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.009393    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istiod-ca-cert\") pod \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") "
Oct 30 13:40:12.009526 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.009477    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-kube-api-access-vlm9d\" (UniqueName: \"kubernetes.io/projected/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-kube-api-access-vlm9d\") pod \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") "
Oct 30 13:40:12.009635 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.009548    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istio-podinfo\") pod \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") "
Oct 30 13:40:12.009635 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.009602    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istio-envoy\") pod \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") "
Oct 30 13:40:12.009741 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.009658    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istio-token\" (UniqueName: \"kubernetes.io/projected/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istio-token\") pod \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") "
Oct 30 13:40:12.009741 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.009708    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-workload-certs\") pod \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\" (UID: \"e308ff03-37ae-4b7f-b27e-05b2f46ce66c\") "
Oct 30 13:40:12.009927 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.009906    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99dd854e-d12d-4624-9c56-eeea387987cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6\" (UID: \"99dd854e-d12d-4624-9c56-eeea387987cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6"
Oct 30 13:40:12.010150 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.010133    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4pc8\" (UniqueName: \"kubernetes.io/projected/99dd854e-d12d-4624-9c56-eeea387987cd-kube-api-access-j4pc8\") pod \"payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6\" (UID: \"99dd854e-d12d-4624-9c56-eeea387987cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6"
Oct 30 13:40:12.010480 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.010458    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istio-data" pod "e308ff03-37ae-4b7f-b27e-05b2f46ce66c" (UID: "e308ff03-37ae-4b7f-b27e-05b2f46ce66c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:12.010650 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.010634    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-credential-socket" pod "e308ff03-37ae-4b7f-b27e-05b2f46ce66c" (UID: "e308ff03-37ae-4b7f-b27e-05b2f46ce66c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:12.010829 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.010810    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-workload-socket" pod "e308ff03-37ae-4b7f-b27e-05b2f46ce66c" (UID: "e308ff03-37ae-4b7f-b27e-05b2f46ce66c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:12.011055 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.011038    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istiod-ca-cert" pod "e308ff03-37ae-4b7f-b27e-05b2f46ce66c" (UID: "e308ff03-37ae-4b7f-b27e-05b2f46ce66c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:12.011644 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.011614    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-workload-certs" pod "e308ff03-37ae-4b7f-b27e-05b2f46ce66c" (UID: "e308ff03-37ae-4b7f-b27e-05b2f46ce66c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:12.011745 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.011707    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-87e742\""
Oct 30 13:40:12.013692 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.013664    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istio-token" pod "e308ff03-37ae-4b7f-b27e-05b2f46ce66c" (UID: "e308ff03-37ae-4b7f-b27e-05b2f46ce66c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:12.013794 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.013719    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istio-podinfo" pod "e308ff03-37ae-4b7f-b27e-05b2f46ce66c" (UID: "e308ff03-37ae-4b7f-b27e-05b2f46ce66c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:40:12.013794 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.013780    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-kube-api-access-vlm9d" pod "e308ff03-37ae-4b7f-b27e-05b2f46ce66c" (UID: "e308ff03-37ae-4b7f-b27e-05b2f46ce66c"). InnerVolumeSpecName "kube-api-access-vlm9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:12.013979 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.013959    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istio-envoy" pod "e308ff03-37ae-4b7f-b27e-05b2f46ce66c" (UID: "e308ff03-37ae-4b7f-b27e-05b2f46ce66c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:12.111540 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.111503    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4pc8\" (UniqueName: \"kubernetes.io/projected/99dd854e-d12d-4624-9c56-eeea387987cd-kube-api-access-j4pc8\") pod \"payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6\" (UID: \"99dd854e-d12d-4624-9c56-eeea387987cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6"
Oct 30 13:40:12.111540 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.111560    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99dd854e-d12d-4624-9c56-eeea387987cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6\" (UID: \"99dd854e-d12d-4624-9c56-eeea387987cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6"
Oct 30 13:40:12.111828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.111775    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:12.111828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.111799    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:12.111828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.111811    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:12.111828 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.111824    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:12.112001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.111838    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:12.112001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.111861    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:12.112001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.111878    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:12.112001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.111891    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:12.112001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.111904    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlm9d\" (UniqueName: \"kubernetes.io/projected/e308ff03-37ae-4b7f-b27e-05b2f46ce66c-kube-api-access-vlm9d\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:12.112427 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.112409    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99dd854e-d12d-4624-9c56-eeea387987cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6\" (UID: \"99dd854e-d12d-4624-9c56-eeea387987cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6"
Oct 30 13:40:12.140329 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.140298    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4pc8\" (UniqueName: \"kubernetes.io/projected/99dd854e-d12d-4624-9c56-eeea387987cd-kube-api-access-j4pc8\") pod \"payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6\" (UID: \"99dd854e-d12d-4624-9c56-eeea387987cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6"
Oct 30 13:40:12.240215 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.240130    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:12.283593 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.283563    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6"
Oct 30 13:40:12.611723 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.611607    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6"]
Oct 30 13:40:12.614887 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:40:12.614858    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99dd854e_d12d_4624_9c56_eeea387987cd.slice/crio-78e93642639d5d0a9b04c284a5fa59d107cb40c917214a104d7ac7a0fb87a68b WatchSource:0}: Error finding container 78e93642639d5d0a9b04c284a5fa59d107cb40c917214a104d7ac7a0fb87a68b: Status 404 returned error can't find the container with id 78e93642639d5d0a9b04c284a5fa59d107cb40c917214a104d7ac7a0fb87a68b
Oct 30 13:40:12.666603 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.666557    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6" event={"ID":"99dd854e-d12d-4624-9c56-eeea387987cd","Type":"ContainerStarted","Data":"78e93642639d5d0a9b04c284a5fa59d107cb40c917214a104d7ac7a0fb87a68b"}
Oct 30 13:40:12.669016 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.668959    4544 generic.go:356] "Generic (PLEG): container finished" podID="e308ff03-37ae-4b7f-b27e-05b2f46ce66c" containerID="6aa0b9aac2f00d985e160d270a25eedf47c770451d610e36f60c1829e38b0023" exitCode=0
Oct 30 13:40:12.669271 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.669040    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"
Oct 30 13:40:12.669271 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.669039    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs" event={"ID":"e308ff03-37ae-4b7f-b27e-05b2f46ce66c","Type":"ContainerDied","Data":"6aa0b9aac2f00d985e160d270a25eedf47c770451d610e36f60c1829e38b0023"}
Oct 30 13:40:12.669271 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.669239    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs" event={"ID":"e308ff03-37ae-4b7f-b27e-05b2f46ce66c","Type":"ContainerDied","Data":"1f9e04a17caeb8e61d9e42180fa4636b2b7b5134ad453d9e11fad111510649bc"}
Oct 30 13:40:12.669444 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.669282    4544 scope.go:122] "RemoveContainer" containerID="6aa0b9aac2f00d985e160d270a25eedf47c770451d610e36f60c1829e38b0023"
Oct 30 13:40:12.695800 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.695773    4544 scope.go:122] "RemoveContainer" containerID="804ab07a9a2dd4eeca0ccd71e14f562405ef85199228a5aeb185b2b29ec62402"
Oct 30 13:40:12.706194 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.706169    4544 scope.go:122] "RemoveContainer" containerID="6aa0b9aac2f00d985e160d270a25eedf47c770451d610e36f60c1829e38b0023"
Oct 30 13:40:12.706565 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:40:12.706536    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aa0b9aac2f00d985e160d270a25eedf47c770451d610e36f60c1829e38b0023\": container with ID starting with 6aa0b9aac2f00d985e160d270a25eedf47c770451d610e36f60c1829e38b0023 not found: ID does not exist" containerID="6aa0b9aac2f00d985e160d270a25eedf47c770451d610e36f60c1829e38b0023"
Oct 30 13:40:12.706624 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.706575    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aa0b9aac2f00d985e160d270a25eedf47c770451d610e36f60c1829e38b0023"} err="failed to get container status \"6aa0b9aac2f00d985e160d270a25eedf47c770451d610e36f60c1829e38b0023\": rpc error: code = NotFound desc = could not find container \"6aa0b9aac2f00d985e160d270a25eedf47c770451d610e36f60c1829e38b0023\": container with ID starting with 6aa0b9aac2f00d985e160d270a25eedf47c770451d610e36f60c1829e38b0023 not found: ID does not exist"
Oct 30 13:40:12.706624 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.706593    4544 scope.go:122] "RemoveContainer" containerID="804ab07a9a2dd4eeca0ccd71e14f562405ef85199228a5aeb185b2b29ec62402"
Oct 30 13:40:12.706900 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:40:12.706880    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"804ab07a9a2dd4eeca0ccd71e14f562405ef85199228a5aeb185b2b29ec62402\": container with ID starting with 804ab07a9a2dd4eeca0ccd71e14f562405ef85199228a5aeb185b2b29ec62402 not found: ID does not exist" containerID="804ab07a9a2dd4eeca0ccd71e14f562405ef85199228a5aeb185b2b29ec62402"
Oct 30 13:40:12.706945 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.706908    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"804ab07a9a2dd4eeca0ccd71e14f562405ef85199228a5aeb185b2b29ec62402"} err="failed to get container status \"804ab07a9a2dd4eeca0ccd71e14f562405ef85199228a5aeb185b2b29ec62402\": rpc error: code = NotFound desc = could not find container \"804ab07a9a2dd4eeca0ccd71e14f562405ef85199228a5aeb185b2b29ec62402\": container with ID starting with 804ab07a9a2dd4eeca0ccd71e14f562405ef85199228a5aeb185b2b29ec62402 not found: ID does not exist"
Oct 30 13:40:12.732017 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.731988    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"]
Oct 30 13:40:12.743937 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:12.743904    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs"]
Oct 30 13:40:13.087101 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:13.087068    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09f03ae5-629b-4a35-8698-0043f221428d" path="/var/lib/kubelet/pods/09f03ae5-629b-4a35-8698-0043f221428d/volumes"
Oct 30 13:40:13.087603 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:13.087588    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e308ff03-37ae-4b7f-b27e-05b2f46ce66c" path="/var/lib/kubelet/pods/e308ff03-37ae-4b7f-b27e-05b2f46ce66c/volumes"
Oct 30 13:40:13.121920 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:13.121870    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:13.681739 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:13.681704    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6" event={"ID":"99dd854e-d12d-4624-9c56-eeea387987cd","Type":"ContainerStarted","Data":"1f1adfe3df9674ee53b9538b877f5db8194e7bd59b3c6cff1a3942057dae9f43"}
Oct 30 13:40:13.682259 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:13.681810    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6"
Oct 30 13:40:13.734705 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:13.734651    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6" podStartSLOduration=2.7346360819999997 podStartE2EDuration="2.734636082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:13.733624383 +0000 UTC m=+3308.780758338" watchObservedRunningTime="2026-10-30 13:40:13.734636082 +0000 UTC m=+3308.781770036"
Oct 30 13:40:14.449571 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:14.449528    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9"]
Oct 30 13:40:14.488012 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:14.487974    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j"]
Oct 30 13:40:14.633890 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:14.633842    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:14.811575 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:14.811479    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:15.148246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:15.133306    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw" podUID="db2e31d3-d423-4671-8449-56d1f380fbd1" containerName="payload-processing" containerID="cri-o://22d31b784325aa478dba1c843a61e3dc2db7964039fc1a4a989dc90f8acd5862" gracePeriod=25
Oct 30 13:40:15.166349 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:15.166304    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5" podUID="8d82c543-383d-4498-8d3c-d6c2a0716736" containerName="payload-pre-processing" containerID="cri-o://564b7c69980224c672c885ff533d40a4f70c9a6e43a071491ae939a39f83cfd6" gracePeriod=25
Oct 30 13:40:15.322538 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:15.322448    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:17.181766 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:17.181720    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9"]
Oct 30 13:40:17.249678 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:17.249635    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p"]
Oct 30 13:40:17.343119 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:17.343085    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:17.468245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:17.466901    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:17.584945 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:17.584910    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:18.189761 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:18.189709    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:18.308364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:18.308308    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:18.991924 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:18.991884    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g"]
Oct 30 13:40:18.992152 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:18.992086    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g" podUID="3a19094f-f1fc-4f65-9702-96dc3a0a2f77" containerName="maas-api" containerID="cri-o://50bca92b8622c87555e4c837d5ace1373ea410b1980770fee53d7ff383249a05" gracePeriod=30
Oct 30 13:40:19.050582 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:40:19.050538    4544 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a19094f_f1fc_4f65_9702_96dc3a0a2f77.slice/crio-conmon-50bca92b8622c87555e4c837d5ace1373ea410b1980770fee53d7ff383249a05.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a19094f_f1fc_4f65_9702_96dc3a0a2f77.slice/crio-50bca92b8622c87555e4c837d5ace1373ea410b1980770fee53d7ff383249a05.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 13:40:19.073006 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.072954    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:19.276422 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.276333    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g"
Oct 30 13:40:19.327665 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.327631    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a19094f-f1fc-4f65-9702-96dc3a0a2f77-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a19094f-f1fc-4f65-9702-96dc3a0a2f77-maas-api-tls\") pod \"3a19094f-f1fc-4f65-9702-96dc3a0a2f77\" (UID: \"3a19094f-f1fc-4f65-9702-96dc3a0a2f77\") "
Oct 30 13:40:19.327881 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.327740    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a19094f-f1fc-4f65-9702-96dc3a0a2f77-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a19094f-f1fc-4f65-9702-96dc3a0a2f77-metrics-certs\") pod \"3a19094f-f1fc-4f65-9702-96dc3a0a2f77\" (UID: \"3a19094f-f1fc-4f65-9702-96dc3a0a2f77\") "
Oct 30 13:40:19.327881 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.327798    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a19094f-f1fc-4f65-9702-96dc3a0a2f77-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a19094f-f1fc-4f65-9702-96dc3a0a2f77-openshift-service-ca\") pod \"3a19094f-f1fc-4f65-9702-96dc3a0a2f77\" (UID: \"3a19094f-f1fc-4f65-9702-96dc3a0a2f77\") "
Oct 30 13:40:19.327881 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.327856    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a19094f-f1fc-4f65-9702-96dc3a0a2f77-kube-api-access-z42xr\" (UniqueName: \"kubernetes.io/projected/3a19094f-f1fc-4f65-9702-96dc3a0a2f77-kube-api-access-z42xr\") pod \"3a19094f-f1fc-4f65-9702-96dc3a0a2f77\" (UID: \"3a19094f-f1fc-4f65-9702-96dc3a0a2f77\") "
Oct 30 13:40:19.328307 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.328272    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a19094f-f1fc-4f65-9702-96dc3a0a2f77-openshift-service-ca" pod "3a19094f-f1fc-4f65-9702-96dc3a0a2f77" (UID: "3a19094f-f1fc-4f65-9702-96dc3a0a2f77"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:19.330266 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.330199    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a19094f-f1fc-4f65-9702-96dc3a0a2f77-metrics-certs" pod "3a19094f-f1fc-4f65-9702-96dc3a0a2f77" (UID: "3a19094f-f1fc-4f65-9702-96dc3a0a2f77"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:40:19.330548 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.330530    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a19094f-f1fc-4f65-9702-96dc3a0a2f77-kube-api-access-z42xr" pod "3a19094f-f1fc-4f65-9702-96dc3a0a2f77" (UID: "3a19094f-f1fc-4f65-9702-96dc3a0a2f77"). InnerVolumeSpecName "kube-api-access-z42xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:19.330643 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.330561    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a19094f-f1fc-4f65-9702-96dc3a0a2f77-maas-api-tls" pod "3a19094f-f1fc-4f65-9702-96dc3a0a2f77" (UID: "3a19094f-f1fc-4f65-9702-96dc3a0a2f77"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:40:19.429065 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.429024    4544 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a19094f-f1fc-4f65-9702-96dc3a0a2f77-metrics-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:19.429065 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.429062    4544 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a19094f-f1fc-4f65-9702-96dc3a0a2f77-openshift-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:19.429065 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.429075    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z42xr\" (UniqueName: \"kubernetes.io/projected/3a19094f-f1fc-4f65-9702-96dc3a0a2f77-kube-api-access-z42xr\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:19.429359 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.429084    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a19094f-f1fc-4f65-9702-96dc3a0a2f77-maas-api-tls\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:19.435648 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.435342    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13fe37db-b66zr"]
Oct 30 13:40:19.436552 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.436522    4544 cpu_manager.go:414] "containerMap: removing container" podUID="e308ff03-37ae-4b7f-b27e-05b2f46ce66c" containerName="istio-proxy"
Oct 30 13:40:19.436855 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.436817    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e308ff03-37ae-4b7f-b27e-05b2f46ce66c" containerName="istio-proxy"
Oct 30 13:40:19.437013 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.436915    4544 cpu_manager.go:414] "containerMap: removing container" podUID="3a19094f-f1fc-4f65-9702-96dc3a0a2f77" containerName="maas-api"
Oct 30 13:40:19.437013 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.436924    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a19094f-f1fc-4f65-9702-96dc3a0a2f77" containerName="maas-api"
Oct 30 13:40:19.437157 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.437068    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="e308ff03-37ae-4b7f-b27e-05b2f46ce66c" containerName="istio-proxy"
Oct 30 13:40:19.437157 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.437087    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a19094f-f1fc-4f65-9702-96dc3a0a2f77" containerName="maas-api"
Oct 30 13:40:19.441909 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.441870    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fe37db-b66zr"
Oct 30 13:40:19.450183 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.450103    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9" podUID="7692a16a-4d89-4f37-85ae-6c43dad93d35" containerName="payload-processing" containerID="cri-o://7477f63cb419fb6a54ccbdb71cd5379f790de4aceb42ecd453a5cc3ce70f76d2" gracePeriod=25
Oct 30 13:40:19.479849 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.465121    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13fe37db-b66zr"]
Oct 30 13:40:19.479849 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.465554    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:40:19.479849 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.465695    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:40:19.496377 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.496324    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j" podUID="cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb" containerName="payload-pre-processing" containerID="cri-o://abc3b345cdc771fc118889cfa8168a951045cee88b541355d43b90069fc23793" gracePeriod=25
Oct 30 13:40:19.530140 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.530038    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19afa216-d512-4110-8641-bdfa5e42c7fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13fe37db-b66zr\" (UID: \"19afa216-d512-4110-8641-bdfa5e42c7fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fe37db-b66zr"
Oct 30 13:40:19.530423 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.530194    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjdwk\" (UniqueName: \"kubernetes.io/projected/19afa216-d512-4110-8641-bdfa5e42c7fd-kube-api-access-xjdwk\") pod \"maas-api-revoke-sub-13fe37db-b66zr\" (UID: \"19afa216-d512-4110-8641-bdfa5e42c7fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fe37db-b66zr"
Oct 30 13:40:19.530423 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.530289    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19afa216-d512-4110-8641-bdfa5e42c7fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13fe37db-b66zr\" (UID: \"19afa216-d512-4110-8641-bdfa5e42c7fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fe37db-b66zr"
Oct 30 13:40:19.538911 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.538871    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16e62535-gwl8v"]
Oct 30 13:40:19.542952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.542925    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16e62535-gwl8v"
Oct 30 13:40:19.578628 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.578591    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16e62535-gwl8v"]
Oct 30 13:40:19.583127 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.583072    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b45652a-nxdb5"]
Oct 30 13:40:19.587875 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.587845    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b45652a-nxdb5"
Oct 30 13:40:19.608423 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.608382    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw" podUID="db2e31d3-d423-4671-8449-56d1f380fbd1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:40:19.609765 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.609656    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:19.610646 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.610619    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b45652a-nxdb5"]
Oct 30 13:40:19.612345 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.612313    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5" podUID="8d82c543-383d-4498-8d3c-d6c2a0716736" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:40:19.630909 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.630865    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84aeefc2-4217-4779-904d-e081cca145bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16e62535-gwl8v\" (UID: \"84aeefc2-4217-4779-904d-e081cca145bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16e62535-gwl8v"
Oct 30 13:40:19.631136 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.630921    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjdwk\" (UniqueName: \"kubernetes.io/projected/19afa216-d512-4110-8641-bdfa5e42c7fd-kube-api-access-xjdwk\") pod \"maas-api-revoke-sub-13fe37db-b66zr\" (UID: \"19afa216-d512-4110-8641-bdfa5e42c7fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fe37db-b66zr"
Oct 30 13:40:19.631136 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.631039    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2msgh\" (UniqueName: \"kubernetes.io/projected/84aeefc2-4217-4779-904d-e081cca145bc-kube-api-access-2msgh\") pod \"maas-api-revoke-sub-16e62535-gwl8v\" (UID: \"84aeefc2-4217-4779-904d-e081cca145bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16e62535-gwl8v"
Oct 30 13:40:19.631136 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.631115    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df75e04-9e0b-46b5-8a95-6b4be84e3314-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b45652a-nxdb5\" (UID: \"9df75e04-9e0b-46b5-8a95-6b4be84e3314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b45652a-nxdb5"
Oct 30 13:40:19.631378 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.631167    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84aeefc2-4217-4779-904d-e081cca145bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16e62535-gwl8v\" (UID: \"84aeefc2-4217-4779-904d-e081cca145bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16e62535-gwl8v"
Oct 30 13:40:19.631378 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.631337    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df75e04-9e0b-46b5-8a95-6b4be84e3314-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b45652a-nxdb5\" (UID: \"9df75e04-9e0b-46b5-8a95-6b4be84e3314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b45652a-nxdb5"
Oct 30 13:40:19.631493 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.631383    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19afa216-d512-4110-8641-bdfa5e42c7fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13fe37db-b66zr\" (UID: \"19afa216-d512-4110-8641-bdfa5e42c7fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fe37db-b66zr"
Oct 30 13:40:19.631579 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.631560    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19afa216-d512-4110-8641-bdfa5e42c7fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13fe37db-b66zr\" (UID: \"19afa216-d512-4110-8641-bdfa5e42c7fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fe37db-b66zr"
Oct 30 13:40:19.631651 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.631601    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4hqb\" (UniqueName: \"kubernetes.io/projected/9df75e04-9e0b-46b5-8a95-6b4be84e3314-kube-api-access-z4hqb\") pod \"maas-api-revoke-sub-0b45652a-nxdb5\" (UID: \"9df75e04-9e0b-46b5-8a95-6b4be84e3314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b45652a-nxdb5"
Oct 30 13:40:19.632469 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.632439    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19afa216-d512-4110-8641-bdfa5e42c7fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13fe37db-b66zr\" (UID: \"19afa216-d512-4110-8641-bdfa5e42c7fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fe37db-b66zr"
Oct 30 13:40:19.633939 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.633907    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19afa216-d512-4110-8641-bdfa5e42c7fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13fe37db-b66zr\" (UID: \"19afa216-d512-4110-8641-bdfa5e42c7fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fe37db-b66zr"
Oct 30 13:40:19.657563 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.657532    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjdwk\" (UniqueName: \"kubernetes.io/projected/19afa216-d512-4110-8641-bdfa5e42c7fd-kube-api-access-xjdwk\") pod \"maas-api-revoke-sub-13fe37db-b66zr\" (UID: \"19afa216-d512-4110-8641-bdfa5e42c7fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fe37db-b66zr"
Oct 30 13:40:19.662465 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.662421    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" containerID="cri-o://8919966ee1cfb5400f07f9d90a533886b4de9baf5d7c84e98271d101ffbc6ec0" gracePeriod=45
Oct 30 13:40:19.731634 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.731589    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:19.731989 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.731963    4544 generic.go:356] "Generic (PLEG): container finished" podID="3a19094f-f1fc-4f65-9702-96dc3a0a2f77" containerID="50bca92b8622c87555e4c837d5ace1373ea410b1980770fee53d7ff383249a05" exitCode=0
Oct 30 13:40:19.732093 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.732022    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g"
Oct 30 13:40:19.732093 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.732060    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g" event={"ID":"3a19094f-f1fc-4f65-9702-96dc3a0a2f77","Type":"ContainerDied","Data":"50bca92b8622c87555e4c837d5ace1373ea410b1980770fee53d7ff383249a05"}
Oct 30 13:40:19.732259 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.732138    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g" event={"ID":"3a19094f-f1fc-4f65-9702-96dc3a0a2f77","Type":"ContainerDied","Data":"6076272dedd3cfac89c8a51174d182739ac8e6e80bb76b8682d5472eb7501a34"}
Oct 30 13:40:19.732259 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.732173    4544 scope.go:122] "RemoveContainer" containerID="50bca92b8622c87555e4c837d5ace1373ea410b1980770fee53d7ff383249a05"
Oct 30 13:40:19.732592 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.732569    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df75e04-9e0b-46b5-8a95-6b4be84e3314-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b45652a-nxdb5\" (UID: \"9df75e04-9e0b-46b5-8a95-6b4be84e3314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b45652a-nxdb5"
Oct 30 13:40:19.732671 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.732612    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84aeefc2-4217-4779-904d-e081cca145bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16e62535-gwl8v\" (UID: \"84aeefc2-4217-4779-904d-e081cca145bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16e62535-gwl8v"
Oct 30 13:40:19.732671 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.732650    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df75e04-9e0b-46b5-8a95-6b4be84e3314-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b45652a-nxdb5\" (UID: \"9df75e04-9e0b-46b5-8a95-6b4be84e3314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b45652a-nxdb5"
Oct 30 13:40:19.732785 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.732732    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4hqb\" (UniqueName: \"kubernetes.io/projected/9df75e04-9e0b-46b5-8a95-6b4be84e3314-kube-api-access-z4hqb\") pod \"maas-api-revoke-sub-0b45652a-nxdb5\" (UID: \"9df75e04-9e0b-46b5-8a95-6b4be84e3314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b45652a-nxdb5"
Oct 30 13:40:19.733013 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.732904    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84aeefc2-4217-4779-904d-e081cca145bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16e62535-gwl8v\" (UID: \"84aeefc2-4217-4779-904d-e081cca145bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16e62535-gwl8v"
Oct 30 13:40:19.733013 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.732972    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2msgh\" (UniqueName: \"kubernetes.io/projected/84aeefc2-4217-4779-904d-e081cca145bc-kube-api-access-2msgh\") pod \"maas-api-revoke-sub-16e62535-gwl8v\" (UID: \"84aeefc2-4217-4779-904d-e081cca145bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16e62535-gwl8v"
Oct 30 13:40:19.734129 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.734037    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df75e04-9e0b-46b5-8a95-6b4be84e3314-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b45652a-nxdb5\" (UID: \"9df75e04-9e0b-46b5-8a95-6b4be84e3314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b45652a-nxdb5"
Oct 30 13:40:19.735529 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.735505    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84aeefc2-4217-4779-904d-e081cca145bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16e62535-gwl8v\" (UID: \"84aeefc2-4217-4779-904d-e081cca145bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16e62535-gwl8v"
Oct 30 13:40:19.737829 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.737796    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df75e04-9e0b-46b5-8a95-6b4be84e3314-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b45652a-nxdb5\" (UID: \"9df75e04-9e0b-46b5-8a95-6b4be84e3314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b45652a-nxdb5"
Oct 30 13:40:19.738043 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.737828    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84aeefc2-4217-4779-904d-e081cca145bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16e62535-gwl8v\" (UID: \"84aeefc2-4217-4779-904d-e081cca145bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16e62535-gwl8v"
Oct 30 13:40:19.763371 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.763301    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2msgh\" (UniqueName: \"kubernetes.io/projected/84aeefc2-4217-4779-904d-e081cca145bc-kube-api-access-2msgh\") pod \"maas-api-revoke-sub-16e62535-gwl8v\" (UID: \"84aeefc2-4217-4779-904d-e081cca145bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16e62535-gwl8v"
Oct 30 13:40:19.764420 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.764298    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4hqb\" (UniqueName: \"kubernetes.io/projected/9df75e04-9e0b-46b5-8a95-6b4be84e3314-kube-api-access-z4hqb\") pod \"maas-api-revoke-sub-0b45652a-nxdb5\" (UID: \"9df75e04-9e0b-46b5-8a95-6b4be84e3314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b45652a-nxdb5"
Oct 30 13:40:19.770051 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.770018    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fe37db-b66zr"
Oct 30 13:40:19.781132 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.781100    4544 scope.go:122] "RemoveContainer" containerID="50bca92b8622c87555e4c837d5ace1373ea410b1980770fee53d7ff383249a05"
Oct 30 13:40:19.781586 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:40:19.781562    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50bca92b8622c87555e4c837d5ace1373ea410b1980770fee53d7ff383249a05\": container with ID starting with 50bca92b8622c87555e4c837d5ace1373ea410b1980770fee53d7ff383249a05 not found: ID does not exist" containerID="50bca92b8622c87555e4c837d5ace1373ea410b1980770fee53d7ff383249a05"
Oct 30 13:40:19.781689 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.781599    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50bca92b8622c87555e4c837d5ace1373ea410b1980770fee53d7ff383249a05"} err="failed to get container status \"50bca92b8622c87555e4c837d5ace1373ea410b1980770fee53d7ff383249a05\": rpc error: code = NotFound desc = could not find container \"50bca92b8622c87555e4c837d5ace1373ea410b1980770fee53d7ff383249a05\": container with ID starting with 50bca92b8622c87555e4c837d5ace1373ea410b1980770fee53d7ff383249a05 not found: ID does not exist"
Oct 30 13:40:19.786677 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.786548    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g"]
Oct 30 13:40:19.790412 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.790361    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3e8ee4-9bdc5586d-p278g"]
Oct 30 13:40:19.861280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.858564    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16e62535-gwl8v"
Oct 30 13:40:19.907019 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:19.906740    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b45652a-nxdb5"
Oct 30 13:40:20.016725 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.016343    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:40:20.037814 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.036540    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/404ea200-b989-4029-be47-f7075c3ed65e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/404ea200-b989-4029-be47-f7075c3ed65e-model-cache\") pod \"404ea200-b989-4029-be47-f7075c3ed65e\" (UID: \"404ea200-b989-4029-be47-f7075c3ed65e\") "
Oct 30 13:40:20.037814 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.036611    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/404ea200-b989-4029-be47-f7075c3ed65e-home\" (UniqueName: \"kubernetes.io/empty-dir/404ea200-b989-4029-be47-f7075c3ed65e-home\") pod \"404ea200-b989-4029-be47-f7075c3ed65e\" (UID: \"404ea200-b989-4029-be47-f7075c3ed65e\") "
Oct 30 13:40:20.037814 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.036734    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/404ea200-b989-4029-be47-f7075c3ed65e-tls-certs\" (UniqueName: \"kubernetes.io/secret/404ea200-b989-4029-be47-f7075c3ed65e-tls-certs\") pod \"404ea200-b989-4029-be47-f7075c3ed65e\" (UID: \"404ea200-b989-4029-be47-f7075c3ed65e\") "
Oct 30 13:40:20.037814 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.036772    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/404ea200-b989-4029-be47-f7075c3ed65e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/404ea200-b989-4029-be47-f7075c3ed65e-dshm\") pod \"404ea200-b989-4029-be47-f7075c3ed65e\" (UID: \"404ea200-b989-4029-be47-f7075c3ed65e\") "
Oct 30 13:40:20.037814 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.036858    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/404ea200-b989-4029-be47-f7075c3ed65e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/404ea200-b989-4029-be47-f7075c3ed65e-tmp-dir\") pod \"404ea200-b989-4029-be47-f7075c3ed65e\" (UID: \"404ea200-b989-4029-be47-f7075c3ed65e\") "
Oct 30 13:40:20.037814 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.036904    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/404ea200-b989-4029-be47-f7075c3ed65e-kube-api-access-mhpjc\" (UniqueName: \"kubernetes.io/projected/404ea200-b989-4029-be47-f7075c3ed65e-kube-api-access-mhpjc\") pod \"404ea200-b989-4029-be47-f7075c3ed65e\" (UID: \"404ea200-b989-4029-be47-f7075c3ed65e\") "
Oct 30 13:40:20.038411 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.038093    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404ea200-b989-4029-be47-f7075c3ed65e-tmp-dir" pod "404ea200-b989-4029-be47-f7075c3ed65e" (UID: "404ea200-b989-4029-be47-f7075c3ed65e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:20.038411 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.038303    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404ea200-b989-4029-be47-f7075c3ed65e-home" pod "404ea200-b989-4029-be47-f7075c3ed65e" (UID: "404ea200-b989-4029-be47-f7075c3ed65e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:20.038411 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.038373    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404ea200-b989-4029-be47-f7075c3ed65e-model-cache" pod "404ea200-b989-4029-be47-f7075c3ed65e" (UID: "404ea200-b989-4029-be47-f7075c3ed65e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:20.040944 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.040670    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/404ea200-b989-4029-be47-f7075c3ed65e-tls-certs" pod "404ea200-b989-4029-be47-f7075c3ed65e" (UID: "404ea200-b989-4029-be47-f7075c3ed65e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:40:20.041237 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.041066    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404ea200-b989-4029-be47-f7075c3ed65e-dshm" pod "404ea200-b989-4029-be47-f7075c3ed65e" (UID: "404ea200-b989-4029-be47-f7075c3ed65e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:20.043187 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.043158    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/404ea200-b989-4029-be47-f7075c3ed65e-kube-api-access-mhpjc" pod "404ea200-b989-4029-be47-f7075c3ed65e" (UID: "404ea200-b989-4029-be47-f7075c3ed65e"). InnerVolumeSpecName "kube-api-access-mhpjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:20.098117 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.096378    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13fe37db-b66zr"]
Oct 30 13:40:20.139091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.138939    4544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/404ea200-b989-4029-be47-f7075c3ed65e-home\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:20.139287 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.139161    4544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/404ea200-b989-4029-be47-f7075c3ed65e-tls-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:20.139287 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.139184    4544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/404ea200-b989-4029-be47-f7075c3ed65e-dshm\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:20.139287 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.139198    4544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/404ea200-b989-4029-be47-f7075c3ed65e-tmp-dir\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:20.139287 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.139215    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhpjc\" (UniqueName: \"kubernetes.io/projected/404ea200-b989-4029-be47-f7075c3ed65e-kube-api-access-mhpjc\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:20.139287 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.139255    4544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/404ea200-b989-4029-be47-f7075c3ed65e-model-cache\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:20.184739 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.184705    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:20.192087 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.192056    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16e62535-gwl8v"]
Oct 30 13:40:20.195108 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:40:20.195078    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84aeefc2_4217_4779_904d_e081cca145bc.slice/crio-4890ade682b6d6f5830f37a4f0119e41d807a19fe0e5f64c877777c3de0a5aed WatchSource:0}: Error finding container 4890ade682b6d6f5830f37a4f0119e41d807a19fe0e5f64c877777c3de0a5aed: Status 404 returned error can't find the container with id 4890ade682b6d6f5830f37a4f0119e41d807a19fe0e5f64c877777c3de0a5aed
Oct 30 13:40:20.254097 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.254067    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b45652a-nxdb5"]
Oct 30 13:40:20.257183 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:40:20.257137    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df75e04_9e0b_46b5_8a95_6b4be84e3314.slice/crio-5aa765a7562ab7aaacd39adae173e6a01af9d0e3eefabc117eb2bfb5696716d7 WatchSource:0}: Error finding container 5aa765a7562ab7aaacd39adae173e6a01af9d0e3eefabc117eb2bfb5696716d7: Status 404 returned error can't find the container with id 5aa765a7562ab7aaacd39adae173e6a01af9d0e3eefabc117eb2bfb5696716d7
Oct 30 13:40:20.484279 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.484210    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:20.742302 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.742269    4544 generic.go:356] "Generic (PLEG): container finished" podID="404ea200-b989-4029-be47-f7075c3ed65e" containerID="8919966ee1cfb5400f07f9d90a533886b4de9baf5d7c84e98271d101ffbc6ec0" exitCode=0
Oct 30 13:40:20.742516 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.742318    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"
Oct 30 13:40:20.742516 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.742358    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" event={"ID":"404ea200-b989-4029-be47-f7075c3ed65e","Type":"ContainerDied","Data":"8919966ee1cfb5400f07f9d90a533886b4de9baf5d7c84e98271d101ffbc6ec0"}
Oct 30 13:40:20.742516 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.742409    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" event={"ID":"404ea200-b989-4029-be47-f7075c3ed65e","Type":"ContainerDied","Data":"532b4634d1dbd31c2485b98c67c01d808aa0f0c03453dc2e06866b967ffc21ba"}
Oct 30 13:40:20.742516 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.742431    4544 scope.go:122] "RemoveContainer" containerID="8919966ee1cfb5400f07f9d90a533886b4de9baf5d7c84e98271d101ffbc6ec0"
Oct 30 13:40:20.744866 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.744790    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b45652a-nxdb5" event={"ID":"9df75e04-9e0b-46b5-8a95-6b4be84e3314","Type":"ContainerStarted","Data":"969d03d92ff38fb0e817d29c5a6985961f59fd377dcdf3e4565568404f3dc2a8"}
Oct 30 13:40:20.745043 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.744870    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b45652a-nxdb5" event={"ID":"9df75e04-9e0b-46b5-8a95-6b4be84e3314","Type":"ContainerStarted","Data":"5aa765a7562ab7aaacd39adae173e6a01af9d0e3eefabc117eb2bfb5696716d7"}
Oct 30 13:40:20.748214 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.748176    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16e62535-gwl8v" event={"ID":"84aeefc2-4217-4779-904d-e081cca145bc","Type":"ContainerStarted","Data":"544c5378f2f0780f40f13b42d38d6df4de0b2653e5e61e5b221ff8eedfe04c4b"}
Oct 30 13:40:20.748395 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.748281    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16e62535-gwl8v" event={"ID":"84aeefc2-4217-4779-904d-e081cca145bc","Type":"ContainerStarted","Data":"4890ade682b6d6f5830f37a4f0119e41d807a19fe0e5f64c877777c3de0a5aed"}
Oct 30 13:40:20.751372 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.751338    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fe37db-b66zr" event={"ID":"19afa216-d512-4110-8641-bdfa5e42c7fd","Type":"ContainerStarted","Data":"119be7047d43cc5c4e3c5561db2e655faf273f826cf8b0afed664b22449e163d"}
Oct 30 13:40:20.751527 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.751403    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fe37db-b66zr" event={"ID":"19afa216-d512-4110-8641-bdfa5e42c7fd","Type":"ContainerStarted","Data":"96bff2d8020a51199d508579debd39a9318bda99914f5fc0560f8cffd8c15b0f"}
Oct 30 13:40:20.754785 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.754764    4544 scope.go:122] "RemoveContainer" containerID="8919966ee1cfb5400f07f9d90a533886b4de9baf5d7c84e98271d101ffbc6ec0"
Oct 30 13:40:20.755119 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:40:20.755097    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8919966ee1cfb5400f07f9d90a533886b4de9baf5d7c84e98271d101ffbc6ec0\": container with ID starting with 8919966ee1cfb5400f07f9d90a533886b4de9baf5d7c84e98271d101ffbc6ec0 not found: ID does not exist" containerID="8919966ee1cfb5400f07f9d90a533886b4de9baf5d7c84e98271d101ffbc6ec0"
Oct 30 13:40:20.755172 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.755129    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8919966ee1cfb5400f07f9d90a533886b4de9baf5d7c84e98271d101ffbc6ec0"} err="failed to get container status \"8919966ee1cfb5400f07f9d90a533886b4de9baf5d7c84e98271d101ffbc6ec0\": rpc error: code = NotFound desc = could not find container \"8919966ee1cfb5400f07f9d90a533886b4de9baf5d7c84e98271d101ffbc6ec0\": container with ID starting with 8919966ee1cfb5400f07f9d90a533886b4de9baf5d7c84e98271d101ffbc6ec0 not found: ID does not exist"
Oct 30 13:40:20.798919 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.798862    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16e62535-gwl8v" podStartSLOduration=1.7988461629999999 podStartE2EDuration="1.798846163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:20.797673556 +0000 UTC m=+3315.844807509" watchObservedRunningTime="2026-10-30 13:40:20.798846163 +0000 UTC m=+3315.845980116"
Oct 30 13:40:20.839630 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.839576    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b45652a-nxdb5" podStartSLOduration=1.8395615589999998 podStartE2EDuration="1.839561559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:20.838690811 +0000 UTC m=+3315.885824764" watchObservedRunningTime="2026-10-30 13:40:20.839561559 +0000 UTC m=+3315.886695511"
Oct 30 13:40:20.879389 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.879272    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fe37db-b66zr" podStartSLOduration=1.879257167 podStartE2EDuration="1.879257167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:20.878255012 +0000 UTC m=+3315.925388963" watchObservedRunningTime="2026-10-30 13:40:20.879257167 +0000 UTC m=+3315.926391117"
Oct 30 13:40:20.918907 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.918606    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:40:20.918907 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.918873    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"]
Oct 30 13:40:20.922783 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:20.922738    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-c2ce75/premium-w2-c2ce75-kserve-c454bd5f9-vgxtv"]
Oct 30 13:40:21.095647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:21.095600    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a19094f-f1fc-4f65-9702-96dc3a0a2f77" path="/var/lib/kubelet/pods/3a19094f-f1fc-4f65-9702-96dc3a0a2f77/volumes"
Oct 30 13:40:21.096263 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:21.096236    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="404ea200-b989-4029-be47-f7075c3ed65e" path="/var/lib/kubelet/pods/404ea200-b989-4029-be47-f7075c3ed65e/volumes"
Oct 30 13:40:21.762744 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:21.762708    4544 generic.go:356] "Generic (PLEG): container finished" podID="9df75e04-9e0b-46b5-8a95-6b4be84e3314" containerID="969d03d92ff38fb0e817d29c5a6985961f59fd377dcdf3e4565568404f3dc2a8" exitCode=0
Oct 30 13:40:21.763178 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:21.762758    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b45652a-nxdb5" event={"ID":"9df75e04-9e0b-46b5-8a95-6b4be84e3314","Type":"ContainerDied","Data":"969d03d92ff38fb0e817d29c5a6985961f59fd377dcdf3e4565568404f3dc2a8"}
Oct 30 13:40:21.764748 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:21.764700    4544 generic.go:356] "Generic (PLEG): container finished" podID="84aeefc2-4217-4779-904d-e081cca145bc" containerID="544c5378f2f0780f40f13b42d38d6df4de0b2653e5e61e5b221ff8eedfe04c4b" exitCode=0
Oct 30 13:40:21.764909 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:21.764786    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16e62535-gwl8v" event={"ID":"84aeefc2-4217-4779-904d-e081cca145bc","Type":"ContainerDied","Data":"544c5378f2f0780f40f13b42d38d6df4de0b2653e5e61e5b221ff8eedfe04c4b"}
Oct 30 13:40:21.766541 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:21.766515    4544 generic.go:356] "Generic (PLEG): container finished" podID="19afa216-d512-4110-8641-bdfa5e42c7fd" containerID="119be7047d43cc5c4e3c5561db2e655faf273f826cf8b0afed664b22449e163d" exitCode=0
Oct 30 13:40:21.766634 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:21.766565    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fe37db-b66zr" event={"ID":"19afa216-d512-4110-8641-bdfa5e42c7fd","Type":"ContainerDied","Data":"119be7047d43cc5c4e3c5561db2e655faf273f826cf8b0afed664b22449e163d"}
Oct 30 13:40:22.170420 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:22.170385    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:22.182072 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:22.182031    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9" podUID="9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69" containerName="payload-processing" containerID="cri-o://904ec00d53d4370c85ee979df2ca10b3d90f92e5d6ea03c6d4b5b3a0369006b7" gracePeriod=25
Oct 30 13:40:22.250092 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:22.250034    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p" podUID="88adbb69-5dc6-4d77-96f4-5b6f61d80e8b" containerName="payload-pre-processing" containerID="cri-o://f28609be50de347a74439be7a1d643291e94f00937792c40b6551fb64848aaa3" gracePeriod=25
Oct 30 13:40:22.965582 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:22.965550    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16e62535-gwl8v"
Oct 30 13:40:22.995070 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:22.995033    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b45652a-nxdb5"
Oct 30 13:40:23.035387 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.035296    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fe37db-b66zr"
Oct 30 13:40:23.071239 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.071182    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9df75e04-9e0b-46b5-8a95-6b4be84e3314-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df75e04-9e0b-46b5-8a95-6b4be84e3314-maas-api-openshift-ca\") pod \"9df75e04-9e0b-46b5-8a95-6b4be84e3314\" (UID: \"9df75e04-9e0b-46b5-8a95-6b4be84e3314\") "
Oct 30 13:40:23.071466 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.071276    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84aeefc2-4217-4779-904d-e081cca145bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84aeefc2-4217-4779-904d-e081cca145bc-maas-api-openshift-ca\") pod \"84aeefc2-4217-4779-904d-e081cca145bc\" (UID: \"84aeefc2-4217-4779-904d-e081cca145bc\") "
Oct 30 13:40:23.071466 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.071372    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84aeefc2-4217-4779-904d-e081cca145bc-kube-api-access-2msgh\" (UniqueName: \"kubernetes.io/projected/84aeefc2-4217-4779-904d-e081cca145bc-kube-api-access-2msgh\") pod \"84aeefc2-4217-4779-904d-e081cca145bc\" (UID: \"84aeefc2-4217-4779-904d-e081cca145bc\") "
Oct 30 13:40:23.071466 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.071410    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19afa216-d512-4110-8641-bdfa5e42c7fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19afa216-d512-4110-8641-bdfa5e42c7fd-maas-api-openshift-ca\") pod \"19afa216-d512-4110-8641-bdfa5e42c7fd\" (UID: \"19afa216-d512-4110-8641-bdfa5e42c7fd\") "
Oct 30 13:40:23.071466 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.071450    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9df75e04-9e0b-46b5-8a95-6b4be84e3314-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df75e04-9e0b-46b5-8a95-6b4be84e3314-maas-api-xks-ca\") pod \"9df75e04-9e0b-46b5-8a95-6b4be84e3314\" (UID: \"9df75e04-9e0b-46b5-8a95-6b4be84e3314\") "
Oct 30 13:40:23.071710 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.071516    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9df75e04-9e0b-46b5-8a95-6b4be84e3314-kube-api-access-z4hqb\" (UniqueName: \"kubernetes.io/projected/9df75e04-9e0b-46b5-8a95-6b4be84e3314-kube-api-access-z4hqb\") pod \"9df75e04-9e0b-46b5-8a95-6b4be84e3314\" (UID: \"9df75e04-9e0b-46b5-8a95-6b4be84e3314\") "
Oct 30 13:40:23.071710 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.071567    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19afa216-d512-4110-8641-bdfa5e42c7fd-kube-api-access-xjdwk\" (UniqueName: \"kubernetes.io/projected/19afa216-d512-4110-8641-bdfa5e42c7fd-kube-api-access-xjdwk\") pod \"19afa216-d512-4110-8641-bdfa5e42c7fd\" (UID: \"19afa216-d512-4110-8641-bdfa5e42c7fd\") "
Oct 30 13:40:23.071710 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.071624    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9df75e04-9e0b-46b5-8a95-6b4be84e3314-maas-api-openshift-ca" pod "9df75e04-9e0b-46b5-8a95-6b4be84e3314" (UID: "9df75e04-9e0b-46b5-8a95-6b4be84e3314"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:23.071710 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.071656    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19afa216-d512-4110-8641-bdfa5e42c7fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19afa216-d512-4110-8641-bdfa5e42c7fd-maas-api-xks-ca\") pod \"19afa216-d512-4110-8641-bdfa5e42c7fd\" (UID: \"19afa216-d512-4110-8641-bdfa5e42c7fd\") "
Oct 30 13:40:23.071921 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.071771    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84aeefc2-4217-4779-904d-e081cca145bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84aeefc2-4217-4779-904d-e081cca145bc-maas-api-xks-ca\") pod \"84aeefc2-4217-4779-904d-e081cca145bc\" (UID: \"84aeefc2-4217-4779-904d-e081cca145bc\") "
Oct 30 13:40:23.071921 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.071875    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19afa216-d512-4110-8641-bdfa5e42c7fd-maas-api-openshift-ca" pod "19afa216-d512-4110-8641-bdfa5e42c7fd" (UID: "19afa216-d512-4110-8641-bdfa5e42c7fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:23.072332 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.072307    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df75e04-9e0b-46b5-8a95-6b4be84e3314-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:23.072470 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.072336    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19afa216-d512-4110-8641-bdfa5e42c7fd-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:23.072470 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.072375    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84aeefc2-4217-4779-904d-e081cca145bc-maas-api-openshift-ca" pod "84aeefc2-4217-4779-904d-e081cca145bc" (UID: "84aeefc2-4217-4779-904d-e081cca145bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:23.074600 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.074550    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19afa216-d512-4110-8641-bdfa5e42c7fd-kube-api-access-xjdwk" pod "19afa216-d512-4110-8641-bdfa5e42c7fd" (UID: "19afa216-d512-4110-8641-bdfa5e42c7fd"). InnerVolumeSpecName "kube-api-access-xjdwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:23.074741 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.074613    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9df75e04-9e0b-46b5-8a95-6b4be84e3314-maas-api-xks-ca" pod "9df75e04-9e0b-46b5-8a95-6b4be84e3314" (UID: "9df75e04-9e0b-46b5-8a95-6b4be84e3314"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:40:23.075033 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.075005    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84aeefc2-4217-4779-904d-e081cca145bc-maas-api-xks-ca" pod "84aeefc2-4217-4779-904d-e081cca145bc" (UID: "84aeefc2-4217-4779-904d-e081cca145bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:40:23.075771 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.075730    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19afa216-d512-4110-8641-bdfa5e42c7fd-maas-api-xks-ca" pod "19afa216-d512-4110-8641-bdfa5e42c7fd" (UID: "19afa216-d512-4110-8641-bdfa5e42c7fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:40:23.075999 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.075979    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df75e04-9e0b-46b5-8a95-6b4be84e3314-kube-api-access-z4hqb" pod "9df75e04-9e0b-46b5-8a95-6b4be84e3314" (UID: "9df75e04-9e0b-46b5-8a95-6b4be84e3314"). InnerVolumeSpecName "kube-api-access-z4hqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:23.076649 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.076633    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84aeefc2-4217-4779-904d-e081cca145bc-kube-api-access-2msgh" pod "84aeefc2-4217-4779-904d-e081cca145bc" (UID: "84aeefc2-4217-4779-904d-e081cca145bc"). InnerVolumeSpecName "kube-api-access-2msgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:23.173292 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.173207    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df75e04-9e0b-46b5-8a95-6b4be84e3314-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:23.173292 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.173291    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4hqb\" (UniqueName: \"kubernetes.io/projected/9df75e04-9e0b-46b5-8a95-6b4be84e3314-kube-api-access-z4hqb\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:23.173292 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.173303    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjdwk\" (UniqueName: \"kubernetes.io/projected/19afa216-d512-4110-8641-bdfa5e42c7fd-kube-api-access-xjdwk\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:23.173562 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.173313    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19afa216-d512-4110-8641-bdfa5e42c7fd-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:23.173562 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.173322    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84aeefc2-4217-4779-904d-e081cca145bc-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:23.173562 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.173330    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84aeefc2-4217-4779-904d-e081cca145bc-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:23.173562 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.173338    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2msgh\" (UniqueName: \"kubernetes.io/projected/84aeefc2-4217-4779-904d-e081cca145bc-kube-api-access-2msgh\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:23.782511 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.782475    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b45652a-nxdb5"
Oct 30 13:40:23.782511 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.782490    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b45652a-nxdb5" event={"ID":"9df75e04-9e0b-46b5-8a95-6b4be84e3314","Type":"ContainerDied","Data":"5aa765a7562ab7aaacd39adae173e6a01af9d0e3eefabc117eb2bfb5696716d7"}
Oct 30 13:40:23.782757 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.782545    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5aa765a7562ab7aaacd39adae173e6a01af9d0e3eefabc117eb2bfb5696716d7"
Oct 30 13:40:23.784402 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.784374    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16e62535-gwl8v"
Oct 30 13:40:23.784584 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.784450    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16e62535-gwl8v" event={"ID":"84aeefc2-4217-4779-904d-e081cca145bc","Type":"ContainerDied","Data":"4890ade682b6d6f5830f37a4f0119e41d807a19fe0e5f64c877777c3de0a5aed"}
Oct 30 13:40:23.784584 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.784499    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4890ade682b6d6f5830f37a4f0119e41d807a19fe0e5f64c877777c3de0a5aed"
Oct 30 13:40:23.786073 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.786051    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fe37db-b66zr"
Oct 30 13:40:23.786200 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.786096    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fe37db-b66zr" event={"ID":"19afa216-d512-4110-8641-bdfa5e42c7fd","Type":"ContainerDied","Data":"96bff2d8020a51199d508579debd39a9318bda99914f5fc0560f8cffd8c15b0f"}
Oct 30 13:40:23.786200 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.786145    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96bff2d8020a51199d508579debd39a9318bda99914f5fc0560f8cffd8c15b0f"
Oct 30 13:40:23.791281 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.791241    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main" containerID="cri-o://b5c593ff77e5a4795dd276ffcb5056cd8706ff649fe625f0a070735eae9004b7" gracePeriod=45
Oct 30 13:40:23.969630 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:23.969593    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:24.099430 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:24.099382    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:40:24.185921 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:24.185878    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-home\" (UniqueName: \"kubernetes.io/empty-dir/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-home\") pod \"0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a\" (UID: \"0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a\") "
Oct 30 13:40:24.186082 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:24.186018    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-dshm\") pod \"0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a\" (UID: \"0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a\") "
Oct 30 13:40:24.186082 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:24.186056    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-model-cache\") pod \"0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a\" (UID: \"0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a\") "
Oct 30 13:40:24.186188 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:24.186165    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-tmp-dir\") pod \"0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a\" (UID: \"0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a\") "
Oct 30 13:40:24.186252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:24.186193    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-kube-api-access-67cc7\" (UniqueName: \"kubernetes.io/projected/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-kube-api-access-67cc7\") pod \"0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a\" (UID: \"0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a\") "
Oct 30 13:40:24.186310 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:24.186242    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-home" pod "0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" (UID: "0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:24.186310 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:24.186287    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-tls-certs\" (UniqueName: \"kubernetes.io/secret/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-tls-certs\") pod \"0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a\" (UID: \"0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a\") "
Oct 30 13:40:24.186502 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:24.186477    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-model-cache" pod "0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" (UID: "0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:24.186818 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:24.186790    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-tmp-dir" pod "0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" (UID: "0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:24.186925 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:24.186840    4544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-home\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:24.186925 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:24.186859    4544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-model-cache\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:24.189126 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:24.189092    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-dshm" pod "0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" (UID: "0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:24.189293 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:24.189136    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-tls-certs" pod "0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" (UID: "0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:40:24.190379 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:24.190351    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-kube-api-access-67cc7" pod "0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" (UID: "0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a"). InnerVolumeSpecName "kube-api-access-67cc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:24.288457 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:24.288419    4544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-dshm\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:24.288457 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:24.288449    4544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-tmp-dir\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:24.288457 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:24.288458    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67cc7\" (UniqueName: \"kubernetes.io/projected/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-kube-api-access-67cc7\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:24.288457 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:24.288469    4544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a-tls-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:24.695154 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:24.695118    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6"
Oct 30 13:40:24.797277 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:24.797244    4544 generic.go:356] "Generic (PLEG): container finished" podID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerID="b5c593ff77e5a4795dd276ffcb5056cd8706ff649fe625f0a070735eae9004b7" exitCode=0
Oct 30 13:40:24.797444 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:24.797307    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" event={"ID":"0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a","Type":"ContainerDied","Data":"b5c593ff77e5a4795dd276ffcb5056cd8706ff649fe625f0a070735eae9004b7"}
Oct 30 13:40:24.797444 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:24.797311    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"
Oct 30 13:40:24.797444 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:24.797341    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj" event={"ID":"0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a","Type":"ContainerDied","Data":"c7450ad6f9b41fbeff83d41687d9a75478de24eba17476ab0271c2b2d0e7f908"}
Oct 30 13:40:24.797444 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:24.797361    4544 scope.go:122] "RemoveContainer" containerID="b5c593ff77e5a4795dd276ffcb5056cd8706ff649fe625f0a070735eae9004b7"
Oct 30 13:40:24.812967 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:24.812802    4544 scope.go:122] "RemoveContainer" containerID="b5c593ff77e5a4795dd276ffcb5056cd8706ff649fe625f0a070735eae9004b7"
Oct 30 13:40:24.813798 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:40:24.813729    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5c593ff77e5a4795dd276ffcb5056cd8706ff649fe625f0a070735eae9004b7\": container with ID starting with b5c593ff77e5a4795dd276ffcb5056cd8706ff649fe625f0a070735eae9004b7 not found: ID does not exist" containerID="b5c593ff77e5a4795dd276ffcb5056cd8706ff649fe625f0a070735eae9004b7"
Oct 30 13:40:24.813798 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:24.813781    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5c593ff77e5a4795dd276ffcb5056cd8706ff649fe625f0a070735eae9004b7"} err="failed to get container status \"b5c593ff77e5a4795dd276ffcb5056cd8706ff649fe625f0a070735eae9004b7\": rpc error: code = NotFound desc = could not find container \"b5c593ff77e5a4795dd276ffcb5056cd8706ff649fe625f0a070735eae9004b7\": container with ID starting with b5c593ff77e5a4795dd276ffcb5056cd8706ff649fe625f0a070735eae9004b7 not found: ID does not exist"
Oct 30 13:40:24.862293 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:24.862243    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"]
Oct 30 13:40:24.882581 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:24.882537    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-8b46c9/distinct-w1-8b46c9-kserve-7c54fd4ddd-v7qzj"]
Oct 30 13:40:25.093427 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:25.090877    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" path="/var/lib/kubelet/pods/0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a/volumes"
Oct 30 13:40:25.747861 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:25.747820    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc"]
Oct 30 13:40:25.748577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:25.748555    4544 cpu_manager.go:414] "containerMap: removing container" podUID="84aeefc2-4217-4779-904d-e081cca145bc" containerName="revoke-keys"
Oct 30 13:40:25.748577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:25.748578    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84aeefc2-4217-4779-904d-e081cca145bc" containerName="revoke-keys"
Oct 30 13:40:25.748682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:25.748590    4544 cpu_manager.go:414] "containerMap: removing container" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main"
Oct 30 13:40:25.748682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:25.748595    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main"
Oct 30 13:40:25.748682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:25.748605    4544 cpu_manager.go:414] "containerMap: removing container" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main"
Oct 30 13:40:25.748682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:25.748611    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main"
Oct 30 13:40:25.748682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:25.748633    4544 cpu_manager.go:414] "containerMap: removing container" podUID="19afa216-d512-4110-8641-bdfa5e42c7fd" containerName="revoke-keys"
Oct 30 13:40:25.748682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:25.748637    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19afa216-d512-4110-8641-bdfa5e42c7fd" containerName="revoke-keys"
Oct 30 13:40:25.748682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:25.748644    4544 cpu_manager.go:414] "containerMap: removing container" podUID="9df75e04-9e0b-46b5-8a95-6b4be84e3314" containerName="revoke-keys"
Oct 30 13:40:25.748682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:25.748648    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df75e04-9e0b-46b5-8a95-6b4be84e3314" containerName="revoke-keys"
Oct 30 13:40:25.748942 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:25.748734    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="19afa216-d512-4110-8641-bdfa5e42c7fd" containerName="revoke-keys"
Oct 30 13:40:25.748942 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:25.748741    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df75e04-9e0b-46b5-8a95-6b4be84e3314" containerName="revoke-keys"
Oct 30 13:40:25.748942 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:25.748752    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="84aeefc2-4217-4779-904d-e081cca145bc" containerName="revoke-keys"
Oct 30 13:40:25.748942 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:25.748760    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d6204f2-be22-4e1a-8aa6-8cf7aab8a78a" containerName="main"
Oct 30 13:40:25.748942 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:25.748766    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="404ea200-b989-4029-be47-f7075c3ed65e" containerName="main"
Oct 30 13:40:25.753874 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:25.753844    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc"
Oct 30 13:40:25.769981 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:25.769950    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-87e742-serving-cert\""
Oct 30 13:40:25.788854 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:25.788811    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc"]
Oct 30 13:40:25.810858 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:25.810743    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5e6bb1-c800-4a0a-a054-0f741c3d8933-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc\" (UID: \"9e5e6bb1-c800-4a0a-a054-0f741c3d8933\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc"
Oct 30 13:40:25.811071 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:25.810927    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e5e6bb1-c800-4a0a-a054-0f741c3d8933-metrics-certs\") pod \"maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc\" (UID: \"9e5e6bb1-c800-4a0a-a054-0f741c3d8933\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc"
Oct 30 13:40:25.811160 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:25.811139    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc88m\" (UniqueName: \"kubernetes.io/projected/9e5e6bb1-c800-4a0a-a054-0f741c3d8933-kube-api-access-gc88m\") pod \"maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc\" (UID: \"9e5e6bb1-c800-4a0a-a054-0f741c3d8933\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc"
Oct 30 13:40:25.811284 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:25.811267    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e5e6bb1-c800-4a0a-a054-0f741c3d8933-maas-api-tls\") pod \"maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc\" (UID: \"9e5e6bb1-c800-4a0a-a054-0f741c3d8933\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc"
Oct 30 13:40:25.854936 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:25.854896    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:25.912961 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:25.912921    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e5e6bb1-c800-4a0a-a054-0f741c3d8933-maas-api-tls\") pod \"maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc\" (UID: \"9e5e6bb1-c800-4a0a-a054-0f741c3d8933\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc"
Oct 30 13:40:25.913116 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:25.913005    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5e6bb1-c800-4a0a-a054-0f741c3d8933-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc\" (UID: \"9e5e6bb1-c800-4a0a-a054-0f741c3d8933\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc"
Oct 30 13:40:25.913116 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:25.913035    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e5e6bb1-c800-4a0a-a054-0f741c3d8933-metrics-certs\") pod \"maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc\" (UID: \"9e5e6bb1-c800-4a0a-a054-0f741c3d8933\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc"
Oct 30 13:40:25.913249 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:40:25.913103    4544 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-87e742-serving-cert: secret "maas-api-e2e-ipp-b-87e742-serving-cert" not found
Oct 30 13:40:25.913249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:25.913138    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc88m\" (UniqueName: \"kubernetes.io/projected/9e5e6bb1-c800-4a0a-a054-0f741c3d8933-kube-api-access-gc88m\") pod \"maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc\" (UID: \"9e5e6bb1-c800-4a0a-a054-0f741c3d8933\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc"
Oct 30 13:40:25.913249 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:40:25.913184    4544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e5e6bb1-c800-4a0a-a054-0f741c3d8933-maas-api-tls podName:9e5e6bb1-c800-4a0a-a054-0f741c3d8933 nodeName:}" failed. No retries permitted until 2026-10-30 13:40:26.413161468 +0000 UTC m=+3321.460295421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9e5e6bb1-c800-4a0a-a054-0f741c3d8933-maas-api-tls") pod "maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc" (UID: "9e5e6bb1-c800-4a0a-a054-0f741c3d8933") : secret "maas-api-e2e-ipp-b-87e742-serving-cert" not found
Oct 30 13:40:25.913860 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:25.913830    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5e6bb1-c800-4a0a-a054-0f741c3d8933-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc\" (UID: \"9e5e6bb1-c800-4a0a-a054-0f741c3d8933\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc"
Oct 30 13:40:25.915864 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:25.915837    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e5e6bb1-c800-4a0a-a054-0f741c3d8933-metrics-certs\") pod \"maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc\" (UID: \"9e5e6bb1-c800-4a0a-a054-0f741c3d8933\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc"
Oct 30 13:40:25.938478 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:25.938435    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc88m\" (UniqueName: \"kubernetes.io/projected/9e5e6bb1-c800-4a0a-a054-0f741c3d8933-kube-api-access-gc88m\") pod \"maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc\" (UID: \"9e5e6bb1-c800-4a0a-a054-0f741c3d8933\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc"
Oct 30 13:40:26.007401 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:25.990884    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:26.158839 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:26.158777    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:26.426312 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:26.426213    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e5e6bb1-c800-4a0a-a054-0f741c3d8933-maas-api-tls\") pod \"maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc\" (UID: \"9e5e6bb1-c800-4a0a-a054-0f741c3d8933\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc"
Oct 30 13:40:26.428878 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:26.428844    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e5e6bb1-c800-4a0a-a054-0f741c3d8933-maas-api-tls\") pod \"maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc\" (UID: \"9e5e6bb1-c800-4a0a-a054-0f741c3d8933\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc"
Oct 30 13:40:26.502572 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:26.502527    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main" containerID="cri-o://29e718e88749e1646a66da0a7a1865e8ada6d73917ba134a98ae4d2b242a797a" gracePeriod=45
Oct 30 13:40:26.670315 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:26.668974    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc"
Oct 30 13:40:26.824103 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:26.823904    4544 generic.go:356] "Generic (PLEG): container finished" podID="b66ef925-6b4b-4859-a706-960515034d64" containerID="29e718e88749e1646a66da0a7a1865e8ada6d73917ba134a98ae4d2b242a797a" exitCode=0
Oct 30 13:40:26.827700 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:26.824155    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" event={"ID":"b66ef925-6b4b-4859-a706-960515034d64","Type":"ContainerDied","Data":"29e718e88749e1646a66da0a7a1865e8ada6d73917ba134a98ae4d2b242a797a"}
Oct 30 13:40:26.835197 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:26.835166    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:40:26.851861 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:26.851814    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:26.860885 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:26.860836    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9" podUID="7692a16a-4d89-4f37-85ae-6c43dad93d35" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:40:26.962364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:26.962258    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b66ef925-6b4b-4859-a706-960515034d64-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b66ef925-6b4b-4859-a706-960515034d64-dshm\") pod \"b66ef925-6b4b-4859-a706-960515034d64\" (UID: \"b66ef925-6b4b-4859-a706-960515034d64\") "
Oct 30 13:40:26.962549 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:26.962442    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b66ef925-6b4b-4859-a706-960515034d64-tls-certs\" (UniqueName: \"kubernetes.io/secret/b66ef925-6b4b-4859-a706-960515034d64-tls-certs\") pod \"b66ef925-6b4b-4859-a706-960515034d64\" (UID: \"b66ef925-6b4b-4859-a706-960515034d64\") "
Oct 30 13:40:26.962549 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:26.962531    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b66ef925-6b4b-4859-a706-960515034d64-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b66ef925-6b4b-4859-a706-960515034d64-tmp-dir\") pod \"b66ef925-6b4b-4859-a706-960515034d64\" (UID: \"b66ef925-6b4b-4859-a706-960515034d64\") "
Oct 30 13:40:26.962677 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:26.962562    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b66ef925-6b4b-4859-a706-960515034d64-home\" (UniqueName: \"kubernetes.io/empty-dir/b66ef925-6b4b-4859-a706-960515034d64-home\") pod \"b66ef925-6b4b-4859-a706-960515034d64\" (UID: \"b66ef925-6b4b-4859-a706-960515034d64\") "
Oct 30 13:40:26.962677 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:26.962619    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b66ef925-6b4b-4859-a706-960515034d64-kube-api-access-24nvl\" (UniqueName: \"kubernetes.io/projected/b66ef925-6b4b-4859-a706-960515034d64-kube-api-access-24nvl\") pod \"b66ef925-6b4b-4859-a706-960515034d64\" (UID: \"b66ef925-6b4b-4859-a706-960515034d64\") "
Oct 30 13:40:26.962796 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:26.962681    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b66ef925-6b4b-4859-a706-960515034d64-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b66ef925-6b4b-4859-a706-960515034d64-model-cache\") pod \"b66ef925-6b4b-4859-a706-960515034d64\" (UID: \"b66ef925-6b4b-4859-a706-960515034d64\") "
Oct 30 13:40:26.962883 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:26.962844    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b66ef925-6b4b-4859-a706-960515034d64-tmp-dir" pod "b66ef925-6b4b-4859-a706-960515034d64" (UID: "b66ef925-6b4b-4859-a706-960515034d64"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:26.963189 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:26.963158    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b66ef925-6b4b-4859-a706-960515034d64-home" pod "b66ef925-6b4b-4859-a706-960515034d64" (UID: "b66ef925-6b4b-4859-a706-960515034d64"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:26.963318 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:26.963169    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b66ef925-6b4b-4859-a706-960515034d64-model-cache" pod "b66ef925-6b4b-4859-a706-960515034d64" (UID: "b66ef925-6b4b-4859-a706-960515034d64"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:26.963318 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:26.963194    4544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b66ef925-6b4b-4859-a706-960515034d64-tmp-dir\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:26.965118 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:26.965089    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b66ef925-6b4b-4859-a706-960515034d64-tls-certs" pod "b66ef925-6b4b-4859-a706-960515034d64" (UID: "b66ef925-6b4b-4859-a706-960515034d64"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:40:26.965662 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:26.965629    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66ef925-6b4b-4859-a706-960515034d64-kube-api-access-24nvl" pod "b66ef925-6b4b-4859-a706-960515034d64" (UID: "b66ef925-6b4b-4859-a706-960515034d64"). InnerVolumeSpecName "kube-api-access-24nvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:26.965745 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:26.965711    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b66ef925-6b4b-4859-a706-960515034d64-dshm" pod "b66ef925-6b4b-4859-a706-960515034d64" (UID: "b66ef925-6b4b-4859-a706-960515034d64"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:27.064832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:27.064784    4544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b66ef925-6b4b-4859-a706-960515034d64-model-cache\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:27.064832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:27.064818    4544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b66ef925-6b4b-4859-a706-960515034d64-dshm\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:27.064832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:27.064827    4544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b66ef925-6b4b-4859-a706-960515034d64-tls-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:27.064832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:27.064835    4544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b66ef925-6b4b-4859-a706-960515034d64-home\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:27.064832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:27.064844    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24nvl\" (UniqueName: \"kubernetes.io/projected/b66ef925-6b4b-4859-a706-960515034d64-kube-api-access-24nvl\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:27.090598 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:27.090563    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc"]
Oct 30 13:40:27.093037 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:40:27.092999    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e5e6bb1_c800_4a0a_a054_0f741c3d8933.slice/crio-ac40eacd4dd95a9fe3fe2cd0394c1a629c966d78a6efdf5fd75ab51127e55f87 WatchSource:0}: Error finding container ac40eacd4dd95a9fe3fe2cd0394c1a629c966d78a6efdf5fd75ab51127e55f87: Status 404 returned error can't find the container with id ac40eacd4dd95a9fe3fe2cd0394c1a629c966d78a6efdf5fd75ab51127e55f87
Oct 30 13:40:27.100506 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:27.100474    4544 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:40:27.317662 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:27.315312    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:27.728700 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:27.728626    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:27.835118 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:27.835081    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"
Oct 30 13:40:27.835350 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:27.835114    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz" event={"ID":"b66ef925-6b4b-4859-a706-960515034d64","Type":"ContainerDied","Data":"6a3f0772ca3f206601178af4bb13856635a192231ea2c2d14b5ba222c640511b"}
Oct 30 13:40:27.835350 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:27.835175    4544 scope.go:122] "RemoveContainer" containerID="29e718e88749e1646a66da0a7a1865e8ada6d73917ba134a98ae4d2b242a797a"
Oct 30 13:40:27.837379 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:27.837334    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc" event={"ID":"9e5e6bb1-c800-4a0a-a054-0f741c3d8933","Type":"ContainerStarted","Data":"2ce2f527b8da9d0621296f3a0472f2ed3b644025551f95dddf5640f0a78df227"}
Oct 30 13:40:27.837510 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:27.837413    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc" event={"ID":"9e5e6bb1-c800-4a0a-a054-0f741c3d8933","Type":"ContainerStarted","Data":"ac40eacd4dd95a9fe3fe2cd0394c1a629c966d78a6efdf5fd75ab51127e55f87"}
Oct 30 13:40:27.837510 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:27.837447    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc"
Oct 30 13:40:27.891993 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:27.891916    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"]
Oct 30 13:40:27.898564 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:27.898186    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j" podUID="cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:40:27.898768 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:27.898591    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c90b9e/distinct-2-w0-c90b9e-kserve-6c444b5c74-nsqnz"]
Oct 30 13:40:27.933281 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:27.933193    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc" podStartSLOduration=2.489942355 podStartE2EDuration="2.933175958s" totalImagesPullingTime="443.233603ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:27.93265242 +0000 UTC m=+3322.979786373" watchObservedRunningTime="2026-10-30 13:40:27.933175958 +0000 UTC m=+3322.980309910"
Oct 30 13:40:28.300269 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:28.300160    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:28.423268 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:28.423198    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:28.462249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:28.459688    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0e874e-xwghg"]
Oct 30 13:40:28.468326 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:28.464828    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0e874e-xwghg"]
Oct 30 13:40:29.008273 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:40:29.008230    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:40:29.080784 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:29.080746    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:29.087967 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:29.087927    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a18b9841-be8b-49c2-9676-87f53f621855" path="/var/lib/kubelet/pods/a18b9841-be8b-49c2-9676-87f53f621855/volumes"
Oct 30 13:40:29.088710 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:29.088684    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b66ef925-6b4b-4859-a706-960515034d64" path="/var/lib/kubelet/pods/b66ef925-6b4b-4859-a706-960515034d64/volumes"
Oct 30 13:40:29.597086 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:29.597044    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:29.615829 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:29.609091    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5" podUID="8d82c543-383d-4498-8d3c-d6c2a0716736" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:40:29.615829 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:29.609262    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw" podUID="db2e31d3-d423-4671-8449-56d1f380fbd1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:40:29.741471 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:29.741431    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:29.946598 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:29.946509    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:30.097185 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:30.097148    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:30.229814 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:30.229706    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:30.407927 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:30.407890    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:30.556155 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:30.556073    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:30.614013 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:30.613973    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9" podUID="9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:40:30.614013 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:30.613999    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p" podUID="88adbb69-5dc6-4d77-96f4-5b6f61d80e8b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:40:30.701272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:30.701236    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:30.861668 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:30.861637    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:32.799450 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:32.799415    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:33.854311 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:33.854277    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc"
Oct 30 13:40:35.357533 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:35.357498    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw"
Oct 30 13:40:35.465650 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:35.465175    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5"
Oct 30 13:40:35.479294 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:35.478264    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db2e31d3-d423-4671-8449-56d1f380fbd1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db2e31d3-d423-4671-8449-56d1f380fbd1-plugins-config-volume\") pod \"db2e31d3-d423-4671-8449-56d1f380fbd1\" (UID: \"db2e31d3-d423-4671-8449-56d1f380fbd1\") "
Oct 30 13:40:35.479294 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:35.478323    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db2e31d3-d423-4671-8449-56d1f380fbd1-kube-api-access-68hfg\" (UniqueName: \"kubernetes.io/projected/db2e31d3-d423-4671-8449-56d1f380fbd1-kube-api-access-68hfg\") pod \"db2e31d3-d423-4671-8449-56d1f380fbd1\" (UID: \"db2e31d3-d423-4671-8449-56d1f380fbd1\") "
Oct 30 13:40:35.481153 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:35.481106    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db2e31d3-d423-4671-8449-56d1f380fbd1-plugins-config-volume" pod "db2e31d3-d423-4671-8449-56d1f380fbd1" (UID: "db2e31d3-d423-4671-8449-56d1f380fbd1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:35.481763 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:35.481711    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db2e31d3-d423-4671-8449-56d1f380fbd1-kube-api-access-68hfg" pod "db2e31d3-d423-4671-8449-56d1f380fbd1" (UID: "db2e31d3-d423-4671-8449-56d1f380fbd1"). InnerVolumeSpecName "kube-api-access-68hfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:35.577500 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:35.577441    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:35.579625 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:35.579593    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d82c543-383d-4498-8d3c-d6c2a0716736-kube-api-access-lm26z\" (UniqueName: \"kubernetes.io/projected/8d82c543-383d-4498-8d3c-d6c2a0716736-kube-api-access-lm26z\") pod \"8d82c543-383d-4498-8d3c-d6c2a0716736\" (UID: \"8d82c543-383d-4498-8d3c-d6c2a0716736\") "
Oct 30 13:40:35.579814 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:35.579792    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d82c543-383d-4498-8d3c-d6c2a0716736-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d82c543-383d-4498-8d3c-d6c2a0716736-plugins-config-volume\") pod \"8d82c543-383d-4498-8d3c-d6c2a0716736\" (UID: \"8d82c543-383d-4498-8d3c-d6c2a0716736\") "
Oct 30 13:40:35.580150 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:35.580127    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d82c543-383d-4498-8d3c-d6c2a0716736-plugins-config-volume" pod "8d82c543-383d-4498-8d3c-d6c2a0716736" (UID: "8d82c543-383d-4498-8d3c-d6c2a0716736"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:35.580406 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:35.580390    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db2e31d3-d423-4671-8449-56d1f380fbd1-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:35.580460 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:35.580407    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68hfg\" (UniqueName: \"kubernetes.io/projected/db2e31d3-d423-4671-8449-56d1f380fbd1-kube-api-access-68hfg\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:35.580460 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:35.580416    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d82c543-383d-4498-8d3c-d6c2a0716736-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:35.581956 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:35.581935    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d82c543-383d-4498-8d3c-d6c2a0716736-kube-api-access-lm26z" pod "8d82c543-383d-4498-8d3c-d6c2a0716736" (UID: "8d82c543-383d-4498-8d3c-d6c2a0716736"). InnerVolumeSpecName "kube-api-access-lm26z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:35.681995 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:35.681954    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm26z\" (UniqueName: \"kubernetes.io/projected/8d82c543-383d-4498-8d3c-d6c2a0716736-kube-api-access-lm26z\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:35.900153 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:35.900111    4544 generic.go:356] "Generic (PLEG): container finished" podID="8d82c543-383d-4498-8d3c-d6c2a0716736" containerID="564b7c69980224c672c885ff533d40a4f70c9a6e43a071491ae939a39f83cfd6" exitCode=0
Oct 30 13:40:35.900153 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:35.900158    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5"
Oct 30 13:40:35.900470 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:35.900199    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5" event={"ID":"8d82c543-383d-4498-8d3c-d6c2a0716736","Type":"ContainerDied","Data":"564b7c69980224c672c885ff533d40a4f70c9a6e43a071491ae939a39f83cfd6"}
Oct 30 13:40:35.900470 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:35.900278    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5" event={"ID":"8d82c543-383d-4498-8d3c-d6c2a0716736","Type":"ContainerDied","Data":"89b304f72a336d720b46b05e6ac23a7f37705b862dda335d0adaf7169abb395a"}
Oct 30 13:40:35.900470 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:35.900308    4544 scope.go:122] "RemoveContainer" containerID="564b7c69980224c672c885ff533d40a4f70c9a6e43a071491ae939a39f83cfd6"
Oct 30 13:40:35.915100 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:35.915045    4544 generic.go:356] "Generic (PLEG): container finished" podID="db2e31d3-d423-4671-8449-56d1f380fbd1" containerID="22d31b784325aa478dba1c843a61e3dc2db7964039fc1a4a989dc90f8acd5862" exitCode=0
Oct 30 13:40:35.915349 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:35.915104    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw" event={"ID":"db2e31d3-d423-4671-8449-56d1f380fbd1","Type":"ContainerDied","Data":"22d31b784325aa478dba1c843a61e3dc2db7964039fc1a4a989dc90f8acd5862"}
Oct 30 13:40:35.915349 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:35.915202    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw" event={"ID":"db2e31d3-d423-4671-8449-56d1f380fbd1","Type":"ContainerDied","Data":"bc028ccb28e9553ea0c95dcab0cab3882a8a68c4d59c923d568cd7a8c304025b"}
Oct 30 13:40:35.915482 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:35.915361    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw"
Oct 30 13:40:35.925073 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:35.925048    4544 scope.go:122] "RemoveContainer" containerID="564b7c69980224c672c885ff533d40a4f70c9a6e43a071491ae939a39f83cfd6"
Oct 30 13:40:35.925460 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:40:35.925430    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"564b7c69980224c672c885ff533d40a4f70c9a6e43a071491ae939a39f83cfd6\": container with ID starting with 564b7c69980224c672c885ff533d40a4f70c9a6e43a071491ae939a39f83cfd6 not found: ID does not exist" containerID="564b7c69980224c672c885ff533d40a4f70c9a6e43a071491ae939a39f83cfd6"
Oct 30 13:40:35.925597 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:35.925566    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"564b7c69980224c672c885ff533d40a4f70c9a6e43a071491ae939a39f83cfd6"} err="failed to get container status \"564b7c69980224c672c885ff533d40a4f70c9a6e43a071491ae939a39f83cfd6\": rpc error: code = NotFound desc = could not find container \"564b7c69980224c672c885ff533d40a4f70c9a6e43a071491ae939a39f83cfd6\": container with ID starting with 564b7c69980224c672c885ff533d40a4f70c9a6e43a071491ae939a39f83cfd6 not found: ID does not exist"
Oct 30 13:40:35.925597 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:35.925598    4544 scope.go:122] "RemoveContainer" containerID="22d31b784325aa478dba1c843a61e3dc2db7964039fc1a4a989dc90f8acd5862"
Oct 30 13:40:35.936671 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:35.936636    4544 scope.go:122] "RemoveContainer" containerID="22d31b784325aa478dba1c843a61e3dc2db7964039fc1a4a989dc90f8acd5862"
Oct 30 13:40:35.936995 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:40:35.936977    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22d31b784325aa478dba1c843a61e3dc2db7964039fc1a4a989dc90f8acd5862\": container with ID starting with 22d31b784325aa478dba1c843a61e3dc2db7964039fc1a4a989dc90f8acd5862 not found: ID does not exist" containerID="22d31b784325aa478dba1c843a61e3dc2db7964039fc1a4a989dc90f8acd5862"
Oct 30 13:40:35.937048 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:35.937005    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22d31b784325aa478dba1c843a61e3dc2db7964039fc1a4a989dc90f8acd5862"} err="failed to get container status \"22d31b784325aa478dba1c843a61e3dc2db7964039fc1a4a989dc90f8acd5862\": rpc error: code = NotFound desc = could not find container \"22d31b784325aa478dba1c843a61e3dc2db7964039fc1a4a989dc90f8acd5862\": container with ID starting with 22d31b784325aa478dba1c843a61e3dc2db7964039fc1a4a989dc90f8acd5862 not found: ID does not exist"
Oct 30 13:40:35.979962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:35.979927    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw"]
Oct 30 13:40:35.991050 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:35.991013    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw"]
Oct 30 13:40:36.017604 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:36.017571    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5"]
Oct 30 13:40:36.018160 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:36.018128    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:36.023850 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:36.023819    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5"]
Oct 30 13:40:36.163356 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:36.163211    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:36.293572 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:36.293536    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:36.509167 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:36.509069    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"]
Oct 30 13:40:36.509652 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:36.509373    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h" podUID="565c6710-ac33-437a-b2cc-5572a325523d" containerName="istio-proxy" containerID="cri-o://3cce96af41f9471fb3b52d776887166145d78a3b078c98cc1ca160c0c9ce9498" gracePeriod=30
Oct 30 13:40:36.637139 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:36.637103    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:36.861856 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:36.861812    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9" podUID="7692a16a-4d89-4f37-85ae-6c43dad93d35" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:40:37.086857 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:37.086819    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d82c543-383d-4498-8d3c-d6c2a0716736" path="/var/lib/kubelet/pods/8d82c543-383d-4498-8d3c-d6c2a0716736/volumes"
Oct 30 13:40:37.087338 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:37.087322    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db2e31d3-d423-4671-8449-56d1f380fbd1" path="/var/lib/kubelet/pods/db2e31d3-d423-4671-8449-56d1f380fbd1/volumes"
Oct 30 13:40:37.156630 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:37.156531    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:37.641523 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:37.641480    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:37.897983 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:37.897883    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j" podUID="cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:40:38.181017 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:38.180920    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:38.299760 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:38.299719    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:38.418901 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:38.418789    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:38.455310 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:38.455212    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c2ce75-qwbbm"]
Oct 30 13:40:38.466870 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:38.466832    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c2ce75-qwbbm"]
Oct 30 13:40:38.758954 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:38.758859    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:38.891744 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:38.891710    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:39.010162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:39.010075    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:39.087565 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:39.087522    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68a65539-9527-425b-a39c-d1d536e41df1" path="/var/lib/kubelet/pods/68a65539-9527-425b-a39c-d1d536e41df1/volumes"
Oct 30 13:40:39.154867 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:39.154819    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:39.159250 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:39.159199    4544 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:40:39.159439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:39.159278    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h" podUID="565c6710-ac33-437a-b2cc-5572a325523d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:40:39.440247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:39.440163    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:39.854807 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:39.854778    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9"
Oct 30 13:40:39.866556 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:39.866529    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7692a16a-4d89-4f37-85ae-6c43dad93d35-kube-api-access-tqhst\" (UniqueName: \"kubernetes.io/projected/7692a16a-4d89-4f37-85ae-6c43dad93d35-kube-api-access-tqhst\") pod \"7692a16a-4d89-4f37-85ae-6c43dad93d35\" (UID: \"7692a16a-4d89-4f37-85ae-6c43dad93d35\") "
Oct 30 13:40:39.866714 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:39.866658    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7692a16a-4d89-4f37-85ae-6c43dad93d35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7692a16a-4d89-4f37-85ae-6c43dad93d35-plugins-config-volume\") pod \"7692a16a-4d89-4f37-85ae-6c43dad93d35\" (UID: \"7692a16a-4d89-4f37-85ae-6c43dad93d35\") "
Oct 30 13:40:39.867308 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:39.867053    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7692a16a-4d89-4f37-85ae-6c43dad93d35-plugins-config-volume" pod "7692a16a-4d89-4f37-85ae-6c43dad93d35" (UID: "7692a16a-4d89-4f37-85ae-6c43dad93d35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:39.875496 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:39.872933    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7692a16a-4d89-4f37-85ae-6c43dad93d35-kube-api-access-tqhst" pod "7692a16a-4d89-4f37-85ae-6c43dad93d35" (UID: "7692a16a-4d89-4f37-85ae-6c43dad93d35"). InnerVolumeSpecName "kube-api-access-tqhst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:39.886720 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:39.886690    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j"
Oct 30 13:40:39.967644 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:39.967550    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb-plugins-config-volume\") pod \"cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb\" (UID: \"cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb\") "
Oct 30 13:40:39.967830 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:39.967725    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb-kube-api-access-8g7g6\" (UniqueName: \"kubernetes.io/projected/cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb-kube-api-access-8g7g6\") pod \"cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb\" (UID: \"cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb\") "
Oct 30 13:40:39.968117 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:39.968091    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7692a16a-4d89-4f37-85ae-6c43dad93d35-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:39.968243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:39.968121    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqhst\" (UniqueName: \"kubernetes.io/projected/7692a16a-4d89-4f37-85ae-6c43dad93d35-kube-api-access-tqhst\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:39.968363 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:39.968331    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb-plugins-config-volume" pod "cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb" (UID: "cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:39.970468 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:39.970438    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb-kube-api-access-8g7g6" pod "cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb" (UID: "cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb"). InnerVolumeSpecName "kube-api-access-8g7g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:39.984720 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:39.984685    4544 generic.go:356] "Generic (PLEG): container finished" podID="cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb" containerID="abc3b345cdc771fc118889cfa8168a951045cee88b541355d43b90069fc23793" exitCode=0
Oct 30 13:40:39.984869 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:39.984733    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j"
Oct 30 13:40:39.984869 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:39.984765    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j" event={"ID":"cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb","Type":"ContainerDied","Data":"abc3b345cdc771fc118889cfa8168a951045cee88b541355d43b90069fc23793"}
Oct 30 13:40:39.984869 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:39.984855    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j" event={"ID":"cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb","Type":"ContainerDied","Data":"6b70c1f5251089853c3a4158694e28abc66219cf5ea7ae909fb7e527f1e6118b"}
Oct 30 13:40:39.985028 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:39.984881    4544 scope.go:122] "RemoveContainer" containerID="abc3b345cdc771fc118889cfa8168a951045cee88b541355d43b90069fc23793"
Oct 30 13:40:39.986858 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:39.986831    4544 generic.go:356] "Generic (PLEG): container finished" podID="7692a16a-4d89-4f37-85ae-6c43dad93d35" containerID="7477f63cb419fb6a54ccbdb71cd5379f790de4aceb42ecd453a5cc3ce70f76d2" exitCode=0
Oct 30 13:40:39.986966 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:39.986894    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9" event={"ID":"7692a16a-4d89-4f37-85ae-6c43dad93d35","Type":"ContainerDied","Data":"7477f63cb419fb6a54ccbdb71cd5379f790de4aceb42ecd453a5cc3ce70f76d2"}
Oct 30 13:40:39.986966 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:39.986936    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9" event={"ID":"7692a16a-4d89-4f37-85ae-6c43dad93d35","Type":"ContainerDied","Data":"391e33c57549068c7a602a1afd6e44961c87e10dbbbfb3ec9751ac063534867e"}
Oct 30 13:40:39.987078 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:39.987021    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9"
Oct 30 13:40:39.998593 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:39.998571    4544 scope.go:122] "RemoveContainer" containerID="abc3b345cdc771fc118889cfa8168a951045cee88b541355d43b90069fc23793"
Oct 30 13:40:39.999445 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:40:39.999421    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abc3b345cdc771fc118889cfa8168a951045cee88b541355d43b90069fc23793\": container with ID starting with abc3b345cdc771fc118889cfa8168a951045cee88b541355d43b90069fc23793 not found: ID does not exist" containerID="abc3b345cdc771fc118889cfa8168a951045cee88b541355d43b90069fc23793"
Oct 30 13:40:39.999594 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:39.999450    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abc3b345cdc771fc118889cfa8168a951045cee88b541355d43b90069fc23793"} err="failed to get container status \"abc3b345cdc771fc118889cfa8168a951045cee88b541355d43b90069fc23793\": rpc error: code = NotFound desc = could not find container \"abc3b345cdc771fc118889cfa8168a951045cee88b541355d43b90069fc23793\": container with ID starting with abc3b345cdc771fc118889cfa8168a951045cee88b541355d43b90069fc23793 not found: ID does not exist"
Oct 30 13:40:39.999594 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:39.999469    4544 scope.go:122] "RemoveContainer" containerID="7477f63cb419fb6a54ccbdb71cd5379f790de4aceb42ecd453a5cc3ce70f76d2"
Oct 30 13:40:40.009514 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:40.009493    4544 scope.go:122] "RemoveContainer" containerID="7477f63cb419fb6a54ccbdb71cd5379f790de4aceb42ecd453a5cc3ce70f76d2"
Oct 30 13:40:40.009860 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:40:40.009837    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7477f63cb419fb6a54ccbdb71cd5379f790de4aceb42ecd453a5cc3ce70f76d2\": container with ID starting with 7477f63cb419fb6a54ccbdb71cd5379f790de4aceb42ecd453a5cc3ce70f76d2 not found: ID does not exist" containerID="7477f63cb419fb6a54ccbdb71cd5379f790de4aceb42ecd453a5cc3ce70f76d2"
Oct 30 13:40:40.009937 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:40.009874    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7477f63cb419fb6a54ccbdb71cd5379f790de4aceb42ecd453a5cc3ce70f76d2"} err="failed to get container status \"7477f63cb419fb6a54ccbdb71cd5379f790de4aceb42ecd453a5cc3ce70f76d2\": rpc error: code = NotFound desc = could not find container \"7477f63cb419fb6a54ccbdb71cd5379f790de4aceb42ecd453a5cc3ce70f76d2\": container with ID starting with 7477f63cb419fb6a54ccbdb71cd5379f790de4aceb42ecd453a5cc3ce70f76d2 not found: ID does not exist"
Oct 30 13:40:40.060524 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:40.060487    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9"]
Oct 30 13:40:40.064842 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:40.064809    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9"]
Oct 30 13:40:40.069438 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:40.069411    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g7g6\" (UniqueName: \"kubernetes.io/projected/cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb-kube-api-access-8g7g6\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:40.069438 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:40.069438    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:40.101510 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:40.101474    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j"]
Oct 30 13:40:40.144093 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:40.144064    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j"]
Oct 30 13:40:40.634481 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:40.634437    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p" podUID="88adbb69-5dc6-4d77-96f4-5b6f61d80e8b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:40:40.636374 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:40.636331    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9" podUID="9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:40:41.054276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:41.054237    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:41.085633 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:41.085600    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7692a16a-4d89-4f37-85ae-6c43dad93d35" path="/var/lib/kubelet/pods/7692a16a-4d89-4f37-85ae-6c43dad93d35/volumes"
Oct 30 13:40:41.086051 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:41.086033    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb" path="/var/lib/kubelet/pods/cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb/volumes"
Oct 30 13:40:41.176149 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:41.176098    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:41.334243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:41.333483    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:41.510969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:41.510926    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg"]
Oct 30 13:40:41.670155 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:41.670067    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:41.788678 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:41.788640    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:42.002984 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.002896    4544 generic.go:356] "Generic (PLEG): container finished" podID="565c6710-ac33-437a-b2cc-5572a325523d" containerID="3cce96af41f9471fb3b52d776887166145d78a3b078c98cc1ca160c0c9ce9498" exitCode=0
Oct 30 13:40:42.002984 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.002971    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h" event={"ID":"565c6710-ac33-437a-b2cc-5572a325523d","Type":"ContainerDied","Data":"3cce96af41f9471fb3b52d776887166145d78a3b078c98cc1ca160c0c9ce9498"}
Oct 30 13:40:42.003252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.003037    4544 scope.go:122] "RemoveContainer" containerID="4c146428fb3ecc1e27887d68f26980067a3c9352a0ecf97b6871ee162561feaa"
Oct 30 13:40:42.253301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.252764    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:42.364044 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.364011    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:40:42.398159 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.398121    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-istio-envoy\") pod \"565c6710-ac33-437a-b2cc-5572a325523d\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") "
Oct 30 13:40:42.398409 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.398175    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/565c6710-ac33-437a-b2cc-5572a325523d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/565c6710-ac33-437a-b2cc-5572a325523d-istiod-ca-cert\") pod \"565c6710-ac33-437a-b2cc-5572a325523d\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") "
Oct 30 13:40:42.398409 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.398382    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-workload-socket\") pod \"565c6710-ac33-437a-b2cc-5572a325523d\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") "
Oct 30 13:40:42.398557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.398419    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-istio-data\") pod \"565c6710-ac33-437a-b2cc-5572a325523d\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") "
Oct 30 13:40:42.398557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.398457    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-workload-certs\") pod \"565c6710-ac33-437a-b2cc-5572a325523d\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") "
Oct 30 13:40:42.398557 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.398486    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/565c6710-ac33-437a-b2cc-5572a325523d-kube-api-access-qqvhj\" (UniqueName: \"kubernetes.io/projected/565c6710-ac33-437a-b2cc-5572a325523d-kube-api-access-qqvhj\") pod \"565c6710-ac33-437a-b2cc-5572a325523d\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") "
Oct 30 13:40:42.398751 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.398600    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/565c6710-ac33-437a-b2cc-5572a325523d-istio-token\" (UniqueName: \"kubernetes.io/projected/565c6710-ac33-437a-b2cc-5572a325523d-istio-token\") pod \"565c6710-ac33-437a-b2cc-5572a325523d\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") "
Oct 30 13:40:42.398751 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.398636    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-workload-socket" pod "565c6710-ac33-437a-b2cc-5572a325523d" (UID: "565c6710-ac33-437a-b2cc-5572a325523d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:42.398751 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.398657    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/565c6710-ac33-437a-b2cc-5572a325523d-istiod-ca-cert" pod "565c6710-ac33-437a-b2cc-5572a325523d" (UID: "565c6710-ac33-437a-b2cc-5572a325523d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:42.398751 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.398646    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-credential-socket\") pod \"565c6710-ac33-437a-b2cc-5572a325523d\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") "
Oct 30 13:40:42.398751 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.398724    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/565c6710-ac33-437a-b2cc-5572a325523d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/565c6710-ac33-437a-b2cc-5572a325523d-istio-podinfo\") pod \"565c6710-ac33-437a-b2cc-5572a325523d\" (UID: \"565c6710-ac33-437a-b2cc-5572a325523d\") "
Oct 30 13:40:42.399038 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.398845    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-istio-data" pod "565c6710-ac33-437a-b2cc-5572a325523d" (UID: "565c6710-ac33-437a-b2cc-5572a325523d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:42.399256 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.399195    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/565c6710-ac33-437a-b2cc-5572a325523d-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:42.399256 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.399242    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:42.399256 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.399258    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:42.399495 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.399288    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-credential-socket" pod "565c6710-ac33-437a-b2cc-5572a325523d" (UID: "565c6710-ac33-437a-b2cc-5572a325523d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:42.399495 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.399439    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-workload-certs" pod "565c6710-ac33-437a-b2cc-5572a325523d" (UID: "565c6710-ac33-437a-b2cc-5572a325523d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:42.400860 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.400816    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/565c6710-ac33-437a-b2cc-5572a325523d-istio-podinfo" pod "565c6710-ac33-437a-b2cc-5572a325523d" (UID: "565c6710-ac33-437a-b2cc-5572a325523d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:40:42.401404 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.401364    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-istio-envoy" pod "565c6710-ac33-437a-b2cc-5572a325523d" (UID: "565c6710-ac33-437a-b2cc-5572a325523d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:42.401527 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.401463    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/565c6710-ac33-437a-b2cc-5572a325523d-kube-api-access-qqvhj" pod "565c6710-ac33-437a-b2cc-5572a325523d" (UID: "565c6710-ac33-437a-b2cc-5572a325523d"). InnerVolumeSpecName "kube-api-access-qqvhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:42.401617 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.401590    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/565c6710-ac33-437a-b2cc-5572a325523d-istio-token" pod "565c6710-ac33-437a-b2cc-5572a325523d" (UID: "565c6710-ac33-437a-b2cc-5572a325523d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:42.500573 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.500538    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:42.500573 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.500570    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqvhj\" (UniqueName: \"kubernetes.io/projected/565c6710-ac33-437a-b2cc-5572a325523d-kube-api-access-qqvhj\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:42.500826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.500584    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/565c6710-ac33-437a-b2cc-5572a325523d-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:42.500826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.500593    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:42.500826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.500603    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/565c6710-ac33-437a-b2cc-5572a325523d-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:42.500826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.500611    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/565c6710-ac33-437a-b2cc-5572a325523d-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:42.516399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.515881    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9"
Oct 30 13:40:42.587624 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.587587    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p"
Oct 30 13:40:42.601665 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.601628    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69-plugins-config-volume\") pod \"9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69\" (UID: \"9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69\") "
Oct 30 13:40:42.601887 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.601682    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69-kube-api-access-kbnkl\" (UniqueName: \"kubernetes.io/projected/9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69-kube-api-access-kbnkl\") pod \"9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69\" (UID: \"9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69\") "
Oct 30 13:40:42.601887 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.601705    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88adbb69-5dc6-4d77-96f4-5b6f61d80e8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88adbb69-5dc6-4d77-96f4-5b6f61d80e8b-plugins-config-volume\") pod \"88adbb69-5dc6-4d77-96f4-5b6f61d80e8b\" (UID: \"88adbb69-5dc6-4d77-96f4-5b6f61d80e8b\") "
Oct 30 13:40:42.602029 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.601895    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88adbb69-5dc6-4d77-96f4-5b6f61d80e8b-kube-api-access-l8xn4\" (UniqueName: \"kubernetes.io/projected/88adbb69-5dc6-4d77-96f4-5b6f61d80e8b-kube-api-access-l8xn4\") pod \"88adbb69-5dc6-4d77-96f4-5b6f61d80e8b\" (UID: \"88adbb69-5dc6-4d77-96f4-5b6f61d80e8b\") "
Oct 30 13:40:42.602100 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.602066    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88adbb69-5dc6-4d77-96f4-5b6f61d80e8b-plugins-config-volume" pod "88adbb69-5dc6-4d77-96f4-5b6f61d80e8b" (UID: "88adbb69-5dc6-4d77-96f4-5b6f61d80e8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:42.602159 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.602117    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69-plugins-config-volume" pod "9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69" (UID: "9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:42.602487 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.602462    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:42.602597 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.602490    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88adbb69-5dc6-4d77-96f4-5b6f61d80e8b-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:42.604360 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.604328    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88adbb69-5dc6-4d77-96f4-5b6f61d80e8b-kube-api-access-l8xn4" pod "88adbb69-5dc6-4d77-96f4-5b6f61d80e8b" (UID: "88adbb69-5dc6-4d77-96f4-5b6f61d80e8b"). InnerVolumeSpecName "kube-api-access-l8xn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:42.605020 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.604985    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69-kube-api-access-kbnkl" pod "9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69" (UID: "9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69"). InnerVolumeSpecName "kube-api-access-kbnkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:42.703952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.703906    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbnkl\" (UniqueName: \"kubernetes.io/projected/9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69-kube-api-access-kbnkl\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:42.703952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:42.703941    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8xn4\" (UniqueName: \"kubernetes.io/projected/88adbb69-5dc6-4d77-96f4-5b6f61d80e8b-kube-api-access-l8xn4\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:43.011777 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.011739    4544 generic.go:356] "Generic (PLEG): container finished" podID="88adbb69-5dc6-4d77-96f4-5b6f61d80e8b" containerID="f28609be50de347a74439be7a1d643291e94f00937792c40b6551fb64848aaa3" exitCode=0
Oct 30 13:40:43.012003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.011789    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p"
Oct 30 13:40:43.012003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.011814    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p" event={"ID":"88adbb69-5dc6-4d77-96f4-5b6f61d80e8b","Type":"ContainerDied","Data":"f28609be50de347a74439be7a1d643291e94f00937792c40b6551fb64848aaa3"}
Oct 30 13:40:43.012003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.011877    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p" event={"ID":"88adbb69-5dc6-4d77-96f4-5b6f61d80e8b","Type":"ContainerDied","Data":"cb22d3ec01f839afec503fb3f2d9f8fe47cf14cdaadb3f4e7b61cd195d065245"}
Oct 30 13:40:43.012003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.011906    4544 scope.go:122] "RemoveContainer" containerID="f28609be50de347a74439be7a1d643291e94f00937792c40b6551fb64848aaa3"
Oct 30 13:40:43.014344 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.014326    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"
Oct 30 13:40:43.014495 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.014344    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h" event={"ID":"565c6710-ac33-437a-b2cc-5572a325523d","Type":"ContainerDied","Data":"4f0ae5283c3f27a5faa911332c32f300bb414d59418e2daae21007fbb2ec632c"}
Oct 30 13:40:43.016295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.016135    4544 generic.go:356] "Generic (PLEG): container finished" podID="9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69" containerID="904ec00d53d4370c85ee979df2ca10b3d90f92e5d6ea03c6d4b5b3a0369006b7" exitCode=0
Oct 30 13:40:43.016295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.016176    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9" event={"ID":"9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69","Type":"ContainerDied","Data":"904ec00d53d4370c85ee979df2ca10b3d90f92e5d6ea03c6d4b5b3a0369006b7"}
Oct 30 13:40:43.016295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.016185    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9"
Oct 30 13:40:43.016295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.016252    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9" event={"ID":"9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69","Type":"ContainerDied","Data":"8196ea15f2043214ef2b0d305a97dfe73a98924f76578073c77858e5b8c8b9ae"}
Oct 30 13:40:43.026035 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.026010    4544 scope.go:122] "RemoveContainer" containerID="f28609be50de347a74439be7a1d643291e94f00937792c40b6551fb64848aaa3"
Oct 30 13:40:43.026834 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:40:43.026808    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f28609be50de347a74439be7a1d643291e94f00937792c40b6551fb64848aaa3\": container with ID starting with f28609be50de347a74439be7a1d643291e94f00937792c40b6551fb64848aaa3 not found: ID does not exist" containerID="f28609be50de347a74439be7a1d643291e94f00937792c40b6551fb64848aaa3"
Oct 30 13:40:43.026950 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.026842    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f28609be50de347a74439be7a1d643291e94f00937792c40b6551fb64848aaa3"} err="failed to get container status \"f28609be50de347a74439be7a1d643291e94f00937792c40b6551fb64848aaa3\": rpc error: code = NotFound desc = could not find container \"f28609be50de347a74439be7a1d643291e94f00937792c40b6551fb64848aaa3\": container with ID starting with f28609be50de347a74439be7a1d643291e94f00937792c40b6551fb64848aaa3 not found: ID does not exist"
Oct 30 13:40:43.026950 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.026861    4544 scope.go:122] "RemoveContainer" containerID="3cce96af41f9471fb3b52d776887166145d78a3b078c98cc1ca160c0c9ce9498"
Oct 30 13:40:43.037241 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.037191    4544 scope.go:122] "RemoveContainer" containerID="904ec00d53d4370c85ee979df2ca10b3d90f92e5d6ea03c6d4b5b3a0369006b7"
Oct 30 13:40:43.047619 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.047600    4544 scope.go:122] "RemoveContainer" containerID="904ec00d53d4370c85ee979df2ca10b3d90f92e5d6ea03c6d4b5b3a0369006b7"
Oct 30 13:40:43.047917 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:40:43.047897    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"904ec00d53d4370c85ee979df2ca10b3d90f92e5d6ea03c6d4b5b3a0369006b7\": container with ID starting with 904ec00d53d4370c85ee979df2ca10b3d90f92e5d6ea03c6d4b5b3a0369006b7 not found: ID does not exist" containerID="904ec00d53d4370c85ee979df2ca10b3d90f92e5d6ea03c6d4b5b3a0369006b7"
Oct 30 13:40:43.047985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.047930    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"904ec00d53d4370c85ee979df2ca10b3d90f92e5d6ea03c6d4b5b3a0369006b7"} err="failed to get container status \"904ec00d53d4370c85ee979df2ca10b3d90f92e5d6ea03c6d4b5b3a0369006b7\": rpc error: code = NotFound desc = could not find container \"904ec00d53d4370c85ee979df2ca10b3d90f92e5d6ea03c6d4b5b3a0369006b7\": container with ID starting with 904ec00d53d4370c85ee979df2ca10b3d90f92e5d6ea03c6d4b5b3a0369006b7 not found: ID does not exist"
Oct 30 13:40:43.088412 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.088382    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p"]
Oct 30 13:40:43.088964 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.088940    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p"]
Oct 30 13:40:43.142462 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.142381    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"]
Oct 30 13:40:43.146902 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.146732    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7ee8148-77kd8"]
Oct 30 13:40:43.147696 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.147669    4544 cpu_manager.go:414] "containerMap: removing container" podUID="565c6710-ac33-437a-b2cc-5572a325523d" containerName="istio-proxy"
Oct 30 13:40:43.147696 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.147696    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="565c6710-ac33-437a-b2cc-5572a325523d" containerName="istio-proxy"
Oct 30 13:40:43.147918 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.147712    4544 cpu_manager.go:414] "containerMap: removing container" podUID="db2e31d3-d423-4671-8449-56d1f380fbd1" containerName="payload-processing"
Oct 30 13:40:43.147918 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.147721    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db2e31d3-d423-4671-8449-56d1f380fbd1" containerName="payload-processing"
Oct 30 13:40:43.147918 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.147760    4544 cpu_manager.go:414] "containerMap: removing container" podUID="8d82c543-383d-4498-8d3c-d6c2a0716736" containerName="payload-pre-processing"
Oct 30 13:40:43.147918 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.147768    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d82c543-383d-4498-8d3c-d6c2a0716736" containerName="payload-pre-processing"
Oct 30 13:40:43.147918 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.147789    4544 cpu_manager.go:414] "containerMap: removing container" podUID="7692a16a-4d89-4f37-85ae-6c43dad93d35" containerName="payload-processing"
Oct 30 13:40:43.147918 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.147796    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7692a16a-4d89-4f37-85ae-6c43dad93d35" containerName="payload-processing"
Oct 30 13:40:43.147918 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.147813    4544 cpu_manager.go:414] "containerMap: removing container" podUID="cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb" containerName="payload-pre-processing"
Oct 30 13:40:43.147918 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.147821    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb" containerName="payload-pre-processing"
Oct 30 13:40:43.147918 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.147835    4544 cpu_manager.go:414] "containerMap: removing container" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main"
Oct 30 13:40:43.147918 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.147842    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main"
Oct 30 13:40:43.147918 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.147856    4544 cpu_manager.go:414] "containerMap: removing container" podUID="88adbb69-5dc6-4d77-96f4-5b6f61d80e8b" containerName="payload-pre-processing"
Oct 30 13:40:43.147918 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.147864    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88adbb69-5dc6-4d77-96f4-5b6f61d80e8b" containerName="payload-pre-processing"
Oct 30 13:40:43.147918 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.147875    4544 cpu_manager.go:414] "containerMap: removing container" podUID="9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69" containerName="payload-processing"
Oct 30 13:40:43.147918 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.147882    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69" containerName="payload-processing"
Oct 30 13:40:43.148924 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.147996    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc60a80e-9ec6-4cbb-9e64-5b1cf0de90bb" containerName="payload-pre-processing"
Oct 30 13:40:43.148924 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.148011    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="b66ef925-6b4b-4859-a706-960515034d64" containerName="main"
Oct 30 13:40:43.148924 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.148024    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="88adbb69-5dc6-4d77-96f4-5b6f61d80e8b" containerName="payload-pre-processing"
Oct 30 13:40:43.148924 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.148034    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69" containerName="payload-processing"
Oct 30 13:40:43.148924 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.148044    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="565c6710-ac33-437a-b2cc-5572a325523d" containerName="istio-proxy"
Oct 30 13:40:43.148924 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.148052    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="db2e31d3-d423-4671-8449-56d1f380fbd1" containerName="payload-processing"
Oct 30 13:40:43.148924 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.148065    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d82c543-383d-4498-8d3c-d6c2a0716736" containerName="payload-pre-processing"
Oct 30 13:40:43.148924 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.148077    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="7692a16a-4d89-4f37-85ae-6c43dad93d35" containerName="payload-processing"
Oct 30 13:40:43.156868 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.156827    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h"]
Oct 30 13:40:43.157015 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.156987    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ee8148-77kd8"
Oct 30 13:40:43.174158 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.174122    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7ee8148-77kd8"]
Oct 30 13:40:43.188248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.188197    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9"]
Oct 30 13:40:43.194590 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.194557    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9"]
Oct 30 13:40:43.209874 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.209840    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmm7v\" (UniqueName: \"kubernetes.io/projected/3e781781-4614-4cb3-84c9-987ca6719f12-kube-api-access-xmm7v\") pod \"maas-api-revoke-sub-c7ee8148-77kd8\" (UID: \"3e781781-4614-4cb3-84c9-987ca6719f12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ee8148-77kd8"
Oct 30 13:40:43.210077 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.209898    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e781781-4614-4cb3-84c9-987ca6719f12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7ee8148-77kd8\" (UID: \"3e781781-4614-4cb3-84c9-987ca6719f12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ee8148-77kd8"
Oct 30 13:40:43.210077 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.210017    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e781781-4614-4cb3-84c9-987ca6719f12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7ee8148-77kd8\" (UID: \"3e781781-4614-4cb3-84c9-987ca6719f12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ee8148-77kd8"
Oct 30 13:40:43.244378 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.244336    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:40:43.244378 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.244337    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:40:43.311106 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.311005    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e781781-4614-4cb3-84c9-987ca6719f12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7ee8148-77kd8\" (UID: \"3e781781-4614-4cb3-84c9-987ca6719f12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ee8148-77kd8"
Oct 30 13:40:43.311528 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.311274    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmm7v\" (UniqueName: \"kubernetes.io/projected/3e781781-4614-4cb3-84c9-987ca6719f12-kube-api-access-xmm7v\") pod \"maas-api-revoke-sub-c7ee8148-77kd8\" (UID: \"3e781781-4614-4cb3-84c9-987ca6719f12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ee8148-77kd8"
Oct 30 13:40:43.311528 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.311323    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e781781-4614-4cb3-84c9-987ca6719f12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7ee8148-77kd8\" (UID: \"3e781781-4614-4cb3-84c9-987ca6719f12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ee8148-77kd8"
Oct 30 13:40:43.311680 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.311660    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e781781-4614-4cb3-84c9-987ca6719f12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7ee8148-77kd8\" (UID: \"3e781781-4614-4cb3-84c9-987ca6719f12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ee8148-77kd8"
Oct 30 13:40:43.313958 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.313918    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e781781-4614-4cb3-84c9-987ca6719f12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7ee8148-77kd8\" (UID: \"3e781781-4614-4cb3-84c9-987ca6719f12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ee8148-77kd8"
Oct 30 13:40:43.339685 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.339649    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmm7v\" (UniqueName: \"kubernetes.io/projected/3e781781-4614-4cb3-84c9-987ca6719f12-kube-api-access-xmm7v\") pod \"maas-api-revoke-sub-c7ee8148-77kd8\" (UID: \"3e781781-4614-4cb3-84c9-987ca6719f12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ee8148-77kd8"
Oct 30 13:40:43.473874 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.473830    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ee8148-77kd8"
Oct 30 13:40:43.714883 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.714847    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:43.789945 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:43.789904    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7ee8148-77kd8"]
Oct 30 13:40:43.793790 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:40:43.793741    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e781781_4614_4cb3_84c9_987ca6719f12.slice/crio-279782b2858f7a211c0cbefd280019024a2a89b4bd4e084454f78d008a10de27 WatchSource:0}: Error finding container 279782b2858f7a211c0cbefd280019024a2a89b4bd4e084454f78d008a10de27: Status 404 returned error can't find the container with id 279782b2858f7a211c0cbefd280019024a2a89b4bd4e084454f78d008a10de27
Oct 30 13:40:44.027795 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:44.027698    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ee8148-77kd8" event={"ID":"3e781781-4614-4cb3-84c9-987ca6719f12","Type":"ContainerStarted","Data":"279782b2858f7a211c0cbefd280019024a2a89b4bd4e084454f78d008a10de27"}
Oct 30 13:40:45.038802 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:45.038759    4544 generic.go:356] "Generic (PLEG): container finished" podID="3e781781-4614-4cb3-84c9-987ca6719f12" containerID="e9a683ec2b08978bdb04768f18f79843217b622048936d0a2cd4f47bad42b7ae" exitCode=0
Oct 30 13:40:45.039203 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:45.038798    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ee8148-77kd8" event={"ID":"3e781781-4614-4cb3-84c9-987ca6719f12","Type":"ContainerDied","Data":"e9a683ec2b08978bdb04768f18f79843217b622048936d0a2cd4f47bad42b7ae"}
Oct 30 13:40:45.090998 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:45.090954    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="565c6710-ac33-437a-b2cc-5572a325523d" path="/var/lib/kubelet/pods/565c6710-ac33-437a-b2cc-5572a325523d/volumes"
Oct 30 13:40:45.091920 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:45.091891    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88adbb69-5dc6-4d77-96f4-5b6f61d80e8b" path="/var/lib/kubelet/pods/88adbb69-5dc6-4d77-96f4-5b6f61d80e8b/volumes"
Oct 30 13:40:45.092459 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:45.092439    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69" path="/var/lib/kubelet/pods/9fa9c95e-f9d7-44b5-b6f0-ccd15df41f69/volumes"
Oct 30 13:40:45.275201 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:45.275160    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp"]
Oct 30 13:40:45.275474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:45.275439    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp" podUID="6db19ff5-f476-4582-8153-b2b750f45032" containerName="maas-api" containerID="cri-o://83f865d4709a52fcdd4269f4bef1f974e4e186574af701c25496ab8c06879851" gracePeriod=30
Oct 30 13:40:45.347534 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:45.347447    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:45.473453 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:45.473408    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:45.541830 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:45.541798    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp"
Oct 30 13:40:45.636598 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:45.636519    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6db19ff5-f476-4582-8153-b2b750f45032-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db19ff5-f476-4582-8153-b2b750f45032-maas-api-tls\") pod \"6db19ff5-f476-4582-8153-b2b750f45032\" (UID: \"6db19ff5-f476-4582-8153-b2b750f45032\") "
Oct 30 13:40:45.636598 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:45.636568    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6db19ff5-f476-4582-8153-b2b750f45032-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6db19ff5-f476-4582-8153-b2b750f45032-metrics-certs\") pod \"6db19ff5-f476-4582-8153-b2b750f45032\" (UID: \"6db19ff5-f476-4582-8153-b2b750f45032\") "
Oct 30 13:40:45.636816 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:45.636713    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6db19ff5-f476-4582-8153-b2b750f45032-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db19ff5-f476-4582-8153-b2b750f45032-openshift-service-ca\") pod \"6db19ff5-f476-4582-8153-b2b750f45032\" (UID: \"6db19ff5-f476-4582-8153-b2b750f45032\") "
Oct 30 13:40:45.636855 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:45.636820    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6db19ff5-f476-4582-8153-b2b750f45032-kube-api-access-gmsvb\" (UniqueName: \"kubernetes.io/projected/6db19ff5-f476-4582-8153-b2b750f45032-kube-api-access-gmsvb\") pod \"6db19ff5-f476-4582-8153-b2b750f45032\" (UID: \"6db19ff5-f476-4582-8153-b2b750f45032\") "
Oct 30 13:40:45.637169 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:45.637136    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db19ff5-f476-4582-8153-b2b750f45032-openshift-service-ca" pod "6db19ff5-f476-4582-8153-b2b750f45032" (UID: "6db19ff5-f476-4582-8153-b2b750f45032"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:45.637345 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:45.637254    4544 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db19ff5-f476-4582-8153-b2b750f45032-openshift-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:45.638982 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:45.638947    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db19ff5-f476-4582-8153-b2b750f45032-kube-api-access-gmsvb" pod "6db19ff5-f476-4582-8153-b2b750f45032" (UID: "6db19ff5-f476-4582-8153-b2b750f45032"). InnerVolumeSpecName "kube-api-access-gmsvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:45.639358 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:45.639336    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6db19ff5-f476-4582-8153-b2b750f45032-maas-api-tls" pod "6db19ff5-f476-4582-8153-b2b750f45032" (UID: "6db19ff5-f476-4582-8153-b2b750f45032"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:40:45.639463 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:45.639374    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6db19ff5-f476-4582-8153-b2b750f45032-metrics-certs" pod "6db19ff5-f476-4582-8153-b2b750f45032" (UID: "6db19ff5-f476-4582-8153-b2b750f45032"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:40:45.742249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:45.739497    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmsvb\" (UniqueName: \"kubernetes.io/projected/6db19ff5-f476-4582-8153-b2b750f45032-kube-api-access-gmsvb\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:45.742249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:45.739540    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db19ff5-f476-4582-8153-b2b750f45032-maas-api-tls\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:45.742249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:45.739556    4544 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6db19ff5-f476-4582-8153-b2b750f45032-metrics-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:45.890550 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:45.890457    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:46.025623 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:46.025573    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:46.047723 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:46.047689    4544 generic.go:356] "Generic (PLEG): container finished" podID="6db19ff5-f476-4582-8153-b2b750f45032" containerID="83f865d4709a52fcdd4269f4bef1f974e4e186574af701c25496ab8c06879851" exitCode=0
Oct 30 13:40:46.048155 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:46.047749    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp"
Oct 30 13:40:46.048155 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:46.047773    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp" event={"ID":"6db19ff5-f476-4582-8153-b2b750f45032","Type":"ContainerDied","Data":"83f865d4709a52fcdd4269f4bef1f974e4e186574af701c25496ab8c06879851"}
Oct 30 13:40:46.048155 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:46.047828    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp" event={"ID":"6db19ff5-f476-4582-8153-b2b750f45032","Type":"ContainerDied","Data":"577d603f349b9d38046afe314cc1ab0cde451829dd935f24db68639abb769845"}
Oct 30 13:40:46.048155 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:46.047852    4544 scope.go:122] "RemoveContainer" containerID="83f865d4709a52fcdd4269f4bef1f974e4e186574af701c25496ab8c06879851"
Oct 30 13:40:46.079772 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:46.079402    4544 scope.go:122] "RemoveContainer" containerID="83f865d4709a52fcdd4269f4bef1f974e4e186574af701c25496ab8c06879851"
Oct 30 13:40:46.080034 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:40:46.079924    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83f865d4709a52fcdd4269f4bef1f974e4e186574af701c25496ab8c06879851\": container with ID starting with 83f865d4709a52fcdd4269f4bef1f974e4e186574af701c25496ab8c06879851 not found: ID does not exist" containerID="83f865d4709a52fcdd4269f4bef1f974e4e186574af701c25496ab8c06879851"
Oct 30 13:40:46.080174 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:46.080051    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83f865d4709a52fcdd4269f4bef1f974e4e186574af701c25496ab8c06879851"} err="failed to get container status \"83f865d4709a52fcdd4269f4bef1f974e4e186574af701c25496ab8c06879851\": rpc error: code = NotFound desc = could not find container \"83f865d4709a52fcdd4269f4bef1f974e4e186574af701c25496ab8c06879851\": container with ID starting with 83f865d4709a52fcdd4269f4bef1f974e4e186574af701c25496ab8c06879851 not found: ID does not exist"
Oct 30 13:40:46.123241 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:46.122306    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp"]
Oct 30 13:40:46.143156 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:46.123586    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b46c9-787565b66f-gspcp"]
Oct 30 13:40:46.160415 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:46.160373    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b46c9-zkzr9"]
Oct 30 13:40:46.165119 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:46.165079    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b46c9-zkzr9"]
Oct 30 13:40:46.177976 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:46.177936    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:46.236441 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:46.236413    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ee8148-77kd8"
Oct 30 13:40:46.346152 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:46.346107    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e781781-4614-4cb3-84c9-987ca6719f12-kube-api-access-xmm7v\" (UniqueName: \"kubernetes.io/projected/3e781781-4614-4cb3-84c9-987ca6719f12-kube-api-access-xmm7v\") pod \"3e781781-4614-4cb3-84c9-987ca6719f12\" (UID: \"3e781781-4614-4cb3-84c9-987ca6719f12\") "
Oct 30 13:40:46.346152 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:46.346152    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e781781-4614-4cb3-84c9-987ca6719f12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e781781-4614-4cb3-84c9-987ca6719f12-maas-api-xks-ca\") pod \"3e781781-4614-4cb3-84c9-987ca6719f12\" (UID: \"3e781781-4614-4cb3-84c9-987ca6719f12\") "
Oct 30 13:40:46.346489 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:46.346190    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e781781-4614-4cb3-84c9-987ca6719f12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e781781-4614-4cb3-84c9-987ca6719f12-maas-api-openshift-ca\") pod \"3e781781-4614-4cb3-84c9-987ca6719f12\" (UID: \"3e781781-4614-4cb3-84c9-987ca6719f12\") "
Oct 30 13:40:46.346607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:46.346577    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e781781-4614-4cb3-84c9-987ca6719f12-maas-api-openshift-ca" pod "3e781781-4614-4cb3-84c9-987ca6719f12" (UID: "3e781781-4614-4cb3-84c9-987ca6719f12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:46.348424 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:46.348389    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e781781-4614-4cb3-84c9-987ca6719f12-maas-api-xks-ca" pod "3e781781-4614-4cb3-84c9-987ca6719f12" (UID: "3e781781-4614-4cb3-84c9-987ca6719f12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:40:46.348570 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:46.348551    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e781781-4614-4cb3-84c9-987ca6719f12-kube-api-access-xmm7v" pod "3e781781-4614-4cb3-84c9-987ca6719f12" (UID: "3e781781-4614-4cb3-84c9-987ca6719f12"). InnerVolumeSpecName "kube-api-access-xmm7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:46.447651 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:46.447558    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmm7v\" (UniqueName: \"kubernetes.io/projected/3e781781-4614-4cb3-84c9-987ca6719f12-kube-api-access-xmm7v\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:46.447651 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:46.447593    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e781781-4614-4cb3-84c9-987ca6719f12-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:46.447651 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:46.447603    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e781781-4614-4cb3-84c9-987ca6719f12-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:46.512320 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:46.512272    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg" podUID="3c07693a-fc68-4472-a5d0-d7d7ba02bdc2" containerName="payload-processing" containerID="cri-o://7fb4b4efc97765b6fe2f8085ac576ada6731188d1a4f6d6c353a1f6be7c83a68" gracePeriod=25
Oct 30 13:40:46.550034 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:46.549994    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:46.672914 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:46.672876    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:46.914378 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:46.914341    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:47.058016 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:47.057976    4544 generic.go:356] "Generic (PLEG): container finished" podID="e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4" containerID="0257c01bd914e8974dfe7a6529ab9f0f275cf73f72c147e567b92ed8d92683d9" exitCode=0
Oct 30 13:40:47.058587 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:47.058135    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r" event={"ID":"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4","Type":"ContainerDied","Data":"0257c01bd914e8974dfe7a6529ab9f0f275cf73f72c147e567b92ed8d92683d9"}
Oct 30 13:40:47.058657 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:47.058624    4544 scope.go:122] "RemoveContainer" containerID="0257c01bd914e8974dfe7a6529ab9f0f275cf73f72c147e567b92ed8d92683d9"
Oct 30 13:40:47.060730 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:47.060690    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:40:47.060868 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:47.060792    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:40:47.060868 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:47.060841    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:40:47.075349 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:47.071832    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ee8148-77kd8" event={"ID":"3e781781-4614-4cb3-84c9-987ca6719f12","Type":"ContainerDied","Data":"279782b2858f7a211c0cbefd280019024a2a89b4bd4e084454f78d008a10de27"}
Oct 30 13:40:47.075349 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:47.071895    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="279782b2858f7a211c0cbefd280019024a2a89b4bd4e084454f78d008a10de27"
Oct 30 13:40:47.075349 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:47.071964    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ee8148-77kd8"
Oct 30 13:40:47.094504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:47.092624    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6db19ff5-f476-4582-8153-b2b750f45032" path="/var/lib/kubelet/pods/6db19ff5-f476-4582-8153-b2b750f45032/volumes"
Oct 30 13:40:47.094504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:47.093275    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7368bce-df71-4576-90e7-923724db392c" path="/var/lib/kubelet/pods/c7368bce-df71-4576-90e7-923724db392c/volumes"
Oct 30 13:40:47.099763 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:47.099733    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"]
Oct 30 13:40:47.099978 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:47.099954    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9" podUID="9f4af86d-c526-4365-b56d-32a7f46247bf" containerName="istio-proxy" containerID="cri-o://adab87174c517cf155db0afc4bc2d2335c6d1077e35d5d1dc6eb43b050ae280c" gracePeriod=30
Oct 30 13:40:47.174022 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:47.173914    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:47.865340 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:47.865302    4544 patch_prober.go:29] interesting pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:40:47.865509 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:47.865360    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9" podUID="9f4af86d-c526-4365-b56d-32a7f46247bf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:40:47.958984 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:47.958946    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj"]
Oct 30 13:40:48.087009 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:48.086967    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r" event={"ID":"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4","Type":"ContainerStarted","Data":"a9c0d02ae26601eb8f4bb6c3aaeb7dec41cfe823c4b498601d9f7edeca61e2b2"}
Oct 30 13:40:48.143300 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:48.112662    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:48.339246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:48.339192    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:48.547326 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:48.547141    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:48.765654 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:48.765617    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:48.980191 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:48.980145    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:48.981513 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:48.981483    4544 patch_prober.go:29] interesting pod/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused" start-of-body=
Oct 30 13:40:48.981682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:48.981582    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r" podUID="e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused"
Oct 30 13:40:49.846169 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:49.846131    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6"]
Oct 30 13:40:49.980783 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:49.980748    4544 patch_prober.go:29] interesting pod/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused" start-of-body=
Oct 30 13:40:49.980968 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:49.980818    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r" podUID="e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused"
Oct 30 13:40:50.004380 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:50.004347    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:50.162735 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:50.162635    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:50.338187 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:50.338138    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:50.939088 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:50.939051    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:50.980647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:50.980579    4544 patch_prober.go:29] interesting pod/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused" start-of-body=
Oct 30 13:40:50.980863 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:50.980666    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r" podUID="e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused"
Oct 30 13:40:51.637774 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:51.637714    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg" podUID="3c07693a-fc68-4472-a5d0-d7d7ba02bdc2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:40:51.981095 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:51.980997    4544 patch_prober.go:29] interesting pod/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused" start-of-body=
Oct 30 13:40:51.981095 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:51.981058    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r" podUID="e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused"
Oct 30 13:40:52.441012 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:52.440982    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:40:52.547814 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:52.547704    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-workload-certs\") pod \"9f4af86d-c526-4365-b56d-32a7f46247bf\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") "
Oct 30 13:40:52.547814 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:52.547780    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f4af86d-c526-4365-b56d-32a7f46247bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4af86d-c526-4365-b56d-32a7f46247bf-istiod-ca-cert\") pod \"9f4af86d-c526-4365-b56d-32a7f46247bf\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") "
Oct 30 13:40:52.548073 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:52.547829    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-workload-socket\") pod \"9f4af86d-c526-4365-b56d-32a7f46247bf\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") "
Oct 30 13:40:52.548073 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:52.547871    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-credential-socket\") pod \"9f4af86d-c526-4365-b56d-32a7f46247bf\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") "
Oct 30 13:40:52.548073 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:52.547923    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f4af86d-c526-4365-b56d-32a7f46247bf-kube-api-access-xszpv\" (UniqueName: \"kubernetes.io/projected/9f4af86d-c526-4365-b56d-32a7f46247bf-kube-api-access-xszpv\") pod \"9f4af86d-c526-4365-b56d-32a7f46247bf\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") "
Oct 30 13:40:52.548073 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:52.547967    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-istio-data\") pod \"9f4af86d-c526-4365-b56d-32a7f46247bf\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") "
Oct 30 13:40:52.548073 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:52.548001    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-workload-certs" pod "9f4af86d-c526-4365-b56d-32a7f46247bf" (UID: "9f4af86d-c526-4365-b56d-32a7f46247bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:52.548073 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:52.548045    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-istio-envoy\") pod \"9f4af86d-c526-4365-b56d-32a7f46247bf\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") "
Oct 30 13:40:52.548491 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:52.548087    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f4af86d-c526-4365-b56d-32a7f46247bf-istio-token\" (UniqueName: \"kubernetes.io/projected/9f4af86d-c526-4365-b56d-32a7f46247bf-istio-token\") pod \"9f4af86d-c526-4365-b56d-32a7f46247bf\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") "
Oct 30 13:40:52.548491 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:52.548124    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-workload-socket" pod "9f4af86d-c526-4365-b56d-32a7f46247bf" (UID: "9f4af86d-c526-4365-b56d-32a7f46247bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:52.548491 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:52.548145    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f4af86d-c526-4365-b56d-32a7f46247bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4af86d-c526-4365-b56d-32a7f46247bf-istio-podinfo\") pod \"9f4af86d-c526-4365-b56d-32a7f46247bf\" (UID: \"9f4af86d-c526-4365-b56d-32a7f46247bf\") "
Oct 30 13:40:52.548491 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:52.548241    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-credential-socket" pod "9f4af86d-c526-4365-b56d-32a7f46247bf" (UID: "9f4af86d-c526-4365-b56d-32a7f46247bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:52.548491 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:52.548282    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f4af86d-c526-4365-b56d-32a7f46247bf-istiod-ca-cert" pod "9f4af86d-c526-4365-b56d-32a7f46247bf" (UID: "9f4af86d-c526-4365-b56d-32a7f46247bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:52.548491 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:52.548384    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-istio-data" pod "9f4af86d-c526-4365-b56d-32a7f46247bf" (UID: "9f4af86d-c526-4365-b56d-32a7f46247bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:52.548832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:52.548703    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:52.548832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:52.548726    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:52.548832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:52.548740    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4af86d-c526-4365-b56d-32a7f46247bf-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:52.548832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:52.548754    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:52.548832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:52.548770    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:52.550881 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:52.550828    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f4af86d-c526-4365-b56d-32a7f46247bf-kube-api-access-xszpv" pod "9f4af86d-c526-4365-b56d-32a7f46247bf" (UID: "9f4af86d-c526-4365-b56d-32a7f46247bf"). InnerVolumeSpecName "kube-api-access-xszpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:52.551623 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:52.551598    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f4af86d-c526-4365-b56d-32a7f46247bf-istio-podinfo" pod "9f4af86d-c526-4365-b56d-32a7f46247bf" (UID: "9f4af86d-c526-4365-b56d-32a7f46247bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:40:52.551741 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:52.551605    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-istio-envoy" pod "9f4af86d-c526-4365-b56d-32a7f46247bf" (UID: "9f4af86d-c526-4365-b56d-32a7f46247bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:52.552077 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:52.552056    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f4af86d-c526-4365-b56d-32a7f46247bf-istio-token" pod "9f4af86d-c526-4365-b56d-32a7f46247bf" (UID: "9f4af86d-c526-4365-b56d-32a7f46247bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:52.650469 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:52.650434    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4af86d-c526-4365-b56d-32a7f46247bf-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:52.650469 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:52.650466    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4af86d-c526-4365-b56d-32a7f46247bf-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:52.650469 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:52.650475    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4af86d-c526-4365-b56d-32a7f46247bf-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:52.650469 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:52.650484    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xszpv\" (UniqueName: \"kubernetes.io/projected/9f4af86d-c526-4365-b56d-32a7f46247bf-kube-api-access-xszpv\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:52.960009 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:52.959962    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj" podUID="3c724287-666b-47b4-8c6b-428a20ff1164" containerName="payload-pre-processing" containerID="cri-o://a87305396d2376abcd260fc0f333ce7c51c54539a32566ff5badd859f36702a7" gracePeriod=25
Oct 30 13:40:52.981050 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:52.981012    4544 patch_prober.go:29] interesting pod/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused" start-of-body=
Oct 30 13:40:52.981256 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:52.981073    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r" podUID="e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused"
Oct 30 13:40:53.140806 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:53.140654    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:53.146364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:53.146254    4544 generic.go:356] "Generic (PLEG): container finished" podID="9f4af86d-c526-4365-b56d-32a7f46247bf" containerID="adab87174c517cf155db0afc4bc2d2335c6d1077e35d5d1dc6eb43b050ae280c" exitCode=0
Oct 30 13:40:53.146364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:53.146318    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"
Oct 30 13:40:53.146364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:53.146347    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9" event={"ID":"9f4af86d-c526-4365-b56d-32a7f46247bf","Type":"ContainerDied","Data":"adab87174c517cf155db0afc4bc2d2335c6d1077e35d5d1dc6eb43b050ae280c"}
Oct 30 13:40:53.146656 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:53.146417    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9" event={"ID":"9f4af86d-c526-4365-b56d-32a7f46247bf","Type":"ContainerDied","Data":"f4e7f7635f092ca98f4310b0c167e1e7813a19fe93e162e9bdf969f4d04f4d35"}
Oct 30 13:40:53.146656 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:53.146440    4544 scope.go:122] "RemoveContainer" containerID="adab87174c517cf155db0afc4bc2d2335c6d1077e35d5d1dc6eb43b050ae280c"
Oct 30 13:40:53.150598 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:53.150571    4544 generic.go:356] "Generic (PLEG): container finished" podID="3c724287-666b-47b4-8c6b-428a20ff1164" containerID="a87305396d2376abcd260fc0f333ce7c51c54539a32566ff5badd859f36702a7" exitCode=0
Oct 30 13:40:53.150887 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:53.150636    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj" event={"ID":"3c724287-666b-47b4-8c6b-428a20ff1164","Type":"ContainerDied","Data":"a87305396d2376abcd260fc0f333ce7c51c54539a32566ff5badd859f36702a7"}
Oct 30 13:40:53.165721 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:53.165119    4544 scope.go:122] "RemoveContainer" containerID="adab87174c517cf155db0afc4bc2d2335c6d1077e35d5d1dc6eb43b050ae280c"
Oct 30 13:40:53.168475 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:40:53.168380    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adab87174c517cf155db0afc4bc2d2335c6d1077e35d5d1dc6eb43b050ae280c\": container with ID starting with adab87174c517cf155db0afc4bc2d2335c6d1077e35d5d1dc6eb43b050ae280c not found: ID does not exist" containerID="adab87174c517cf155db0afc4bc2d2335c6d1077e35d5d1dc6eb43b050ae280c"
Oct 30 13:40:53.168475 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:53.168432    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adab87174c517cf155db0afc4bc2d2335c6d1077e35d5d1dc6eb43b050ae280c"} err="failed to get container status \"adab87174c517cf155db0afc4bc2d2335c6d1077e35d5d1dc6eb43b050ae280c\": rpc error: code = NotFound desc = could not find container \"adab87174c517cf155db0afc4bc2d2335c6d1077e35d5d1dc6eb43b050ae280c\": container with ID starting with adab87174c517cf155db0afc4bc2d2335c6d1077e35d5d1dc6eb43b050ae280c not found: ID does not exist"
Oct 30 13:40:53.238641 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:53.238546    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"]
Oct 30 13:40:53.238928 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:53.238909    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9"]
Oct 30 13:40:53.257778 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:53.257735    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:53.283959 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:53.283926    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj"
Oct 30 13:40:53.358457 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:53.358423    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c724287-666b-47b4-8c6b-428a20ff1164-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c724287-666b-47b4-8c6b-428a20ff1164-plugins-config-volume\") pod \"3c724287-666b-47b4-8c6b-428a20ff1164\" (UID: \"3c724287-666b-47b4-8c6b-428a20ff1164\") "
Oct 30 13:40:53.358672 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:53.358543    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c724287-666b-47b4-8c6b-428a20ff1164-kube-api-access-ntkp2\" (UniqueName: \"kubernetes.io/projected/3c724287-666b-47b4-8c6b-428a20ff1164-kube-api-access-ntkp2\") pod \"3c724287-666b-47b4-8c6b-428a20ff1164\" (UID: \"3c724287-666b-47b4-8c6b-428a20ff1164\") "
Oct 30 13:40:53.358907 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:53.358881    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c724287-666b-47b4-8c6b-428a20ff1164-plugins-config-volume" pod "3c724287-666b-47b4-8c6b-428a20ff1164" (UID: "3c724287-666b-47b4-8c6b-428a20ff1164"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:53.360852 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:53.360816    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c724287-666b-47b4-8c6b-428a20ff1164-kube-api-access-ntkp2" pod "3c724287-666b-47b4-8c6b-428a20ff1164" (UID: "3c724287-666b-47b4-8c6b-428a20ff1164"). InnerVolumeSpecName "kube-api-access-ntkp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:53.459965 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:53.459912    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c724287-666b-47b4-8c6b-428a20ff1164-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:53.459965 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:53.459957    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntkp2\" (UniqueName: \"kubernetes.io/projected/3c724287-666b-47b4-8c6b-428a20ff1164-kube-api-access-ntkp2\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:53.498276 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:53.498128    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:53.623128 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:53.623081    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:53.639324 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:53.639287    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:53.985520 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:53.985483    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:54.159892 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:54.159856    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj" event={"ID":"3c724287-666b-47b4-8c6b-428a20ff1164","Type":"ContainerDied","Data":"0dfed796d08d67214b5443a0770e65b76bde330b88b9efbb93f6274065eb536d"}
Oct 30 13:40:54.159892 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:54.159884    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj"
Oct 30 13:40:54.160141 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:54.159918    4544 scope.go:122] "RemoveContainer" containerID="a87305396d2376abcd260fc0f333ce7c51c54539a32566ff5badd859f36702a7"
Oct 30 13:40:54.161213 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:54.161199    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:40:54.214643 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:54.214605    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj"]
Oct 30 13:40:54.217364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:54.217331    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj"]
Oct 30 13:40:54.847448 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:54.847396    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6" podUID="99dd854e-d12d-4624-9c56-eeea387987cd" containerName="payload-pre-processing" containerID="cri-o://1f1adfe3df9674ee53b9538b877f5db8194e7bd59b3c6cff1a3942057dae9f43" gracePeriod=25
Oct 30 13:40:55.087269 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:55.087235    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c724287-666b-47b4-8c6b-428a20ff1164" path="/var/lib/kubelet/pods/3c724287-666b-47b4-8c6b-428a20ff1164/volumes"
Oct 30 13:40:55.087857 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:55.087835    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f4af86d-c526-4365-b56d-32a7f46247bf" path="/var/lib/kubelet/pods/9f4af86d-c526-4365-b56d-32a7f46247bf/volumes"
Oct 30 13:40:55.111142 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:55.111067    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6"
Oct 30 13:40:55.170441 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:55.170404    4544 generic.go:356] "Generic (PLEG): container finished" podID="99dd854e-d12d-4624-9c56-eeea387987cd" containerID="1f1adfe3df9674ee53b9538b877f5db8194e7bd59b3c6cff1a3942057dae9f43" exitCode=0
Oct 30 13:40:55.170633 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:55.170448    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6"
Oct 30 13:40:55.170633 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:55.170489    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6" event={"ID":"99dd854e-d12d-4624-9c56-eeea387987cd","Type":"ContainerDied","Data":"1f1adfe3df9674ee53b9538b877f5db8194e7bd59b3c6cff1a3942057dae9f43"}
Oct 30 13:40:55.170633 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:55.170554    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6" event={"ID":"99dd854e-d12d-4624-9c56-eeea387987cd","Type":"ContainerDied","Data":"78e93642639d5d0a9b04c284a5fa59d107cb40c917214a104d7ac7a0fb87a68b"}
Oct 30 13:40:55.170633 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:55.170588    4544 scope.go:122] "RemoveContainer" containerID="1f1adfe3df9674ee53b9538b877f5db8194e7bd59b3c6cff1a3942057dae9f43"
Oct 30 13:40:55.181773 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:55.181737    4544 scope.go:122] "RemoveContainer" containerID="1f1adfe3df9674ee53b9538b877f5db8194e7bd59b3c6cff1a3942057dae9f43"
Oct 30 13:40:55.182139 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:40:55.182118    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f1adfe3df9674ee53b9538b877f5db8194e7bd59b3c6cff1a3942057dae9f43\": container with ID starting with 1f1adfe3df9674ee53b9538b877f5db8194e7bd59b3c6cff1a3942057dae9f43 not found: ID does not exist" containerID="1f1adfe3df9674ee53b9538b877f5db8194e7bd59b3c6cff1a3942057dae9f43"
Oct 30 13:40:55.182192 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:55.182150    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f1adfe3df9674ee53b9538b877f5db8194e7bd59b3c6cff1a3942057dae9f43"} err="failed to get container status \"1f1adfe3df9674ee53b9538b877f5db8194e7bd59b3c6cff1a3942057dae9f43\": rpc error: code = NotFound desc = could not find container \"1f1adfe3df9674ee53b9538b877f5db8194e7bd59b3c6cff1a3942057dae9f43\": container with ID starting with 1f1adfe3df9674ee53b9538b877f5db8194e7bd59b3c6cff1a3942057dae9f43 not found: ID does not exist"
Oct 30 13:40:55.184981 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:55.184960    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99dd854e-d12d-4624-9c56-eeea387987cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99dd854e-d12d-4624-9c56-eeea387987cd-plugins-config-volume\") pod \"99dd854e-d12d-4624-9c56-eeea387987cd\" (UID: \"99dd854e-d12d-4624-9c56-eeea387987cd\") "
Oct 30 13:40:55.185170 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:55.185155    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99dd854e-d12d-4624-9c56-eeea387987cd-kube-api-access-j4pc8\" (UniqueName: \"kubernetes.io/projected/99dd854e-d12d-4624-9c56-eeea387987cd-kube-api-access-j4pc8\") pod \"99dd854e-d12d-4624-9c56-eeea387987cd\" (UID: \"99dd854e-d12d-4624-9c56-eeea387987cd\") "
Oct 30 13:40:55.185336 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:55.185311    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99dd854e-d12d-4624-9c56-eeea387987cd-plugins-config-volume" pod "99dd854e-d12d-4624-9c56-eeea387987cd" (UID: "99dd854e-d12d-4624-9c56-eeea387987cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:55.185560 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:55.185533    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99dd854e-d12d-4624-9c56-eeea387987cd-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:55.187405 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:55.187380    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99dd854e-d12d-4624-9c56-eeea387987cd-kube-api-access-j4pc8" pod "99dd854e-d12d-4624-9c56-eeea387987cd" (UID: "99dd854e-d12d-4624-9c56-eeea387987cd"). InnerVolumeSpecName "kube-api-access-j4pc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:55.286972 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:55.286929    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4pc8\" (UniqueName: \"kubernetes.io/projected/99dd854e-d12d-4624-9c56-eeea387987cd-kube-api-access-j4pc8\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:40:55.369191 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:55.369083    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:55.519458 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:55.519414    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6"]
Oct 30 13:40:55.523590 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:55.523554    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6"]
Oct 30 13:40:56.989529 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:56.989496    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:57.105011 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:57.103898    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99dd854e-d12d-4624-9c56-eeea387987cd" path="/var/lib/kubelet/pods/99dd854e-d12d-4624-9c56-eeea387987cd/volumes"
Oct 30 13:40:57.413587 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:57.413550    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:57.583849 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:57.583814    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:57.722682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:57.722592    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:57.924884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:57.924848    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:58.521753 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:58.521672    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:58.959934 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:58.959896    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:59.006424 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:40:59.006382    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:40:59.477536 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:59.477492    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:59.605490 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:59.605446    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:40:59.717297 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:40:59.717257    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:00.126447 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:00.126415    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:01.052027 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:01.051989    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:01.226105 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:01.226072    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:01.618699 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:01.618660    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-87e742-f67665954-qqm68"]
Oct 30 13:41:01.619282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:01.619257    4544 cpu_manager.go:414] "containerMap: removing container" podUID="3e781781-4614-4cb3-84c9-987ca6719f12" containerName="revoke-keys"
Oct 30 13:41:01.619282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:01.619277    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e781781-4614-4cb3-84c9-987ca6719f12" containerName="revoke-keys"
Oct 30 13:41:01.619484 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:01.619290    4544 cpu_manager.go:414] "containerMap: removing container" podUID="6db19ff5-f476-4582-8153-b2b750f45032" containerName="maas-api"
Oct 30 13:41:01.619484 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:01.619298    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6db19ff5-f476-4582-8153-b2b750f45032" containerName="maas-api"
Oct 30 13:41:01.619484 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:01.619313    4544 cpu_manager.go:414] "containerMap: removing container" podUID="9f4af86d-c526-4365-b56d-32a7f46247bf" containerName="istio-proxy"
Oct 30 13:41:01.619484 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:01.619323    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f4af86d-c526-4365-b56d-32a7f46247bf" containerName="istio-proxy"
Oct 30 13:41:01.619484 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:01.619338    4544 cpu_manager.go:414] "containerMap: removing container" podUID="3c724287-666b-47b4-8c6b-428a20ff1164" containerName="payload-pre-processing"
Oct 30 13:41:01.619484 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:01.619345    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c724287-666b-47b4-8c6b-428a20ff1164" containerName="payload-pre-processing"
Oct 30 13:41:01.619484 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:01.619367    4544 cpu_manager.go:414] "containerMap: removing container" podUID="99dd854e-d12d-4624-9c56-eeea387987cd" containerName="payload-pre-processing"
Oct 30 13:41:01.619484 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:01.619372    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99dd854e-d12d-4624-9c56-eeea387987cd" containerName="payload-pre-processing"
Oct 30 13:41:01.619484 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:01.619378    4544 cpu_manager.go:414] "containerMap: removing container" podUID="565c6710-ac33-437a-b2cc-5572a325523d" containerName="istio-proxy"
Oct 30 13:41:01.619484 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:01.619383    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="565c6710-ac33-437a-b2cc-5572a325523d" containerName="istio-proxy"
Oct 30 13:41:01.619884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:01.619500    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="99dd854e-d12d-4624-9c56-eeea387987cd" containerName="payload-pre-processing"
Oct 30 13:41:01.619884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:01.619509    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="565c6710-ac33-437a-b2cc-5572a325523d" containerName="istio-proxy"
Oct 30 13:41:01.619884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:01.619519    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e781781-4614-4cb3-84c9-987ca6719f12" containerName="revoke-keys"
Oct 30 13:41:01.619884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:01.619525    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="6db19ff5-f476-4582-8153-b2b750f45032" containerName="maas-api"
Oct 30 13:41:01.619884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:01.619531    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f4af86d-c526-4365-b56d-32a7f46247bf" containerName="istio-proxy"
Oct 30 13:41:01.619884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:01.619536    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c724287-666b-47b4-8c6b-428a20ff1164" containerName="payload-pre-processing"
Oct 30 13:41:01.620125 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:01.620007    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg" podUID="3c07693a-fc68-4472-a5d0-d7d7ba02bdc2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:41:01.624371 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:01.624354    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-f67665954-qqm68"
Oct 30 13:41:01.654286 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:01.654253    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-87e742\""
Oct 30 13:41:01.654473 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:01.654346    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-87e742-dockercfg-dmt2h\""
Oct 30 13:41:01.654857 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:01.654834    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-87e742-f67665954-qqm68"]
Oct 30 13:41:01.767197 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:01.767152    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r774\" (UniqueName: \"kubernetes.io/projected/bee4a067-c9b7-4b58-8015-88c7d6673b3b-kube-api-access-5r774\") pod \"payload-processing-e2e-ipp-b-87e742-f67665954-qqm68\" (UID: \"bee4a067-c9b7-4b58-8015-88c7d6673b3b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-f67665954-qqm68"
Oct 30 13:41:01.767435 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:01.767252    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bee4a067-c9b7-4b58-8015-88c7d6673b3b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-87e742-f67665954-qqm68\" (UID: \"bee4a067-c9b7-4b58-8015-88c7d6673b3b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-f67665954-qqm68"
Oct 30 13:41:01.868751 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:01.868674    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r774\" (UniqueName: \"kubernetes.io/projected/bee4a067-c9b7-4b58-8015-88c7d6673b3b-kube-api-access-5r774\") pod \"payload-processing-e2e-ipp-b-87e742-f67665954-qqm68\" (UID: \"bee4a067-c9b7-4b58-8015-88c7d6673b3b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-f67665954-qqm68"
Oct 30 13:41:01.868751 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:01.868739    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bee4a067-c9b7-4b58-8015-88c7d6673b3b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-87e742-f67665954-qqm68\" (UID: \"bee4a067-c9b7-4b58-8015-88c7d6673b3b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-f67665954-qqm68"
Oct 30 13:41:01.869528 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:01.869508    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bee4a067-c9b7-4b58-8015-88c7d6673b3b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-87e742-f67665954-qqm68\" (UID: \"bee4a067-c9b7-4b58-8015-88c7d6673b3b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-f67665954-qqm68"
Oct 30 13:41:01.913003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:01.912970    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r774\" (UniqueName: \"kubernetes.io/projected/bee4a067-c9b7-4b58-8015-88c7d6673b3b-kube-api-access-5r774\") pod \"payload-processing-e2e-ipp-b-87e742-f67665954-qqm68\" (UID: \"bee4a067-c9b7-4b58-8015-88c7d6673b3b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-f67665954-qqm68"
Oct 30 13:41:01.938917 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:01.938882    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-f67665954-qqm68"
Oct 30 13:41:01.997079 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:01.997043    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a25d83d-7pm25"]
Oct 30 13:41:02.011778 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:02.011748    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a25d83d-7pm25"
Oct 30 13:41:02.030839 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:02.027079    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a25d83d-7pm25"]
Oct 30 13:41:02.045970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:02.045941    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:41:02.046150 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:02.046057    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:41:02.171314 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:02.171207    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/364bad41-bbc0-47e1-9f68-a66e5c73be65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a25d83d-7pm25\" (UID: \"364bad41-bbc0-47e1-9f68-a66e5c73be65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a25d83d-7pm25"
Oct 30 13:41:02.171314 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:02.171302    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcf8p\" (UniqueName: \"kubernetes.io/projected/364bad41-bbc0-47e1-9f68-a66e5c73be65-kube-api-access-gcf8p\") pod \"maas-api-revoke-sub-8a25d83d-7pm25\" (UID: \"364bad41-bbc0-47e1-9f68-a66e5c73be65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a25d83d-7pm25"
Oct 30 13:41:02.171833 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:02.171587    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/364bad41-bbc0-47e1-9f68-a66e5c73be65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a25d83d-7pm25\" (UID: \"364bad41-bbc0-47e1-9f68-a66e5c73be65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a25d83d-7pm25"
Oct 30 13:41:02.253749 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:02.253713    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-87e742-f67665954-qqm68"]
Oct 30 13:41:02.257865 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:41:02.257827    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee4a067_c9b7_4b58_8015_88c7d6673b3b.slice/crio-0895dba52a1add59a6b94acd3d8a682e74d373f3cf56d6e050b4ec7d62dd34f9 WatchSource:0}: Error finding container 0895dba52a1add59a6b94acd3d8a682e74d373f3cf56d6e050b4ec7d62dd34f9: Status 404 returned error can't find the container with id 0895dba52a1add59a6b94acd3d8a682e74d373f3cf56d6e050b4ec7d62dd34f9
Oct 30 13:41:02.272750 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:02.272726    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/364bad41-bbc0-47e1-9f68-a66e5c73be65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a25d83d-7pm25\" (UID: \"364bad41-bbc0-47e1-9f68-a66e5c73be65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a25d83d-7pm25"
Oct 30 13:41:02.272874 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:02.272854    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/364bad41-bbc0-47e1-9f68-a66e5c73be65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a25d83d-7pm25\" (UID: \"364bad41-bbc0-47e1-9f68-a66e5c73be65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a25d83d-7pm25"
Oct 30 13:41:02.272918 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:02.272905    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcf8p\" (UniqueName: \"kubernetes.io/projected/364bad41-bbc0-47e1-9f68-a66e5c73be65-kube-api-access-gcf8p\") pod \"maas-api-revoke-sub-8a25d83d-7pm25\" (UID: \"364bad41-bbc0-47e1-9f68-a66e5c73be65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a25d83d-7pm25"
Oct 30 13:41:02.273592 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:02.273567    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/364bad41-bbc0-47e1-9f68-a66e5c73be65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a25d83d-7pm25\" (UID: \"364bad41-bbc0-47e1-9f68-a66e5c73be65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a25d83d-7pm25"
Oct 30 13:41:02.275149 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:02.275129    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/364bad41-bbc0-47e1-9f68-a66e5c73be65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a25d83d-7pm25\" (UID: \"364bad41-bbc0-47e1-9f68-a66e5c73be65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a25d83d-7pm25"
Oct 30 13:41:02.307291 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:02.307264    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcf8p\" (UniqueName: \"kubernetes.io/projected/364bad41-bbc0-47e1-9f68-a66e5c73be65-kube-api-access-gcf8p\") pod \"maas-api-revoke-sub-8a25d83d-7pm25\" (UID: \"364bad41-bbc0-47e1-9f68-a66e5c73be65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a25d83d-7pm25"
Oct 30 13:41:02.340309 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:02.340281    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a25d83d-7pm25"
Oct 30 13:41:02.480311 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:02.480237    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:02.649444 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:02.649409    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a25d83d-7pm25"]
Oct 30 13:41:02.652315 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:41:02.652286    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod364bad41_bbc0_47e1_9f68_a66e5c73be65.slice/crio-cf4d520c3eae2b2975af7f60d2acbf520a65d95bbe27909247f011260ce86731 WatchSource:0}: Error finding container cf4d520c3eae2b2975af7f60d2acbf520a65d95bbe27909247f011260ce86731: Status 404 returned error can't find the container with id cf4d520c3eae2b2975af7f60d2acbf520a65d95bbe27909247f011260ce86731
Oct 30 13:41:03.229314 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:03.229201    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a25d83d-7pm25" event={"ID":"364bad41-bbc0-47e1-9f68-a66e5c73be65","Type":"ContainerStarted","Data":"26091745f2ca1da4f54bcb4d5edac6b4238dd72665a941e14a425fa662133a11"}
Oct 30 13:41:03.229314 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:03.229278    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a25d83d-7pm25" event={"ID":"364bad41-bbc0-47e1-9f68-a66e5c73be65","Type":"ContainerStarted","Data":"cf4d520c3eae2b2975af7f60d2acbf520a65d95bbe27909247f011260ce86731"}
Oct 30 13:41:03.230968 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:03.230944    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-f67665954-qqm68" event={"ID":"bee4a067-c9b7-4b58-8015-88c7d6673b3b","Type":"ContainerStarted","Data":"f4e40708263a35e91160c46cd7ce51eef0d094448ce5eba04a910c8fc7a6e844"}
Oct 30 13:41:03.231062 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:03.230984    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-f67665954-qqm68" event={"ID":"bee4a067-c9b7-4b58-8015-88c7d6673b3b","Type":"ContainerStarted","Data":"0895dba52a1add59a6b94acd3d8a682e74d373f3cf56d6e050b4ec7d62dd34f9"}
Oct 30 13:41:03.231121 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:03.231070    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-f67665954-qqm68"
Oct 30 13:41:03.271670 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:03.271607    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a25d83d-7pm25" podStartSLOduration=2.271590314 podStartE2EDuration="2.271590314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:41:03.269681931 +0000 UTC m=+3358.316815884" watchObservedRunningTime="2026-10-30 13:41:03.271590314 +0000 UTC m=+3358.318724268"
Oct 30 13:41:03.320179 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:03.320116    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-f67665954-qqm68" podStartSLOduration=2.320104015 podStartE2EDuration="2.320104015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:41:03.319521413 +0000 UTC m=+3358.366655367" watchObservedRunningTime="2026-10-30 13:41:03.320104015 +0000 UTC m=+3358.367237968"
Oct 30 13:41:04.238647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:04.238603    4544 generic.go:356] "Generic (PLEG): container finished" podID="364bad41-bbc0-47e1-9f68-a66e5c73be65" containerID="26091745f2ca1da4f54bcb4d5edac6b4238dd72665a941e14a425fa662133a11" exitCode=0
Oct 30 13:41:04.239112 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:04.238694    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a25d83d-7pm25" event={"ID":"364bad41-bbc0-47e1-9f68-a66e5c73be65","Type":"ContainerDied","Data":"26091745f2ca1da4f54bcb4d5edac6b4238dd72665a941e14a425fa662133a11"}
Oct 30 13:41:04.396824 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:04.396785    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:05.400894 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:05.400861    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a25d83d-7pm25"
Oct 30 13:41:05.508516 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:05.508419    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/364bad41-bbc0-47e1-9f68-a66e5c73be65-kube-api-access-gcf8p\" (UniqueName: \"kubernetes.io/projected/364bad41-bbc0-47e1-9f68-a66e5c73be65-kube-api-access-gcf8p\") pod \"364bad41-bbc0-47e1-9f68-a66e5c73be65\" (UID: \"364bad41-bbc0-47e1-9f68-a66e5c73be65\") "
Oct 30 13:41:05.508516 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:05.508468    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/364bad41-bbc0-47e1-9f68-a66e5c73be65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/364bad41-bbc0-47e1-9f68-a66e5c73be65-maas-api-xks-ca\") pod \"364bad41-bbc0-47e1-9f68-a66e5c73be65\" (UID: \"364bad41-bbc0-47e1-9f68-a66e5c73be65\") "
Oct 30 13:41:05.508516 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:05.508502    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/364bad41-bbc0-47e1-9f68-a66e5c73be65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/364bad41-bbc0-47e1-9f68-a66e5c73be65-maas-api-openshift-ca\") pod \"364bad41-bbc0-47e1-9f68-a66e5c73be65\" (UID: \"364bad41-bbc0-47e1-9f68-a66e5c73be65\") "
Oct 30 13:41:05.508960 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:05.508937    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/364bad41-bbc0-47e1-9f68-a66e5c73be65-maas-api-openshift-ca" pod "364bad41-bbc0-47e1-9f68-a66e5c73be65" (UID: "364bad41-bbc0-47e1-9f68-a66e5c73be65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:05.510736 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:05.510707    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/364bad41-bbc0-47e1-9f68-a66e5c73be65-kube-api-access-gcf8p" pod "364bad41-bbc0-47e1-9f68-a66e5c73be65" (UID: "364bad41-bbc0-47e1-9f68-a66e5c73be65"). InnerVolumeSpecName "kube-api-access-gcf8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:05.511144 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:05.511119    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/364bad41-bbc0-47e1-9f68-a66e5c73be65-maas-api-xks-ca" pod "364bad41-bbc0-47e1-9f68-a66e5c73be65" (UID: "364bad41-bbc0-47e1-9f68-a66e5c73be65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:41:05.610477 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:05.610420    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcf8p\" (UniqueName: \"kubernetes.io/projected/364bad41-bbc0-47e1-9f68-a66e5c73be65-kube-api-access-gcf8p\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:41:05.610477 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:05.610466    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/364bad41-bbc0-47e1-9f68-a66e5c73be65-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:41:05.610477 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:05.610477    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/364bad41-bbc0-47e1-9f68-a66e5c73be65-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:41:06.255061 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:06.255032    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a25d83d-7pm25"
Oct 30 13:41:06.255272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:06.255068    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a25d83d-7pm25" event={"ID":"364bad41-bbc0-47e1-9f68-a66e5c73be65","Type":"ContainerDied","Data":"cf4d520c3eae2b2975af7f60d2acbf520a65d95bbe27909247f011260ce86731"}
Oct 30 13:41:06.255272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:06.255118    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf4d520c3eae2b2975af7f60d2acbf520a65d95bbe27909247f011260ce86731"
Oct 30 13:41:06.347746 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:06.347707    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:06.712403 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:06.712357    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:06.811029 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:06.810995    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg"
Oct 30 13:41:06.902619 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:06.902585    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:06.925127 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:06.925091    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c07693a-fc68-4472-a5d0-d7d7ba02bdc2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c07693a-fc68-4472-a5d0-d7d7ba02bdc2-plugins-config-volume\") pod \"3c07693a-fc68-4472-a5d0-d7d7ba02bdc2\" (UID: \"3c07693a-fc68-4472-a5d0-d7d7ba02bdc2\") "
Oct 30 13:41:06.925302 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:06.925256    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c07693a-fc68-4472-a5d0-d7d7ba02bdc2-kube-api-access-cl4z9\" (UniqueName: \"kubernetes.io/projected/3c07693a-fc68-4472-a5d0-d7d7ba02bdc2-kube-api-access-cl4z9\") pod \"3c07693a-fc68-4472-a5d0-d7d7ba02bdc2\" (UID: \"3c07693a-fc68-4472-a5d0-d7d7ba02bdc2\") "
Oct 30 13:41:06.925553 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:06.925526    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c07693a-fc68-4472-a5d0-d7d7ba02bdc2-plugins-config-volume" pod "3c07693a-fc68-4472-a5d0-d7d7ba02bdc2" (UID: "3c07693a-fc68-4472-a5d0-d7d7ba02bdc2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:06.925702 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:06.925683    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c07693a-fc68-4472-a5d0-d7d7ba02bdc2-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:41:06.927514 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:06.927485    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c07693a-fc68-4472-a5d0-d7d7ba02bdc2-kube-api-access-cl4z9" pod "3c07693a-fc68-4472-a5d0-d7d7ba02bdc2" (UID: "3c07693a-fc68-4472-a5d0-d7d7ba02bdc2"). InnerVolumeSpecName "kube-api-access-cl4z9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:07.026894 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:07.026852    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl4z9\" (UniqueName: \"kubernetes.io/projected/3c07693a-fc68-4472-a5d0-d7d7ba02bdc2-kube-api-access-cl4z9\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:41:07.069982 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:07.069897    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:07.273238 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:07.270628    4544 generic.go:356] "Generic (PLEG): container finished" podID="3c07693a-fc68-4472-a5d0-d7d7ba02bdc2" containerID="7fb4b4efc97765b6fe2f8085ac576ada6731188d1a4f6d6c353a1f6be7c83a68" exitCode=0
Oct 30 13:41:07.273238 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:07.270699    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg"
Oct 30 13:41:07.273238 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:07.270702    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg" event={"ID":"3c07693a-fc68-4472-a5d0-d7d7ba02bdc2","Type":"ContainerDied","Data":"7fb4b4efc97765b6fe2f8085ac576ada6731188d1a4f6d6c353a1f6be7c83a68"}
Oct 30 13:41:07.273238 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:07.270830    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg" event={"ID":"3c07693a-fc68-4472-a5d0-d7d7ba02bdc2","Type":"ContainerDied","Data":"f5cd851ed43674df3715e9e6532e0e371d8679503020e0a67cd4d47cfa506454"}
Oct 30 13:41:07.273238 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:07.270871    4544 scope.go:122] "RemoveContainer" containerID="7fb4b4efc97765b6fe2f8085ac576ada6731188d1a4f6d6c353a1f6be7c83a68"
Oct 30 13:41:07.296042 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:07.293391    4544 scope.go:122] "RemoveContainer" containerID="7fb4b4efc97765b6fe2f8085ac576ada6731188d1a4f6d6c353a1f6be7c83a68"
Oct 30 13:41:07.298758 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:41:07.298716    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fb4b4efc97765b6fe2f8085ac576ada6731188d1a4f6d6c353a1f6be7c83a68\": container with ID starting with 7fb4b4efc97765b6fe2f8085ac576ada6731188d1a4f6d6c353a1f6be7c83a68 not found: ID does not exist" containerID="7fb4b4efc97765b6fe2f8085ac576ada6731188d1a4f6d6c353a1f6be7c83a68"
Oct 30 13:41:07.298954 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:07.298769    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fb4b4efc97765b6fe2f8085ac576ada6731188d1a4f6d6c353a1f6be7c83a68"} err="failed to get container status \"7fb4b4efc97765b6fe2f8085ac576ada6731188d1a4f6d6c353a1f6be7c83a68\": rpc error: code = NotFound desc = could not find container \"7fb4b4efc97765b6fe2f8085ac576ada6731188d1a4f6d6c353a1f6be7c83a68\": container with ID starting with 7fb4b4efc97765b6fe2f8085ac576ada6731188d1a4f6d6c353a1f6be7c83a68 not found: ID does not exist"
Oct 30 13:41:07.335346 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:07.335311    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg"]
Oct 30 13:41:07.338024 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:07.337972    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg"]
Oct 30 13:41:07.366178 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:07.366135    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:08.773496 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:08.773435    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:09.084962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:09.084927    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c07693a-fc68-4472-a5d0-d7d7ba02bdc2" path="/var/lib/kubelet/pods/3c07693a-fc68-4472-a5d0-d7d7ba02bdc2/volumes"
Oct 30 13:41:10.453280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:10.453241    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:12.199687 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:12.199650    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:14.241408 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:14.241379    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-f67665954-qqm68"
Oct 30 13:41:14.278933 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:14.278891    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:14.458901 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:14.458864    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:14.886125 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:14.886089    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:15.112788 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:15.112753    4544 scope.go:122] "RemoveContainer" containerID="e233384f8c5723049834601be4eb32fc061ba41351b790b423f7e7a4bf01f7b7"
Oct 30 13:41:15.123651 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:15.123622    4544 scope.go:122] "RemoveContainer" containerID="27911f49001a65d44895b932c5f5b68092f9fa23b3867f2c3e2260aeb48e8eda"
Oct 30 13:41:15.133589 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:15.133565    4544 scope.go:122] "RemoveContainer" containerID="18be4a2509897282841202e22ae92eafe6b361bb348b776a16223ba6e0634ff7"
Oct 30 13:41:15.143426 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:15.143404    4544 scope.go:122] "RemoveContainer" containerID="5a6c7f38649a5178d037c95c7920401ea42dc0376a50a3df90a6ea2f48edacb4"
Oct 30 13:41:15.654800 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:15.654761    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:16.683805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:16.683732    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:18.747261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:18.747205    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:18.878566 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:18.878530    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:19.338621 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:19.338584    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:21.482787 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:21.482727    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:22.160465 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:22.160431    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:22.288127 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:22.288097    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:22.492827 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:22.492706    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:22.860691 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:22.860658    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:24.469449 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:24.469412    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:24.612789 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:24.612718    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:26.404621 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:26.404589    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:28.675760 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:28.675721    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:28.813449 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:28.813404    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:29.010976 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:41:29.010875    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:41:29.157442 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:29.157398    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:29.329114 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:29.329081    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:29.499485 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:29.499442    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"]
Oct 30 13:41:29.499714 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:29.499682    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt" podUID="6bdb6b8f-2389-407e-a9e7-a98907722803" containerName="istio-proxy" containerID="cri-o://8bb90c57c4d8bbe1cfc564e75719afa4077ce0c9589faf4f635e001c421bf2f8" gracePeriod=30
Oct 30 13:41:29.537453 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:29.537416    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:30.280807 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:30.280764    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:30.504856 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:30.504824    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:30.626114 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:30.626080    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:30.996540 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:30.996429    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:31.127357 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:31.127322    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:31.665955 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:31.665921    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:33.231319 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:33.231268    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:34.770427 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:34.770348    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:41:34.844874 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:34.844839    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bdb6b8f-2389-407e-a9e7-a98907722803-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bdb6b8f-2389-407e-a9e7-a98907722803-istiod-ca-cert\") pod \"6bdb6b8f-2389-407e-a9e7-a98907722803\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") "
Oct 30 13:41:34.845109 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:34.844905    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bdb6b8f-2389-407e-a9e7-a98907722803-istio-token\" (UniqueName: \"kubernetes.io/projected/6bdb6b8f-2389-407e-a9e7-a98907722803-istio-token\") pod \"6bdb6b8f-2389-407e-a9e7-a98907722803\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") "
Oct 30 13:41:34.845109 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:34.844934    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6bdb6b8f-2389-407e-a9e7-a98907722803-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bdb6b8f-2389-407e-a9e7-a98907722803-istio-podinfo\") pod \"6bdb6b8f-2389-407e-a9e7-a98907722803\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") "
Oct 30 13:41:34.845109 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:34.844985    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bdb6b8f-2389-407e-a9e7-a98907722803-kube-api-access-jdrwd\" (UniqueName: \"kubernetes.io/projected/6bdb6b8f-2389-407e-a9e7-a98907722803-kube-api-access-jdrwd\") pod \"6bdb6b8f-2389-407e-a9e7-a98907722803\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") "
Oct 30 13:41:34.845109 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:34.845019    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-istio-data\") pod \"6bdb6b8f-2389-407e-a9e7-a98907722803\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") "
Oct 30 13:41:34.845109 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:34.845080    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-workload-certs\") pod \"6bdb6b8f-2389-407e-a9e7-a98907722803\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") "
Oct 30 13:41:34.845425 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:34.845164    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-workload-socket\") pod \"6bdb6b8f-2389-407e-a9e7-a98907722803\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") "
Oct 30 13:41:34.845425 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:34.845257    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-istio-envoy\") pod \"6bdb6b8f-2389-407e-a9e7-a98907722803\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") "
Oct 30 13:41:34.845425 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:34.845298    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-credential-socket\") pod \"6bdb6b8f-2389-407e-a9e7-a98907722803\" (UID: \"6bdb6b8f-2389-407e-a9e7-a98907722803\") "
Oct 30 13:41:34.845425 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:34.845309    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bdb6b8f-2389-407e-a9e7-a98907722803-istiod-ca-cert" pod "6bdb6b8f-2389-407e-a9e7-a98907722803" (UID: "6bdb6b8f-2389-407e-a9e7-a98907722803"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:34.846044 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:34.845669    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-workload-socket" pod "6bdb6b8f-2389-407e-a9e7-a98907722803" (UID: "6bdb6b8f-2389-407e-a9e7-a98907722803"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:34.846044 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:34.845686    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bdb6b8f-2389-407e-a9e7-a98907722803-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:41:34.846044 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:34.845797    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-istio-data" pod "6bdb6b8f-2389-407e-a9e7-a98907722803" (UID: "6bdb6b8f-2389-407e-a9e7-a98907722803"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:34.846044 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:34.845880    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-workload-certs" pod "6bdb6b8f-2389-407e-a9e7-a98907722803" (UID: "6bdb6b8f-2389-407e-a9e7-a98907722803"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:34.846044 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:34.845895    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-credential-socket" pod "6bdb6b8f-2389-407e-a9e7-a98907722803" (UID: "6bdb6b8f-2389-407e-a9e7-a98907722803"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:34.848247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:34.848122    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bdb6b8f-2389-407e-a9e7-a98907722803-istio-podinfo" pod "6bdb6b8f-2389-407e-a9e7-a98907722803" (UID: "6bdb6b8f-2389-407e-a9e7-a98907722803"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:41:34.848247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:34.848195    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-istio-envoy" pod "6bdb6b8f-2389-407e-a9e7-a98907722803" (UID: "6bdb6b8f-2389-407e-a9e7-a98907722803"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:34.848434 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:34.848261    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bdb6b8f-2389-407e-a9e7-a98907722803-kube-api-access-jdrwd" pod "6bdb6b8f-2389-407e-a9e7-a98907722803" (UID: "6bdb6b8f-2389-407e-a9e7-a98907722803"). InnerVolumeSpecName "kube-api-access-jdrwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:34.849637 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:34.849613    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bdb6b8f-2389-407e-a9e7-a98907722803-istio-token" pod "6bdb6b8f-2389-407e-a9e7-a98907722803" (UID: "6bdb6b8f-2389-407e-a9e7-a98907722803"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:34.946687 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:34.946649    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdrwd\" (UniqueName: \"kubernetes.io/projected/6bdb6b8f-2389-407e-a9e7-a98907722803-kube-api-access-jdrwd\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:41:34.946687 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:34.946684    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:41:34.946923 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:34.946699    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:41:34.946923 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:34.946714    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:41:34.946923 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:34.946726    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:41:34.946923 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:34.946739    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bdb6b8f-2389-407e-a9e7-a98907722803-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:41:34.946923 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:34.946752    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bdb6b8f-2389-407e-a9e7-a98907722803-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:41:34.946923 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:34.946767    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bdb6b8f-2389-407e-a9e7-a98907722803-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:41:34.994139 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:34.994095    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:35.463483 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:35.463381    4544 generic.go:356] "Generic (PLEG): container finished" podID="6bdb6b8f-2389-407e-a9e7-a98907722803" containerID="8bb90c57c4d8bbe1cfc564e75719afa4077ce0c9589faf4f635e001c421bf2f8" exitCode=0
Oct 30 13:41:35.463483 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:35.463441    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"
Oct 30 13:41:35.463483 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:35.463462    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt" event={"ID":"6bdb6b8f-2389-407e-a9e7-a98907722803","Type":"ContainerDied","Data":"8bb90c57c4d8bbe1cfc564e75719afa4077ce0c9589faf4f635e001c421bf2f8"}
Oct 30 13:41:35.463776 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:35.463522    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt" event={"ID":"6bdb6b8f-2389-407e-a9e7-a98907722803","Type":"ContainerDied","Data":"804e73fa9b9a665c74a422b7321789f13703bd3166c935e1bf4e1b1319354303"}
Oct 30 13:41:35.463776 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:35.463545    4544 scope.go:122] "RemoveContainer" containerID="8bb90c57c4d8bbe1cfc564e75719afa4077ce0c9589faf4f635e001c421bf2f8"
Oct 30 13:41:35.474148 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:35.474125    4544 scope.go:122] "RemoveContainer" containerID="8bb90c57c4d8bbe1cfc564e75719afa4077ce0c9589faf4f635e001c421bf2f8"
Oct 30 13:41:35.474561 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:41:35.474535    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bb90c57c4d8bbe1cfc564e75719afa4077ce0c9589faf4f635e001c421bf2f8\": container with ID starting with 8bb90c57c4d8bbe1cfc564e75719afa4077ce0c9589faf4f635e001c421bf2f8 not found: ID does not exist" containerID="8bb90c57c4d8bbe1cfc564e75719afa4077ce0c9589faf4f635e001c421bf2f8"
Oct 30 13:41:35.474665 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:35.474572    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bb90c57c4d8bbe1cfc564e75719afa4077ce0c9589faf4f635e001c421bf2f8"} err="failed to get container status \"8bb90c57c4d8bbe1cfc564e75719afa4077ce0c9589faf4f635e001c421bf2f8\": rpc error: code = NotFound desc = could not find container \"8bb90c57c4d8bbe1cfc564e75719afa4077ce0c9589faf4f635e001c421bf2f8\": container with ID starting with 8bb90c57c4d8bbe1cfc564e75719afa4077ce0c9589faf4f635e001c421bf2f8 not found: ID does not exist"
Oct 30 13:41:35.520265 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:35.520212    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"]
Oct 30 13:41:35.536975 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:35.536944    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt"]
Oct 30 13:41:36.731056 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:36.731018    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:37.086358 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:37.086326    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bdb6b8f-2389-407e-a9e7-a98907722803" path="/var/lib/kubelet/pods/6bdb6b8f-2389-407e-a9e7-a98907722803/volumes"
Oct 30 13:41:38.069066 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:38.069028    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:38.254967 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:38.254933    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:38.577095 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:38.577053    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:39.276433 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:39.276391    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:41.183640 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:41.183606    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:41.334872 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:41.334837    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:42.981409 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:42.981373    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:44.789370 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:44.789330    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:46.747914 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:46.747878    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:49.040019 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:49.039981    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:50.858385 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:50.858350    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:52.781899 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:52.781859    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:54.456294 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:54.456251    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:55.831635 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:55.831572    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc"]
Oct 30 13:41:55.832337 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:55.831892    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc" podUID="9e5e6bb1-c800-4a0a-a054-0f741c3d8933" containerName="maas-api" containerID="cri-o://2ce2f527b8da9d0621296f3a0472f2ed3b644025551f95dddf5640f0a78df227" gracePeriod=30
Oct 30 13:41:55.904369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:55.904331    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:56.015320 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.015274    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-87e742-f67665954-qqm68"]
Oct 30 13:41:56.015603 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.015554    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-f67665954-qqm68" podUID="bee4a067-c9b7-4b58-8015-88c7d6673b3b" containerName="payload-processing" containerID="cri-o://f4e40708263a35e91160c46cd7ce51eef0d094448ce5eba04a910c8fc7a6e844" gracePeriod=30
Oct 30 13:41:56.132291 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.132258    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:56.145849 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.145821    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc"
Oct 30 13:41:56.177007 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.176966    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e5e6bb1-c800-4a0a-a054-0f741c3d8933-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e5e6bb1-c800-4a0a-a054-0f741c3d8933-metrics-certs\") pod \"9e5e6bb1-c800-4a0a-a054-0f741c3d8933\" (UID: \"9e5e6bb1-c800-4a0a-a054-0f741c3d8933\") "
Oct 30 13:41:56.177292 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.177046    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e5e6bb1-c800-4a0a-a054-0f741c3d8933-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e5e6bb1-c800-4a0a-a054-0f741c3d8933-maas-api-tls\") pod \"9e5e6bb1-c800-4a0a-a054-0f741c3d8933\" (UID: \"9e5e6bb1-c800-4a0a-a054-0f741c3d8933\") "
Oct 30 13:41:56.177292 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.177104    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e5e6bb1-c800-4a0a-a054-0f741c3d8933-kube-api-access-gc88m\" (UniqueName: \"kubernetes.io/projected/9e5e6bb1-c800-4a0a-a054-0f741c3d8933-kube-api-access-gc88m\") pod \"9e5e6bb1-c800-4a0a-a054-0f741c3d8933\" (UID: \"9e5e6bb1-c800-4a0a-a054-0f741c3d8933\") "
Oct 30 13:41:56.177292 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.177144    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e5e6bb1-c800-4a0a-a054-0f741c3d8933-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5e6bb1-c800-4a0a-a054-0f741c3d8933-openshift-service-ca\") pod \"9e5e6bb1-c800-4a0a-a054-0f741c3d8933\" (UID: \"9e5e6bb1-c800-4a0a-a054-0f741c3d8933\") "
Oct 30 13:41:56.177712 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.177622    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e5e6bb1-c800-4a0a-a054-0f741c3d8933-openshift-service-ca" pod "9e5e6bb1-c800-4a0a-a054-0f741c3d8933" (UID: "9e5e6bb1-c800-4a0a-a054-0f741c3d8933"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:56.179508 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.179477    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5e6bb1-c800-4a0a-a054-0f741c3d8933-kube-api-access-gc88m" pod "9e5e6bb1-c800-4a0a-a054-0f741c3d8933" (UID: "9e5e6bb1-c800-4a0a-a054-0f741c3d8933"). InnerVolumeSpecName "kube-api-access-gc88m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:56.179767 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.179577    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e5e6bb1-c800-4a0a-a054-0f741c3d8933-metrics-certs" pod "9e5e6bb1-c800-4a0a-a054-0f741c3d8933" (UID: "9e5e6bb1-c800-4a0a-a054-0f741c3d8933"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:41:56.179767 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.179599    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e5e6bb1-c800-4a0a-a054-0f741c3d8933-maas-api-tls" pod "9e5e6bb1-c800-4a0a-a054-0f741c3d8933" (UID: "9e5e6bb1-c800-4a0a-a054-0f741c3d8933"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:41:56.240392 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.240312    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:56.264106 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.264081    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-f67665954-qqm68"
Oct 30 13:41:56.279033 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.279001    4544 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e5e6bb1-c800-4a0a-a054-0f741c3d8933-metrics-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:41:56.279033 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.279028    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e5e6bb1-c800-4a0a-a054-0f741c3d8933-maas-api-tls\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:41:56.279033 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.279038    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc88m\" (UniqueName: \"kubernetes.io/projected/9e5e6bb1-c800-4a0a-a054-0f741c3d8933-kube-api-access-gc88m\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:41:56.279322 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.279049    4544 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5e6bb1-c800-4a0a-a054-0f741c3d8933-openshift-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:41:56.379915 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.379823    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bee4a067-c9b7-4b58-8015-88c7d6673b3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bee4a067-c9b7-4b58-8015-88c7d6673b3b-plugins-config-volume\") pod \"bee4a067-c9b7-4b58-8015-88c7d6673b3b\" (UID: \"bee4a067-c9b7-4b58-8015-88c7d6673b3b\") "
Oct 30 13:41:56.380081 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.379993    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bee4a067-c9b7-4b58-8015-88c7d6673b3b-kube-api-access-5r774\" (UniqueName: \"kubernetes.io/projected/bee4a067-c9b7-4b58-8015-88c7d6673b3b-kube-api-access-5r774\") pod \"bee4a067-c9b7-4b58-8015-88c7d6673b3b\" (UID: \"bee4a067-c9b7-4b58-8015-88c7d6673b3b\") "
Oct 30 13:41:56.380301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.380277    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bee4a067-c9b7-4b58-8015-88c7d6673b3b-plugins-config-volume" pod "bee4a067-c9b7-4b58-8015-88c7d6673b3b" (UID: "bee4a067-c9b7-4b58-8015-88c7d6673b3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:56.380419 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.380401    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bee4a067-c9b7-4b58-8015-88c7d6673b3b-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:41:56.382386 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.382352    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee4a067-c9b7-4b58-8015-88c7d6673b3b-kube-api-access-5r774" pod "bee4a067-c9b7-4b58-8015-88c7d6673b3b" (UID: "bee4a067-c9b7-4b58-8015-88c7d6673b3b"). InnerVolumeSpecName "kube-api-access-5r774". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:56.481647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.481603    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r774\" (UniqueName: \"kubernetes.io/projected/bee4a067-c9b7-4b58-8015-88c7d6673b3b-kube-api-access-5r774\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:41:56.534174 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.534137    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:56.593432 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.593397    4544 generic.go:356] "Generic (PLEG): container finished" podID="bee4a067-c9b7-4b58-8015-88c7d6673b3b" containerID="f4e40708263a35e91160c46cd7ce51eef0d094448ce5eba04a910c8fc7a6e844" exitCode=0
Oct 30 13:41:56.593654 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.593446    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-f67665954-qqm68"
Oct 30 13:41:56.593654 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.593486    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-f67665954-qqm68" event={"ID":"bee4a067-c9b7-4b58-8015-88c7d6673b3b","Type":"ContainerDied","Data":"f4e40708263a35e91160c46cd7ce51eef0d094448ce5eba04a910c8fc7a6e844"}
Oct 30 13:41:56.593654 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.593546    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-f67665954-qqm68" event={"ID":"bee4a067-c9b7-4b58-8015-88c7d6673b3b","Type":"ContainerDied","Data":"0895dba52a1add59a6b94acd3d8a682e74d373f3cf56d6e050b4ec7d62dd34f9"}
Oct 30 13:41:56.593654 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.593578    4544 scope.go:122] "RemoveContainer" containerID="f4e40708263a35e91160c46cd7ce51eef0d094448ce5eba04a910c8fc7a6e844"
Oct 30 13:41:56.595157 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.595130    4544 generic.go:356] "Generic (PLEG): container finished" podID="9e5e6bb1-c800-4a0a-a054-0f741c3d8933" containerID="2ce2f527b8da9d0621296f3a0472f2ed3b644025551f95dddf5640f0a78df227" exitCode=0
Oct 30 13:41:56.595318 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.595179    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc" event={"ID":"9e5e6bb1-c800-4a0a-a054-0f741c3d8933","Type":"ContainerDied","Data":"2ce2f527b8da9d0621296f3a0472f2ed3b644025551f95dddf5640f0a78df227"}
Oct 30 13:41:56.595318 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.595190    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc"
Oct 30 13:41:56.595318 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.595255    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc" event={"ID":"9e5e6bb1-c800-4a0a-a054-0f741c3d8933","Type":"ContainerDied","Data":"ac40eacd4dd95a9fe3fe2cd0394c1a629c966d78a6efdf5fd75ab51127e55f87"}
Oct 30 13:41:56.606067 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.606045    4544 scope.go:122] "RemoveContainer" containerID="f4e40708263a35e91160c46cd7ce51eef0d094448ce5eba04a910c8fc7a6e844"
Oct 30 13:41:56.606805 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:41:56.606466    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4e40708263a35e91160c46cd7ce51eef0d094448ce5eba04a910c8fc7a6e844\": container with ID starting with f4e40708263a35e91160c46cd7ce51eef0d094448ce5eba04a910c8fc7a6e844 not found: ID does not exist" containerID="f4e40708263a35e91160c46cd7ce51eef0d094448ce5eba04a910c8fc7a6e844"
Oct 30 13:41:56.606805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.606504    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4e40708263a35e91160c46cd7ce51eef0d094448ce5eba04a910c8fc7a6e844"} err="failed to get container status \"f4e40708263a35e91160c46cd7ce51eef0d094448ce5eba04a910c8fc7a6e844\": rpc error: code = NotFound desc = could not find container \"f4e40708263a35e91160c46cd7ce51eef0d094448ce5eba04a910c8fc7a6e844\": container with ID starting with f4e40708263a35e91160c46cd7ce51eef0d094448ce5eba04a910c8fc7a6e844 not found: ID does not exist"
Oct 30 13:41:56.606805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.606532    4544 scope.go:122] "RemoveContainer" containerID="2ce2f527b8da9d0621296f3a0472f2ed3b644025551f95dddf5640f0a78df227"
Oct 30 13:41:56.616173 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.616150    4544 scope.go:122] "RemoveContainer" containerID="2ce2f527b8da9d0621296f3a0472f2ed3b644025551f95dddf5640f0a78df227"
Oct 30 13:41:56.616508 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:41:56.616472    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ce2f527b8da9d0621296f3a0472f2ed3b644025551f95dddf5640f0a78df227\": container with ID starting with 2ce2f527b8da9d0621296f3a0472f2ed3b644025551f95dddf5640f0a78df227 not found: ID does not exist" containerID="2ce2f527b8da9d0621296f3a0472f2ed3b644025551f95dddf5640f0a78df227"
Oct 30 13:41:56.616613 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.616518    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ce2f527b8da9d0621296f3a0472f2ed3b644025551f95dddf5640f0a78df227"} err="failed to get container status \"2ce2f527b8da9d0621296f3a0472f2ed3b644025551f95dddf5640f0a78df227\": rpc error: code = NotFound desc = could not find container \"2ce2f527b8da9d0621296f3a0472f2ed3b644025551f95dddf5640f0a78df227\": container with ID starting with 2ce2f527b8da9d0621296f3a0472f2ed3b644025551f95dddf5640f0a78df227 not found: ID does not exist"
Oct 30 13:41:56.655989 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.655902    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-87e742-f67665954-qqm68"]
Oct 30 13:41:56.678568 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.678526    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-87e742-f67665954-qqm68"]
Oct 30 13:41:56.687751 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.687716    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:56.723541 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.723503    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc"]
Oct 30 13:41:56.727275 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.727238    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-87e742-bbcf8cd97-2zsxc"]
Oct 30 13:41:56.831668 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:56.831631    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:57.085061 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:57.085030    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e5e6bb1-c800-4a0a-a054-0f741c3d8933" path="/var/lib/kubelet/pods/9e5e6bb1-c800-4a0a-a054-0f741c3d8933/volumes"
Oct 30 13:41:57.085485 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:57.085470    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bee4a067-c9b7-4b58-8015-88c7d6673b3b" path="/var/lib/kubelet/pods/bee4a067-c9b7-4b58-8015-88c7d6673b3b/volumes"
Oct 30 13:41:57.284948 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:57.284916    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:58.649289 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:58.649245    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:58.883444 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:41:58.883388    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:41:59.006809 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:41:59.006715    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:42:00.556027 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:00.555991    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:02.347208 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:02.347174    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:04.061752 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:04.061721    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:04.199508 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:04.199472    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:04.317740 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:04.317645    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:04.403335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:04.403293    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"]
Oct 30 13:42:04.403595 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:04.403561    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r" podUID="e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4" containerName="istio-proxy" containerID="cri-o://a9c0d02ae26601eb8f4bb6c3aaeb7dec41cfe823c4b498601d9f7edeca61e2b2" gracePeriod=30
Oct 30 13:42:04.460751 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:04.460714    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:04.998600 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:04.998559    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-51288b-bfvlg"]
Oct 30 13:42:04.999069 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:04.999054    4544 cpu_manager.go:414] "containerMap: removing container" podUID="6bdb6b8f-2389-407e-a9e7-a98907722803" containerName="istio-proxy"
Oct 30 13:42:04.999120 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:04.999070    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bdb6b8f-2389-407e-a9e7-a98907722803" containerName="istio-proxy"
Oct 30 13:42:04.999120 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:04.999093    4544 cpu_manager.go:414] "containerMap: removing container" podUID="bee4a067-c9b7-4b58-8015-88c7d6673b3b" containerName="payload-processing"
Oct 30 13:42:04.999120 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:04.999098    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bee4a067-c9b7-4b58-8015-88c7d6673b3b" containerName="payload-processing"
Oct 30 13:42:04.999120 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:04.999110    4544 cpu_manager.go:414] "containerMap: removing container" podUID="3c07693a-fc68-4472-a5d0-d7d7ba02bdc2" containerName="payload-processing"
Oct 30 13:42:04.999120 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:04.999116    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c07693a-fc68-4472-a5d0-d7d7ba02bdc2" containerName="payload-processing"
Oct 30 13:42:04.999335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:04.999128    4544 cpu_manager.go:414] "containerMap: removing container" podUID="364bad41-bbc0-47e1-9f68-a66e5c73be65" containerName="revoke-keys"
Oct 30 13:42:04.999335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:04.999133    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="364bad41-bbc0-47e1-9f68-a66e5c73be65" containerName="revoke-keys"
Oct 30 13:42:04.999335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:04.999140    4544 cpu_manager.go:414] "containerMap: removing container" podUID="9e5e6bb1-c800-4a0a-a054-0f741c3d8933" containerName="maas-api"
Oct 30 13:42:04.999335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:04.999146    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e5e6bb1-c800-4a0a-a054-0f741c3d8933" containerName="maas-api"
Oct 30 13:42:04.999335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:04.999256    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bdb6b8f-2389-407e-a9e7-a98907722803" containerName="istio-proxy"
Oct 30 13:42:04.999335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:04.999268    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="bee4a067-c9b7-4b58-8015-88c7d6673b3b" containerName="payload-processing"
Oct 30 13:42:04.999335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:04.999274    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c07693a-fc68-4472-a5d0-d7d7ba02bdc2" containerName="payload-processing"
Oct 30 13:42:04.999335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:04.999281    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="364bad41-bbc0-47e1-9f68-a66e5c73be65" containerName="revoke-keys"
Oct 30 13:42:04.999335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:04.999286    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e5e6bb1-c800-4a0a-a054-0f741c3d8933" containerName="maas-api"
Oct 30 13:42:05.014128 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:05.014093    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-51288b-bfvlg"
Oct 30 13:42:05.032400 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:05.029602    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-51288b-bfvlg"]
Oct 30 13:42:05.040452 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:05.038580    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:42:05.040452 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:05.038682    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:42:05.070711 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:05.070646    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b77bf44-eb3f-47dd-b143-10cb62c7de3c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-51288b-bfvlg\" (UID: \"0b77bf44-eb3f-47dd-b143-10cb62c7de3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-51288b-bfvlg"
Oct 30 13:42:05.071309 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:05.070835    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf95w\" (UniqueName: \"kubernetes.io/projected/0b77bf44-eb3f-47dd-b143-10cb62c7de3c-kube-api-access-cf95w\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-51288b-bfvlg\" (UID: \"0b77bf44-eb3f-47dd-b143-10cb62c7de3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-51288b-bfvlg"
Oct 30 13:42:05.071309 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:05.070970    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b77bf44-eb3f-47dd-b143-10cb62c7de3c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-51288b-bfvlg\" (UID: \"0b77bf44-eb3f-47dd-b143-10cb62c7de3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-51288b-bfvlg"
Oct 30 13:42:05.171747 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:05.171704    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b77bf44-eb3f-47dd-b143-10cb62c7de3c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-51288b-bfvlg\" (UID: \"0b77bf44-eb3f-47dd-b143-10cb62c7de3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-51288b-bfvlg"
Oct 30 13:42:05.171984 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:05.171812    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b77bf44-eb3f-47dd-b143-10cb62c7de3c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-51288b-bfvlg\" (UID: \"0b77bf44-eb3f-47dd-b143-10cb62c7de3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-51288b-bfvlg"
Oct 30 13:42:05.171984 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:05.171846    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf95w\" (UniqueName: \"kubernetes.io/projected/0b77bf44-eb3f-47dd-b143-10cb62c7de3c-kube-api-access-cf95w\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-51288b-bfvlg\" (UID: \"0b77bf44-eb3f-47dd-b143-10cb62c7de3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-51288b-bfvlg"
Oct 30 13:42:05.172519 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:05.172493    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b77bf44-eb3f-47dd-b143-10cb62c7de3c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-51288b-bfvlg\" (UID: \"0b77bf44-eb3f-47dd-b143-10cb62c7de3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-51288b-bfvlg"
Oct 30 13:42:05.187317 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:05.187283    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:42:05.194707 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:05.194677    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b77bf44-eb3f-47dd-b143-10cb62c7de3c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-51288b-bfvlg\" (UID: \"0b77bf44-eb3f-47dd-b143-10cb62c7de3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-51288b-bfvlg"
Oct 30 13:42:05.201115 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:05.201084    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf95w\" (UniqueName: \"kubernetes.io/projected/0b77bf44-eb3f-47dd-b143-10cb62c7de3c-kube-api-access-cf95w\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-51288b-bfvlg\" (UID: \"0b77bf44-eb3f-47dd-b143-10cb62c7de3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-51288b-bfvlg"
Oct 30 13:42:05.342159 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:05.342130    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:42:05.348264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:05.348216    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-51288b-bfvlg"
Oct 30 13:42:05.678968 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:05.678893    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-51288b-bfvlg"]
Oct 30 13:42:05.683360 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:42:05.683326    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b77bf44_eb3f_47dd_b143_10cb62c7de3c.slice/crio-c2088721b450c54122f2476c4e783ab23dacb9785172ef5d3dd2b11cbcb9b974 WatchSource:0}: Error finding container c2088721b450c54122f2476c4e783ab23dacb9785172ef5d3dd2b11cbcb9b974: Status 404 returned error can't find the container with id c2088721b450c54122f2476c4e783ab23dacb9785172ef5d3dd2b11cbcb9b974
Oct 30 13:42:06.248303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:06.248193    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:06.660732 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:06.660694    4544 generic.go:356] "Generic (PLEG): container finished" podID="0b77bf44-eb3f-47dd-b143-10cb62c7de3c" containerID="d23e1e88e41960458d322df59f15d9915fb67a6b166aaf5f339ef9171ead782b" exitCode=0
Oct 30 13:42:06.660943 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:06.660779    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-51288b-bfvlg" event={"ID":"0b77bf44-eb3f-47dd-b143-10cb62c7de3c","Type":"ContainerDied","Data":"d23e1e88e41960458d322df59f15d9915fb67a6b166aaf5f339ef9171ead782b"}
Oct 30 13:42:06.660943 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:06.660843    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-51288b-bfvlg" event={"ID":"0b77bf44-eb3f-47dd-b143-10cb62c7de3c","Type":"ContainerStarted","Data":"c2088721b450c54122f2476c4e783ab23dacb9785172ef5d3dd2b11cbcb9b974"}
Oct 30 13:42:07.816463 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:07.816430    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-51288b-bfvlg"
Oct 30 13:42:07.898885 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:07.898840    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b77bf44-eb3f-47dd-b143-10cb62c7de3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b77bf44-eb3f-47dd-b143-10cb62c7de3c-maas-api-xks-ca\") pod \"0b77bf44-eb3f-47dd-b143-10cb62c7de3c\" (UID: \"0b77bf44-eb3f-47dd-b143-10cb62c7de3c\") "
Oct 30 13:42:07.898885 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:07.898889    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b77bf44-eb3f-47dd-b143-10cb62c7de3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b77bf44-eb3f-47dd-b143-10cb62c7de3c-maas-api-openshift-ca\") pod \"0b77bf44-eb3f-47dd-b143-10cb62c7de3c\" (UID: \"0b77bf44-eb3f-47dd-b143-10cb62c7de3c\") "
Oct 30 13:42:07.899186 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:07.899105    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b77bf44-eb3f-47dd-b143-10cb62c7de3c-kube-api-access-cf95w\" (UniqueName: \"kubernetes.io/projected/0b77bf44-eb3f-47dd-b143-10cb62c7de3c-kube-api-access-cf95w\") pod \"0b77bf44-eb3f-47dd-b143-10cb62c7de3c\" (UID: \"0b77bf44-eb3f-47dd-b143-10cb62c7de3c\") "
Oct 30 13:42:07.899301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:07.899277    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b77bf44-eb3f-47dd-b143-10cb62c7de3c-maas-api-openshift-ca" pod "0b77bf44-eb3f-47dd-b143-10cb62c7de3c" (UID: "0b77bf44-eb3f-47dd-b143-10cb62c7de3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:42:07.899497 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:07.899481    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b77bf44-eb3f-47dd-b143-10cb62c7de3c-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:07.901111 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:07.901086    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b77bf44-eb3f-47dd-b143-10cb62c7de3c-maas-api-xks-ca" pod "0b77bf44-eb3f-47dd-b143-10cb62c7de3c" (UID: "0b77bf44-eb3f-47dd-b143-10cb62c7de3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:42:07.901270 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:07.901200    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b77bf44-eb3f-47dd-b143-10cb62c7de3c-kube-api-access-cf95w" pod "0b77bf44-eb3f-47dd-b143-10cb62c7de3c" (UID: "0b77bf44-eb3f-47dd-b143-10cb62c7de3c"). InnerVolumeSpecName "kube-api-access-cf95w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:42:08.000827 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:08.000741    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b77bf44-eb3f-47dd-b143-10cb62c7de3c-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:08.000827 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:08.000774    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf95w\" (UniqueName: \"kubernetes.io/projected/0b77bf44-eb3f-47dd-b143-10cb62c7de3c-kube-api-access-cf95w\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:08.064236 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:08.064187    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:08.674404 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:08.674371    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-51288b-bfvlg"
Oct 30 13:42:08.674404 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:08.674388    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-51288b-bfvlg" event={"ID":"0b77bf44-eb3f-47dd-b143-10cb62c7de3c","Type":"ContainerDied","Data":"c2088721b450c54122f2476c4e783ab23dacb9785172ef5d3dd2b11cbcb9b974"}
Oct 30 13:42:08.674634 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:08.674461    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2088721b450c54122f2476c4e783ab23dacb9785172ef5d3dd2b11cbcb9b974"
Oct 30 13:42:09.162920 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.162880    4544 patch_prober.go:29] interesting pod/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:42:09.163377 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.162947    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r" podUID="e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:42:09.682810 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.682776    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:42:09.683356 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.683335    4544 generic.go:356] "Generic (PLEG): container finished" podID="e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4" containerID="a9c0d02ae26601eb8f4bb6c3aaeb7dec41cfe823c4b498601d9f7edeca61e2b2" exitCode=0
Oct 30 13:42:09.683446 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.683408    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r" event={"ID":"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4","Type":"ContainerDied","Data":"a9c0d02ae26601eb8f4bb6c3aaeb7dec41cfe823c4b498601d9f7edeca61e2b2"}
Oct 30 13:42:09.683505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.683493    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r" event={"ID":"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4","Type":"ContainerDied","Data":"a82ced9aae19235bcbd98888999e8875cea756e621dc3cec81c1886d664515a1"}
Oct 30 13:42:09.683553 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.683504    4544 scope.go:122] "RemoveContainer" containerID="a9c0d02ae26601eb8f4bb6c3aaeb7dec41cfe823c4b498601d9f7edeca61e2b2"
Oct 30 13:42:09.692912 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.692889    4544 scope.go:122] "RemoveContainer" containerID="0257c01bd914e8974dfe7a6529ab9f0f275cf73f72c147e567b92ed8d92683d9"
Oct 30 13:42:09.702061 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.702039    4544 scope.go:122] "RemoveContainer" containerID="a9c0d02ae26601eb8f4bb6c3aaeb7dec41cfe823c4b498601d9f7edeca61e2b2"
Oct 30 13:42:09.702404 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:42:09.702376    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9c0d02ae26601eb8f4bb6c3aaeb7dec41cfe823c4b498601d9f7edeca61e2b2\": container with ID starting with a9c0d02ae26601eb8f4bb6c3aaeb7dec41cfe823c4b498601d9f7edeca61e2b2 not found: ID does not exist" containerID="a9c0d02ae26601eb8f4bb6c3aaeb7dec41cfe823c4b498601d9f7edeca61e2b2"
Oct 30 13:42:09.702486 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.702417    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9c0d02ae26601eb8f4bb6c3aaeb7dec41cfe823c4b498601d9f7edeca61e2b2"} err="failed to get container status \"a9c0d02ae26601eb8f4bb6c3aaeb7dec41cfe823c4b498601d9f7edeca61e2b2\": rpc error: code = NotFound desc = could not find container \"a9c0d02ae26601eb8f4bb6c3aaeb7dec41cfe823c4b498601d9f7edeca61e2b2\": container with ID starting with a9c0d02ae26601eb8f4bb6c3aaeb7dec41cfe823c4b498601d9f7edeca61e2b2 not found: ID does not exist"
Oct 30 13:42:09.702486 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.702439    4544 scope.go:122] "RemoveContainer" containerID="0257c01bd914e8974dfe7a6529ab9f0f275cf73f72c147e567b92ed8d92683d9"
Oct 30 13:42:09.702727 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:42:09.702709    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0257c01bd914e8974dfe7a6529ab9f0f275cf73f72c147e567b92ed8d92683d9\": container with ID starting with 0257c01bd914e8974dfe7a6529ab9f0f275cf73f72c147e567b92ed8d92683d9 not found: ID does not exist" containerID="0257c01bd914e8974dfe7a6529ab9f0f275cf73f72c147e567b92ed8d92683d9"
Oct 30 13:42:09.702778 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.702733    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0257c01bd914e8974dfe7a6529ab9f0f275cf73f72c147e567b92ed8d92683d9"} err="failed to get container status \"0257c01bd914e8974dfe7a6529ab9f0f275cf73f72c147e567b92ed8d92683d9\": rpc error: code = NotFound desc = could not find container \"0257c01bd914e8974dfe7a6529ab9f0f275cf73f72c147e567b92ed8d92683d9\": container with ID starting with 0257c01bd914e8974dfe7a6529ab9f0f275cf73f72c147e567b92ed8d92683d9 not found: ID does not exist"
Oct 30 13:42:09.839247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.839183    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-credential-socket\") pod \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") "
Oct 30 13:42:09.839479 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.839282    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-workload-socket\") pod \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") "
Oct 30 13:42:09.839479 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.839346    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-workload-certs\") pod \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") "
Oct 30 13:42:09.839479 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.839397    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istio-token\" (UniqueName: \"kubernetes.io/projected/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istio-token\") pod \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") "
Oct 30 13:42:09.839479 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.839437    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istiod-ca-cert\") pod \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") "
Oct 30 13:42:09.839732 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.839517    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istio-data\") pod \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") "
Oct 30 13:42:09.839732 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.839520    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-credential-socket" pod "e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4" (UID: "e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:09.839732 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.839560    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istio-envoy\") pod \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") "
Oct 30 13:42:09.839732 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.839647    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-kube-api-access-wlb75\" (UniqueName: \"kubernetes.io/projected/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-kube-api-access-wlb75\") pod \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") "
Oct 30 13:42:09.839732 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.839679    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istio-podinfo\") pod \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\" (UID: \"e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4\") "
Oct 30 13:42:09.839974 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.839646    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-workload-socket" pod "e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4" (UID: "e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:09.839974 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.839695    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-workload-certs" pod "e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4" (UID: "e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:09.839974 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.839891    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istio-data" pod "e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4" (UID: "e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:09.839974 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.839898    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istiod-ca-cert" pod "e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4" (UID: "e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:42:09.840179 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.840035    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:09.840179 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.840051    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:09.840179 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.840060    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:09.840179 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.840069    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:09.840179 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.840077    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:09.841876 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.841844    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-kube-api-access-wlb75" pod "e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4" (UID: "e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4"). InnerVolumeSpecName "kube-api-access-wlb75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:42:09.841876 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.841856    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istio-token" pod "e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4" (UID: "e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:42:09.842060 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.841943    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istio-envoy" pod "e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4" (UID: "e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:09.842060 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.842025    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istio-podinfo" pod "e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4" (UID: "e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:42:09.940984 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.940887    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:09.940984 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.940920    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:09.940984 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.940931    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlb75\" (UniqueName: \"kubernetes.io/projected/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-kube-api-access-wlb75\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:09.940984 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.940947    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:09.958258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:09.958212    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:10.691598 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:10.691561    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"
Oct 30 13:42:10.763942 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:10.763906    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"]
Oct 30 13:42:10.775452 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:10.775418    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r"]
Oct 30 13:42:11.085816 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:11.085772    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4" path="/var/lib/kubelet/pods/e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4/volumes"
Oct 30 13:42:11.639139 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:11.639100    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:13.143513 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:13.143473    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:13.285706 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:13.285671    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:13.444455 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:13.444360    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:15.266658 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:15.266623    4544 scope.go:122] "RemoveContainer" containerID="4a5222e162eba5ed413d154e65f07ce07be363296910adfffd4fa45657cedcab"
Oct 30 13:42:15.295286 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:15.295212    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:15.491888 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:15.491849    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77"]
Oct 30 13:42:15.522328 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:15.522201    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6"]
Oct 30 13:42:15.652141 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:15.652089    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:15.767834 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:15.767772    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:16.784326 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:16.784276    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd"]
Oct 30 13:42:16.784845 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:16.784537    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd" podUID="3253ffac-6c08-4f4c-a3e1-c0bc6debf82a" containerName="maas-api" containerID="cri-o://850a7c7b7f1eb6eab2c1e52810d698dd07cec94ab6c4908996850977d3215656" gracePeriod=30
Oct 30 13:42:16.856146 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:16.856111    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:17.070442 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:17.070413    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd"
Oct 30 13:42:17.219608 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:17.219571    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3253ffac-6c08-4f4c-a3e1-c0bc6debf82a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3253ffac-6c08-4f4c-a3e1-c0bc6debf82a-maas-api-tls\") pod \"3253ffac-6c08-4f4c-a3e1-c0bc6debf82a\" (UID: \"3253ffac-6c08-4f4c-a3e1-c0bc6debf82a\") "
Oct 30 13:42:17.219826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:17.219698    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3253ffac-6c08-4f4c-a3e1-c0bc6debf82a-kube-api-access-xhdzq\" (UniqueName: \"kubernetes.io/projected/3253ffac-6c08-4f4c-a3e1-c0bc6debf82a-kube-api-access-xhdzq\") pod \"3253ffac-6c08-4f4c-a3e1-c0bc6debf82a\" (UID: \"3253ffac-6c08-4f4c-a3e1-c0bc6debf82a\") "
Oct 30 13:42:17.219826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:17.219764    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3253ffac-6c08-4f4c-a3e1-c0bc6debf82a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3253ffac-6c08-4f4c-a3e1-c0bc6debf82a-openshift-service-ca\") pod \"3253ffac-6c08-4f4c-a3e1-c0bc6debf82a\" (UID: \"3253ffac-6c08-4f4c-a3e1-c0bc6debf82a\") "
Oct 30 13:42:17.219826 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:17.219800    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3253ffac-6c08-4f4c-a3e1-c0bc6debf82a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3253ffac-6c08-4f4c-a3e1-c0bc6debf82a-metrics-certs\") pod \"3253ffac-6c08-4f4c-a3e1-c0bc6debf82a\" (UID: \"3253ffac-6c08-4f4c-a3e1-c0bc6debf82a\") "
Oct 30 13:42:17.220190 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:17.220158    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3253ffac-6c08-4f4c-a3e1-c0bc6debf82a-openshift-service-ca" pod "3253ffac-6c08-4f4c-a3e1-c0bc6debf82a" (UID: "3253ffac-6c08-4f4c-a3e1-c0bc6debf82a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:42:17.222128 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:17.222094    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3253ffac-6c08-4f4c-a3e1-c0bc6debf82a-maas-api-tls" pod "3253ffac-6c08-4f4c-a3e1-c0bc6debf82a" (UID: "3253ffac-6c08-4f4c-a3e1-c0bc6debf82a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:42:17.222390 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:17.222369    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3253ffac-6c08-4f4c-a3e1-c0bc6debf82a-kube-api-access-xhdzq" pod "3253ffac-6c08-4f4c-a3e1-c0bc6debf82a" (UID: "3253ffac-6c08-4f4c-a3e1-c0bc6debf82a"). InnerVolumeSpecName "kube-api-access-xhdzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:42:17.222482 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:17.222466    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3253ffac-6c08-4f4c-a3e1-c0bc6debf82a-metrics-certs" pod "3253ffac-6c08-4f4c-a3e1-c0bc6debf82a" (UID: "3253ffac-6c08-4f4c-a3e1-c0bc6debf82a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:42:17.321004 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:17.320914    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3253ffac-6c08-4f4c-a3e1-c0bc6debf82a-maas-api-tls\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:17.321004 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:17.320950    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhdzq\" (UniqueName: \"kubernetes.io/projected/3253ffac-6c08-4f4c-a3e1-c0bc6debf82a-kube-api-access-xhdzq\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:17.321004 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:17.320960    4544 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3253ffac-6c08-4f4c-a3e1-c0bc6debf82a-openshift-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:17.321004 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:17.320970    4544 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3253ffac-6c08-4f4c-a3e1-c0bc6debf82a-metrics-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:17.415345 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:17.415310    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:17.544943 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:17.544900    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-51288b-bfvlg"]
Oct 30 13:42:17.546128 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:17.546099    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:17.553921 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:17.553891    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-51288b-bfvlg"]
Oct 30 13:42:17.737319 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:17.737278    4544 generic.go:356] "Generic (PLEG): container finished" podID="3253ffac-6c08-4f4c-a3e1-c0bc6debf82a" containerID="850a7c7b7f1eb6eab2c1e52810d698dd07cec94ab6c4908996850977d3215656" exitCode=0
Oct 30 13:42:17.737503 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:17.737337    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd"
Oct 30 13:42:17.737503 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:17.737372    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd" event={"ID":"3253ffac-6c08-4f4c-a3e1-c0bc6debf82a","Type":"ContainerDied","Data":"850a7c7b7f1eb6eab2c1e52810d698dd07cec94ab6c4908996850977d3215656"}
Oct 30 13:42:17.737503 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:17.737427    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd" event={"ID":"3253ffac-6c08-4f4c-a3e1-c0bc6debf82a","Type":"ContainerDied","Data":"fb7f08bcc2b0785761bf28ab242a9e2c18ea3f9d68ec99f29d23452136bda8b9"}
Oct 30 13:42:17.737503 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:17.737452    4544 scope.go:122] "RemoveContainer" containerID="850a7c7b7f1eb6eab2c1e52810d698dd07cec94ab6c4908996850977d3215656"
Oct 30 13:42:17.748441 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:17.748418    4544 scope.go:122] "RemoveContainer" containerID="850a7c7b7f1eb6eab2c1e52810d698dd07cec94ab6c4908996850977d3215656"
Oct 30 13:42:17.748808 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:42:17.748788    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"850a7c7b7f1eb6eab2c1e52810d698dd07cec94ab6c4908996850977d3215656\": container with ID starting with 850a7c7b7f1eb6eab2c1e52810d698dd07cec94ab6c4908996850977d3215656 not found: ID does not exist" containerID="850a7c7b7f1eb6eab2c1e52810d698dd07cec94ab6c4908996850977d3215656"
Oct 30 13:42:17.748856 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:17.748819    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"850a7c7b7f1eb6eab2c1e52810d698dd07cec94ab6c4908996850977d3215656"} err="failed to get container status \"850a7c7b7f1eb6eab2c1e52810d698dd07cec94ab6c4908996850977d3215656\": rpc error: code = NotFound desc = could not find container \"850a7c7b7f1eb6eab2c1e52810d698dd07cec94ab6c4908996850977d3215656\": container with ID starting with 850a7c7b7f1eb6eab2c1e52810d698dd07cec94ab6c4908996850977d3215656 not found: ID does not exist"
Oct 30 13:42:17.797920 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:17.797877    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd"]
Oct 30 13:42:17.805618 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:17.805582    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-51288b-5d57bcc8c9-p2prd"]
Oct 30 13:42:18.129987 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:18.129950    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:19.086645 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:19.086612    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b77bf44-eb3f-47dd-b143-10cb62c7de3c" path="/var/lib/kubelet/pods/0b77bf44-eb3f-47dd-b143-10cb62c7de3c/volumes"
Oct 30 13:42:19.087064 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:19.087049    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3253ffac-6c08-4f4c-a3e1-c0bc6debf82a" path="/var/lib/kubelet/pods/3253ffac-6c08-4f4c-a3e1-c0bc6debf82a/volumes"
Oct 30 13:42:19.154658 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:19.154624    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:19.545164 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:19.545078    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:20.492973 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:20.492917    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77" podUID="95241140-c937-47c1-9413-8bc8a68c840c" containerName="payload-processing" containerID="cri-o://abb695578156c6415810b0f7c200de205f3cb192fe6aa53448436344a42167f7" gracePeriod=25
Oct 30 13:42:20.523327 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:20.523263    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6" podUID="480fecc2-7659-45df-b613-3e0f2b0f7fff" containerName="payload-pre-processing" containerID="cri-o://85d2ff318602232335b2e698eb4b684b3ac2c6772e540faebfa9c0d30145a327" gracePeriod=25
Oct 30 13:42:21.268392 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:21.268358    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:21.324117 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:21.324074    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6" podUID="480fecc2-7659-45df-b613-3e0f2b0f7fff" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:42:21.324342 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:21.324083    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77" podUID="95241140-c937-47c1-9413-8bc8a68c840c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:42:22.953129 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:22.953087    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:24.510889 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:24.510850    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:24.635783 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:24.635745    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:24.878440 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:24.878404    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:24.918893 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:24.918852    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"]
Oct 30 13:42:24.919095 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:24.919064    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z" podUID="e16e9bc8-8d01-4ab3-a840-190ffa478e8f" containerName="istio-proxy" containerID="cri-o://122ac7f78ea0efc1fe7d89b1ddb88301a5a1cd431b202e93246e578c2a98aa6e" gracePeriod=30
Oct 30 13:42:25.051591 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:25.051547    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:25.437076 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:25.437030    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-627dfa-884dl"]
Oct 30 13:42:25.437618 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:25.437597    4544 cpu_manager.go:414] "containerMap: removing container" podUID="e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4" containerName="istio-proxy"
Oct 30 13:42:25.437687 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:25.437619    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4" containerName="istio-proxy"
Oct 30 13:42:25.437687 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:25.437631    4544 cpu_manager.go:414] "containerMap: removing container" podUID="0b77bf44-eb3f-47dd-b143-10cb62c7de3c" containerName="revoke-keys"
Oct 30 13:42:25.437687 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:25.437636    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b77bf44-eb3f-47dd-b143-10cb62c7de3c" containerName="revoke-keys"
Oct 30 13:42:25.437687 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:25.437651    4544 cpu_manager.go:414] "containerMap: removing container" podUID="3253ffac-6c08-4f4c-a3e1-c0bc6debf82a" containerName="maas-api"
Oct 30 13:42:25.437687 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:25.437657    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3253ffac-6c08-4f4c-a3e1-c0bc6debf82a" containerName="maas-api"
Oct 30 13:42:25.437864 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:25.437734    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="3253ffac-6c08-4f4c-a3e1-c0bc6debf82a" containerName="maas-api"
Oct 30 13:42:25.437864 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:25.437746    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4" containerName="istio-proxy"
Oct 30 13:42:25.437864 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:25.437754    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b77bf44-eb3f-47dd-b143-10cb62c7de3c" containerName="revoke-keys"
Oct 30 13:42:25.442412 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:25.442384    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-627dfa-884dl"
Oct 30 13:42:25.459851 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:25.459821    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:42:25.464460 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:25.464423    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-627dfa-884dl"]
Oct 30 13:42:25.472121 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:25.472090    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:42:25.514913 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:25.514874    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da60b86f-3525-4ae1-9909-03db57734368-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-627dfa-884dl\" (UID: \"da60b86f-3525-4ae1-9909-03db57734368\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-627dfa-884dl"
Oct 30 13:42:25.515360 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:25.514998    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnllv\" (UniqueName: \"kubernetes.io/projected/da60b86f-3525-4ae1-9909-03db57734368-kube-api-access-wnllv\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-627dfa-884dl\" (UID: \"da60b86f-3525-4ae1-9909-03db57734368\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-627dfa-884dl"
Oct 30 13:42:25.515360 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:25.515057    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da60b86f-3525-4ae1-9909-03db57734368-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-627dfa-884dl\" (UID: \"da60b86f-3525-4ae1-9909-03db57734368\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-627dfa-884dl"
Oct 30 13:42:25.616080 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:25.616034    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da60b86f-3525-4ae1-9909-03db57734368-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-627dfa-884dl\" (UID: \"da60b86f-3525-4ae1-9909-03db57734368\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-627dfa-884dl"
Oct 30 13:42:25.616318 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:25.616147    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da60b86f-3525-4ae1-9909-03db57734368-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-627dfa-884dl\" (UID: \"da60b86f-3525-4ae1-9909-03db57734368\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-627dfa-884dl"
Oct 30 13:42:25.616318 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:25.616235    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnllv\" (UniqueName: \"kubernetes.io/projected/da60b86f-3525-4ae1-9909-03db57734368-kube-api-access-wnllv\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-627dfa-884dl\" (UID: \"da60b86f-3525-4ae1-9909-03db57734368\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-627dfa-884dl"
Oct 30 13:42:25.616856 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:25.616836    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da60b86f-3525-4ae1-9909-03db57734368-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-627dfa-884dl\" (UID: \"da60b86f-3525-4ae1-9909-03db57734368\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-627dfa-884dl"
Oct 30 13:42:25.618512 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:25.618478    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da60b86f-3525-4ae1-9909-03db57734368-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-627dfa-884dl\" (UID: \"da60b86f-3525-4ae1-9909-03db57734368\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-627dfa-884dl"
Oct 30 13:42:25.640743 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:25.640702    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnllv\" (UniqueName: \"kubernetes.io/projected/da60b86f-3525-4ae1-9909-03db57734368-kube-api-access-wnllv\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-627dfa-884dl\" (UID: \"da60b86f-3525-4ae1-9909-03db57734368\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-627dfa-884dl"
Oct 30 13:42:25.755562 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:25.755448    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-627dfa-884dl"
Oct 30 13:42:26.061538 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:26.061442    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-627dfa-884dl"]
Oct 30 13:42:26.065796 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:42:26.065750    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda60b86f_3525_4ae1_9909_03db57734368.slice/crio-6763da95d82267b6f2c205aafd31336cbf247339ddabeee65259f6a4c0aca426 WatchSource:0}: Error finding container 6763da95d82267b6f2c205aafd31336cbf247339ddabeee65259f6a4c0aca426: Status 404 returned error can't find the container with id 6763da95d82267b6f2c205aafd31336cbf247339ddabeee65259f6a4c0aca426
Oct 30 13:42:26.657693 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:26.657649    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:26.792620 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:26.792584    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-627dfa-884dl" event={"ID":"da60b86f-3525-4ae1-9909-03db57734368","Type":"ContainerStarted","Data":"488de18aa1625eaf36ab73a68985a0ed8fa71f6bfe8fad0ebf05d92c4a7d0b1b"}
Oct 30 13:42:26.792835 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:26.792676    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-627dfa-884dl" event={"ID":"da60b86f-3525-4ae1-9909-03db57734368","Type":"ContainerStarted","Data":"6763da95d82267b6f2c205aafd31336cbf247339ddabeee65259f6a4c0aca426"}
Oct 30 13:42:26.835553 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:26.835487    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-627dfa-884dl" podStartSLOduration=1.8354717059999999 podStartE2EDuration="1.835471706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:42:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:42:26.833624734 +0000 UTC m=+3441.880758686" watchObservedRunningTime="2026-10-30 13:42:26.835471706 +0000 UTC m=+3441.882605658"
Oct 30 13:42:27.800385 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:27.800351    4544 generic.go:356] "Generic (PLEG): container finished" podID="da60b86f-3525-4ae1-9909-03db57734368" containerID="488de18aa1625eaf36ab73a68985a0ed8fa71f6bfe8fad0ebf05d92c4a7d0b1b" exitCode=0
Oct 30 13:42:27.800812 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:27.800434    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-627dfa-884dl" event={"ID":"da60b86f-3525-4ae1-9909-03db57734368","Type":"ContainerDied","Data":"488de18aa1625eaf36ab73a68985a0ed8fa71f6bfe8fad0ebf05d92c4a7d0b1b"}
Oct 30 13:42:28.520505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:28.520467    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:28.956626 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:28.956593    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-627dfa-884dl"
Oct 30 13:42:29.012476 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:42:29.012433    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:42:29.052131 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:29.052090    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da60b86f-3525-4ae1-9909-03db57734368-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da60b86f-3525-4ae1-9909-03db57734368-maas-api-openshift-ca\") pod \"da60b86f-3525-4ae1-9909-03db57734368\" (UID: \"da60b86f-3525-4ae1-9909-03db57734368\") "
Oct 30 13:42:29.052347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:29.052174    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da60b86f-3525-4ae1-9909-03db57734368-kube-api-access-wnllv\" (UniqueName: \"kubernetes.io/projected/da60b86f-3525-4ae1-9909-03db57734368-kube-api-access-wnllv\") pod \"da60b86f-3525-4ae1-9909-03db57734368\" (UID: \"da60b86f-3525-4ae1-9909-03db57734368\") "
Oct 30 13:42:29.052347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:29.052306    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da60b86f-3525-4ae1-9909-03db57734368-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da60b86f-3525-4ae1-9909-03db57734368-maas-api-xks-ca\") pod \"da60b86f-3525-4ae1-9909-03db57734368\" (UID: \"da60b86f-3525-4ae1-9909-03db57734368\") "
Oct 30 13:42:29.052546 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:29.052517    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da60b86f-3525-4ae1-9909-03db57734368-maas-api-openshift-ca" pod "da60b86f-3525-4ae1-9909-03db57734368" (UID: "da60b86f-3525-4ae1-9909-03db57734368"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:42:29.054424 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:29.054379    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da60b86f-3525-4ae1-9909-03db57734368-maas-api-xks-ca" pod "da60b86f-3525-4ae1-9909-03db57734368" (UID: "da60b86f-3525-4ae1-9909-03db57734368"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:42:29.054570 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:29.054464    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da60b86f-3525-4ae1-9909-03db57734368-kube-api-access-wnllv" pod "da60b86f-3525-4ae1-9909-03db57734368" (UID: "da60b86f-3525-4ae1-9909-03db57734368"). InnerVolumeSpecName "kube-api-access-wnllv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:42:29.153885 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:29.153834    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da60b86f-3525-4ae1-9909-03db57734368-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:29.153885 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:29.153873    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da60b86f-3525-4ae1-9909-03db57734368-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:29.153885 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:29.153885    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnllv\" (UniqueName: \"kubernetes.io/projected/da60b86f-3525-4ae1-9909-03db57734368-kube-api-access-wnllv\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:29.813264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:29.813207    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-627dfa-884dl"
Oct 30 13:42:29.813264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:29.813251    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-627dfa-884dl" event={"ID":"da60b86f-3525-4ae1-9909-03db57734368","Type":"ContainerDied","Data":"6763da95d82267b6f2c205aafd31336cbf247339ddabeee65259f6a4c0aca426"}
Oct 30 13:42:29.813505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:29.813304    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6763da95d82267b6f2c205aafd31336cbf247339ddabeee65259f6a4c0aca426"
Oct 30 13:42:30.192818 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.192789    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:42:30.264774 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.264738    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istio-data\") pod \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") "
Oct 30 13:42:30.264774 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.264779    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-credential-socket\") pod \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") "
Oct 30 13:42:30.265043 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.264807    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-workload-socket\") pod \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") "
Oct 30 13:42:30.265043 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.264852    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istio-envoy\") pod \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") "
Oct 30 13:42:30.265043 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.264936    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-kube-api-access-mkt22\" (UniqueName: \"kubernetes.io/projected/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-kube-api-access-mkt22\") pod \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") "
Oct 30 13:42:30.265043 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.264973    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-workload-certs\") pod \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") "
Oct 30 13:42:30.265043 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.265024    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istiod-ca-cert\") pod \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") "
Oct 30 13:42:30.265344 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.265105    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-credential-socket" pod "e16e9bc8-8d01-4ab3-a840-190ffa478e8f" (UID: "e16e9bc8-8d01-4ab3-a840-190ffa478e8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:30.265344 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.265116    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istio-data" pod "e16e9bc8-8d01-4ab3-a840-190ffa478e8f" (UID: "e16e9bc8-8d01-4ab3-a840-190ffa478e8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:30.265344 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.265137    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-workload-socket" pod "e16e9bc8-8d01-4ab3-a840-190ffa478e8f" (UID: "e16e9bc8-8d01-4ab3-a840-190ffa478e8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:30.265344 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.265137    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istio-token\" (UniqueName: \"kubernetes.io/projected/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istio-token\") pod \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") "
Oct 30 13:42:30.265344 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.265215    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istio-podinfo\") pod \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\" (UID: \"e16e9bc8-8d01-4ab3-a840-190ffa478e8f\") "
Oct 30 13:42:30.265344 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.265333    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-workload-certs" pod "e16e9bc8-8d01-4ab3-a840-190ffa478e8f" (UID: "e16e9bc8-8d01-4ab3-a840-190ffa478e8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:30.265907 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.265577    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istiod-ca-cert" pod "e16e9bc8-8d01-4ab3-a840-190ffa478e8f" (UID: "e16e9bc8-8d01-4ab3-a840-190ffa478e8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:42:30.265907 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.265898    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:30.266059 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.265920    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:30.266059 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.265929    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:30.266059 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.265940    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:30.266059 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.265953    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:30.267490 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.267459    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-kube-api-access-mkt22" pod "e16e9bc8-8d01-4ab3-a840-190ffa478e8f" (UID: "e16e9bc8-8d01-4ab3-a840-190ffa478e8f"). InnerVolumeSpecName "kube-api-access-mkt22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:42:30.267490 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.267469    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istio-token" pod "e16e9bc8-8d01-4ab3-a840-190ffa478e8f" (UID: "e16e9bc8-8d01-4ab3-a840-190ffa478e8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:42:30.267838 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.267671    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istio-podinfo" pod "e16e9bc8-8d01-4ab3-a840-190ffa478e8f" (UID: "e16e9bc8-8d01-4ab3-a840-190ffa478e8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:42:30.267838 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.267799    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istio-envoy" pod "e16e9bc8-8d01-4ab3-a840-190ffa478e8f" (UID: "e16e9bc8-8d01-4ab3-a840-190ffa478e8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:30.269700 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.269673    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:30.367074 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.366977    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkt22\" (UniqueName: \"kubernetes.io/projected/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-kube-api-access-mkt22\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:30.367074 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.367014    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:30.367074 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.367026    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:30.367074 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.367034    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e16e9bc8-8d01-4ab3-a840-190ffa478e8f-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:30.821047 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.820939    4544 generic.go:356] "Generic (PLEG): container finished" podID="e16e9bc8-8d01-4ab3-a840-190ffa478e8f" containerID="122ac7f78ea0efc1fe7d89b1ddb88301a5a1cd431b202e93246e578c2a98aa6e" exitCode=0
Oct 30 13:42:30.821047 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.820990    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"
Oct 30 13:42:30.821047 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.821002    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z" event={"ID":"e16e9bc8-8d01-4ab3-a840-190ffa478e8f","Type":"ContainerDied","Data":"122ac7f78ea0efc1fe7d89b1ddb88301a5a1cd431b202e93246e578c2a98aa6e"}
Oct 30 13:42:30.821401 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.821071    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z" event={"ID":"e16e9bc8-8d01-4ab3-a840-190ffa478e8f","Type":"ContainerDied","Data":"925fb155516771ad0d1fc0267875634720dc2d2cef51541ff27c714809fcd213"}
Oct 30 13:42:30.821401 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.821103    4544 scope.go:122] "RemoveContainer" containerID="122ac7f78ea0efc1fe7d89b1ddb88301a5a1cd431b202e93246e578c2a98aa6e"
Oct 30 13:42:30.832422 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.832401    4544 scope.go:122] "RemoveContainer" containerID="122ac7f78ea0efc1fe7d89b1ddb88301a5a1cd431b202e93246e578c2a98aa6e"
Oct 30 13:42:30.832789 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:42:30.832758    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"122ac7f78ea0efc1fe7d89b1ddb88301a5a1cd431b202e93246e578c2a98aa6e\": container with ID starting with 122ac7f78ea0efc1fe7d89b1ddb88301a5a1cd431b202e93246e578c2a98aa6e not found: ID does not exist" containerID="122ac7f78ea0efc1fe7d89b1ddb88301a5a1cd431b202e93246e578c2a98aa6e"
Oct 30 13:42:30.832854 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.832803    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"122ac7f78ea0efc1fe7d89b1ddb88301a5a1cd431b202e93246e578c2a98aa6e"} err="failed to get container status \"122ac7f78ea0efc1fe7d89b1ddb88301a5a1cd431b202e93246e578c2a98aa6e\": rpc error: code = NotFound desc = could not find container \"122ac7f78ea0efc1fe7d89b1ddb88301a5a1cd431b202e93246e578c2a98aa6e\": container with ID starting with 122ac7f78ea0efc1fe7d89b1ddb88301a5a1cd431b202e93246e578c2a98aa6e not found: ID does not exist"
Oct 30 13:42:30.898253 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.898184    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"]
Oct 30 13:42:30.909704 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:30.909670    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z"]
Oct 30 13:42:31.087505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:31.086848    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e16e9bc8-8d01-4ab3-a840-190ffa478e8f" path="/var/lib/kubelet/pods/e16e9bc8-8d01-4ab3-a840-190ffa478e8f/volumes"
Oct 30 13:42:31.324756 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:31.324713    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6" podUID="480fecc2-7659-45df-b613-3e0f2b0f7fff" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:42:31.325146 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:31.324723    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77" podUID="95241140-c937-47c1-9413-8bc8a68c840c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:42:32.005195 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:32.005157    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:33.867975 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:33.867935    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:35.544623 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:35.544576    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:35.929107 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:35.929069    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q"]
Oct 30 13:42:35.979987 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:35.979954    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558"]
Oct 30 13:42:36.109954 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:36.109924    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:36.226577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:36.226486    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:37.421811 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:37.421762    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:38.942902 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:38.942867    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:39.630716 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:39.630672    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:39.721785 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:39.721739    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-627dfa-884dl"]
Oct 30 13:42:39.734334 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:39.734296    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-627dfa-884dl"]
Oct 30 13:42:39.831377 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:39.831335    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:40.328706 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.328661    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:40.786748 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.786719    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77"
Oct 30 13:42:40.815036 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.814999    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6"
Oct 30 13:42:40.871551 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.871506    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95241140-c937-47c1-9413-8bc8a68c840c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95241140-c937-47c1-9413-8bc8a68c840c-plugins-config-volume\") pod \"95241140-c937-47c1-9413-8bc8a68c840c\" (UID: \"95241140-c937-47c1-9413-8bc8a68c840c\") "
Oct 30 13:42:40.871725 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.871588    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95241140-c937-47c1-9413-8bc8a68c840c-kube-api-access-l9mbb\" (UniqueName: \"kubernetes.io/projected/95241140-c937-47c1-9413-8bc8a68c840c-kube-api-access-l9mbb\") pod \"95241140-c937-47c1-9413-8bc8a68c840c\" (UID: \"95241140-c937-47c1-9413-8bc8a68c840c\") "
Oct 30 13:42:40.871725 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.871688    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/480fecc2-7659-45df-b613-3e0f2b0f7fff-kube-api-access-pcr29\" (UniqueName: \"kubernetes.io/projected/480fecc2-7659-45df-b613-3e0f2b0f7fff-kube-api-access-pcr29\") pod \"480fecc2-7659-45df-b613-3e0f2b0f7fff\" (UID: \"480fecc2-7659-45df-b613-3e0f2b0f7fff\") "
Oct 30 13:42:40.871824 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.871731    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/480fecc2-7659-45df-b613-3e0f2b0f7fff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/480fecc2-7659-45df-b613-3e0f2b0f7fff-plugins-config-volume\") pod \"480fecc2-7659-45df-b613-3e0f2b0f7fff\" (UID: \"480fecc2-7659-45df-b613-3e0f2b0f7fff\") "
Oct 30 13:42:40.872285 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.872257    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95241140-c937-47c1-9413-8bc8a68c840c-plugins-config-volume" pod "95241140-c937-47c1-9413-8bc8a68c840c" (UID: "95241140-c937-47c1-9413-8bc8a68c840c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:42:40.872285 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.872256    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/480fecc2-7659-45df-b613-3e0f2b0f7fff-plugins-config-volume" pod "480fecc2-7659-45df-b613-3e0f2b0f7fff" (UID: "480fecc2-7659-45df-b613-3e0f2b0f7fff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:42:40.874453 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.874433    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/480fecc2-7659-45df-b613-3e0f2b0f7fff-kube-api-access-pcr29" pod "480fecc2-7659-45df-b613-3e0f2b0f7fff" (UID: "480fecc2-7659-45df-b613-3e0f2b0f7fff"). InnerVolumeSpecName "kube-api-access-pcr29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:42:40.874593 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.874457    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95241140-c937-47c1-9413-8bc8a68c840c-kube-api-access-l9mbb" pod "95241140-c937-47c1-9413-8bc8a68c840c" (UID: "95241140-c937-47c1-9413-8bc8a68c840c"). InnerVolumeSpecName "kube-api-access-l9mbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:42:40.884498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.884394    4544 generic.go:356] "Generic (PLEG): container finished" podID="95241140-c937-47c1-9413-8bc8a68c840c" containerID="abb695578156c6415810b0f7c200de205f3cb192fe6aa53448436344a42167f7" exitCode=0
Oct 30 13:42:40.884498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.884439    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77"
Oct 30 13:42:40.884498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.884479    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77" event={"ID":"95241140-c937-47c1-9413-8bc8a68c840c","Type":"ContainerDied","Data":"abb695578156c6415810b0f7c200de205f3cb192fe6aa53448436344a42167f7"}
Oct 30 13:42:40.884886 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.884537    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77" event={"ID":"95241140-c937-47c1-9413-8bc8a68c840c","Type":"ContainerDied","Data":"bcb326a362c34cf268e2a661860d8568e5f88ed9f02d2c553d270834f88eab09"}
Oct 30 13:42:40.884886 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.884572    4544 scope.go:122] "RemoveContainer" containerID="abb695578156c6415810b0f7c200de205f3cb192fe6aa53448436344a42167f7"
Oct 30 13:42:40.889959 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.886726    4544 generic.go:356] "Generic (PLEG): container finished" podID="480fecc2-7659-45df-b613-3e0f2b0f7fff" containerID="85d2ff318602232335b2e698eb4b684b3ac2c6772e540faebfa9c0d30145a327" exitCode=0
Oct 30 13:42:40.889959 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.886782    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6"
Oct 30 13:42:40.889959 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.886791    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6" event={"ID":"480fecc2-7659-45df-b613-3e0f2b0f7fff","Type":"ContainerDied","Data":"85d2ff318602232335b2e698eb4b684b3ac2c6772e540faebfa9c0d30145a327"}
Oct 30 13:42:40.889959 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.886903    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6" event={"ID":"480fecc2-7659-45df-b613-3e0f2b0f7fff","Type":"ContainerDied","Data":"d9c74b63cd3a9f0e66e13de689a3f03bc780eb531fc7befbae61731dcc4b0754"}
Oct 30 13:42:40.897094 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.897067    4544 scope.go:122] "RemoveContainer" containerID="abb695578156c6415810b0f7c200de205f3cb192fe6aa53448436344a42167f7"
Oct 30 13:42:40.897558 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:42:40.897524    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abb695578156c6415810b0f7c200de205f3cb192fe6aa53448436344a42167f7\": container with ID starting with abb695578156c6415810b0f7c200de205f3cb192fe6aa53448436344a42167f7 not found: ID does not exist" containerID="abb695578156c6415810b0f7c200de205f3cb192fe6aa53448436344a42167f7"
Oct 30 13:42:40.897698 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.897572    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abb695578156c6415810b0f7c200de205f3cb192fe6aa53448436344a42167f7"} err="failed to get container status \"abb695578156c6415810b0f7c200de205f3cb192fe6aa53448436344a42167f7\": rpc error: code = NotFound desc = could not find container \"abb695578156c6415810b0f7c200de205f3cb192fe6aa53448436344a42167f7\": container with ID starting with abb695578156c6415810b0f7c200de205f3cb192fe6aa53448436344a42167f7 not found: ID does not exist"
Oct 30 13:42:40.897698 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.897599    4544 scope.go:122] "RemoveContainer" containerID="85d2ff318602232335b2e698eb4b684b3ac2c6772e540faebfa9c0d30145a327"
Oct 30 13:42:40.909866 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.909840    4544 scope.go:122] "RemoveContainer" containerID="85d2ff318602232335b2e698eb4b684b3ac2c6772e540faebfa9c0d30145a327"
Oct 30 13:42:40.910290 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:42:40.910264    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85d2ff318602232335b2e698eb4b684b3ac2c6772e540faebfa9c0d30145a327\": container with ID starting with 85d2ff318602232335b2e698eb4b684b3ac2c6772e540faebfa9c0d30145a327 not found: ID does not exist" containerID="85d2ff318602232335b2e698eb4b684b3ac2c6772e540faebfa9c0d30145a327"
Oct 30 13:42:40.910375 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.910304    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85d2ff318602232335b2e698eb4b684b3ac2c6772e540faebfa9c0d30145a327"} err="failed to get container status \"85d2ff318602232335b2e698eb4b684b3ac2c6772e540faebfa9c0d30145a327\": rpc error: code = NotFound desc = could not find container \"85d2ff318602232335b2e698eb4b684b3ac2c6772e540faebfa9c0d30145a327\": container with ID starting with 85d2ff318602232335b2e698eb4b684b3ac2c6772e540faebfa9c0d30145a327 not found: ID does not exist"
Oct 30 13:42:40.930061 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.930016    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q" podUID="b63682f2-c062-4ef4-801f-b84a3a7c10e0" containerName="payload-processing" containerID="cri-o://c0dad761436aeb0a5652b16312563ee313497554ada9a11e09b8a647a7cd5206" gracePeriod=25
Oct 30 13:42:40.942669 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.942634    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6"]
Oct 30 13:42:40.949012 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.948978    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6"]
Oct 30 13:42:40.972877 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.972830    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcr29\" (UniqueName: \"kubernetes.io/projected/480fecc2-7659-45df-b613-3e0f2b0f7fff-kube-api-access-pcr29\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:40.972877 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.972869    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/480fecc2-7659-45df-b613-3e0f2b0f7fff-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:40.972877 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.972881    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95241140-c937-47c1-9413-8bc8a68c840c-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:40.973145 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.972889    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9mbb\" (UniqueName: \"kubernetes.io/projected/95241140-c937-47c1-9413-8bc8a68c840c-kube-api-access-l9mbb\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:40.980751 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.980669    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558" podUID="d7da56bd-b1ed-419f-b90d-c9a68a8f756d" containerName="payload-pre-processing" containerID="cri-o://10566545545ef4c2c6cb6c768c48d6805dcb63683bb3e1bfa783b1851e71f7c2" gracePeriod=25
Oct 30 13:42:40.983784 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.983696    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77"]
Oct 30 13:42:40.990469 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:40.990439    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77"]
Oct 30 13:42:41.085103 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:41.085066    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="480fecc2-7659-45df-b613-3e0f2b0f7fff" path="/var/lib/kubelet/pods/480fecc2-7659-45df-b613-3e0f2b0f7fff/volumes"
Oct 30 13:42:41.085537 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:41.085522    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95241140-c937-47c1-9413-8bc8a68c840c" path="/var/lib/kubelet/pods/95241140-c937-47c1-9413-8bc8a68c840c/volumes"
Oct 30 13:42:41.085947 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:41.085933    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da60b86f-3525-4ae1-9909-03db57734368" path="/var/lib/kubelet/pods/da60b86f-3525-4ae1-9909-03db57734368/volumes"
Oct 30 13:42:41.691869 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:41.691834    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:43.468730 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:43.468690    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:45.216183 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:45.216146    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:46.942597 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:46.942497    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:47.115609 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:47.115575    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:47.497419 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:47.497386    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"]
Oct 30 13:42:47.497708 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:47.497657    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5" podUID="265b2410-a8b7-4d99-9cda-32565416a1d6" containerName="istio-proxy" containerID="cri-o://da9dd40a34d315792924f330b490886541402d55bdab34f8150adce630fa519d" gracePeriod=30
Oct 30 13:42:47.538313 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:47.538275    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:47.997108 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:47.997068    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:48.171851 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.171813    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-67f56e-zxf8z"]
Oct 30 13:42:48.172279 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.172264    4544 cpu_manager.go:414] "containerMap: removing container" podUID="95241140-c937-47c1-9413-8bc8a68c840c" containerName="payload-processing"
Oct 30 13:42:48.172331 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.172281    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95241140-c937-47c1-9413-8bc8a68c840c" containerName="payload-processing"
Oct 30 13:42:48.172331 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.172294    4544 cpu_manager.go:414] "containerMap: removing container" podUID="e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4" containerName="istio-proxy"
Oct 30 13:42:48.172331 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.172300    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4" containerName="istio-proxy"
Oct 30 13:42:48.172435 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.172329    4544 cpu_manager.go:414] "containerMap: removing container" podUID="480fecc2-7659-45df-b613-3e0f2b0f7fff" containerName="payload-pre-processing"
Oct 30 13:42:48.172435 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.172338    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="480fecc2-7659-45df-b613-3e0f2b0f7fff" containerName="payload-pre-processing"
Oct 30 13:42:48.172435 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.172354    4544 cpu_manager.go:414] "containerMap: removing container" podUID="da60b86f-3525-4ae1-9909-03db57734368" containerName="revoke-keys"
Oct 30 13:42:48.172435 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.172359    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da60b86f-3525-4ae1-9909-03db57734368" containerName="revoke-keys"
Oct 30 13:42:48.172435 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.172369    4544 cpu_manager.go:414] "containerMap: removing container" podUID="e16e9bc8-8d01-4ab3-a840-190ffa478e8f" containerName="istio-proxy"
Oct 30 13:42:48.172435 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.172373    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e16e9bc8-8d01-4ab3-a840-190ffa478e8f" containerName="istio-proxy"
Oct 30 13:42:48.172633 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.172439    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="e16e9bc8-8d01-4ab3-a840-190ffa478e8f" containerName="istio-proxy"
Oct 30 13:42:48.172633 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.172448    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="95241140-c937-47c1-9413-8bc8a68c840c" containerName="payload-processing"
Oct 30 13:42:48.172633 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.172459    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="480fecc2-7659-45df-b613-3e0f2b0f7fff" containerName="payload-pre-processing"
Oct 30 13:42:48.172633 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.172464    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="e44b0460-7cc5-4fd3-9771-e5a6d53e0cf4" containerName="istio-proxy"
Oct 30 13:42:48.172633 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.172470    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="da60b86f-3525-4ae1-9909-03db57734368" containerName="revoke-keys"
Oct 30 13:42:48.177130 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.177101    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-67f56e-zxf8z"
Oct 30 13:42:48.196698 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.196670    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-67f56e-zxf8z"]
Oct 30 13:42:48.247501 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.247408    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/061866b2-700e-4b08-ad03-ef19858bb2e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-67f56e-zxf8z\" (UID: \"061866b2-700e-4b08-ad03-ef19858bb2e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-67f56e-zxf8z"
Oct 30 13:42:48.247688 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.247545    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/061866b2-700e-4b08-ad03-ef19858bb2e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-67f56e-zxf8z\" (UID: \"061866b2-700e-4b08-ad03-ef19858bb2e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-67f56e-zxf8z"
Oct 30 13:42:48.247688 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.247613    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzkc6\" (UniqueName: \"kubernetes.io/projected/061866b2-700e-4b08-ad03-ef19858bb2e9-kube-api-access-vzkc6\") pod \"maas-api-revoke-keys-e2e-worker-w4-67f56e-zxf8z\" (UID: \"061866b2-700e-4b08-ad03-ef19858bb2e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-67f56e-zxf8z"
Oct 30 13:42:48.252174 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.252145    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558" podUID="d7da56bd-b1ed-419f-b90d-c9a68a8f756d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:42:48.252330 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.252258    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q" podUID="b63682f2-c062-4ef4-801f-b84a3a7c10e0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:42:48.268792 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.268763    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:42:48.269023 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.269006    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:42:48.349056 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.349018    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/061866b2-700e-4b08-ad03-ef19858bb2e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-67f56e-zxf8z\" (UID: \"061866b2-700e-4b08-ad03-ef19858bb2e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-67f56e-zxf8z"
Oct 30 13:42:48.349257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.349073    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzkc6\" (UniqueName: \"kubernetes.io/projected/061866b2-700e-4b08-ad03-ef19858bb2e9-kube-api-access-vzkc6\") pod \"maas-api-revoke-keys-e2e-worker-w4-67f56e-zxf8z\" (UID: \"061866b2-700e-4b08-ad03-ef19858bb2e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-67f56e-zxf8z"
Oct 30 13:42:48.349257 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.349139    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/061866b2-700e-4b08-ad03-ef19858bb2e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-67f56e-zxf8z\" (UID: \"061866b2-700e-4b08-ad03-ef19858bb2e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-67f56e-zxf8z"
Oct 30 13:42:48.349923 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.349896    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/061866b2-700e-4b08-ad03-ef19858bb2e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-67f56e-zxf8z\" (UID: \"061866b2-700e-4b08-ad03-ef19858bb2e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-67f56e-zxf8z"
Oct 30 13:42:48.351634 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.351615    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/061866b2-700e-4b08-ad03-ef19858bb2e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-67f56e-zxf8z\" (UID: \"061866b2-700e-4b08-ad03-ef19858bb2e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-67f56e-zxf8z"
Oct 30 13:42:48.372776 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.372738    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzkc6\" (UniqueName: \"kubernetes.io/projected/061866b2-700e-4b08-ad03-ef19858bb2e9-kube-api-access-vzkc6\") pod \"maas-api-revoke-keys-e2e-worker-w4-67f56e-zxf8z\" (UID: \"061866b2-700e-4b08-ad03-ef19858bb2e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-67f56e-zxf8z"
Oct 30 13:42:48.490444 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.490402    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-67f56e-zxf8z"
Oct 30 13:42:48.797708 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.797622    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-67f56e-zxf8z"]
Oct 30 13:42:48.801054 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:42:48.801025    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061866b2_700e_4b08_ad03_ef19858bb2e9.slice/crio-932600b8e9c52ec155d619f130b52bf7234dd81ffdb6ee78ed6f5cbd255f1256 WatchSource:0}: Error finding container 932600b8e9c52ec155d619f130b52bf7234dd81ffdb6ee78ed6f5cbd255f1256: Status 404 returned error can't find the container with id 932600b8e9c52ec155d619f130b52bf7234dd81ffdb6ee78ed6f5cbd255f1256
Oct 30 13:42:48.886545 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.886510    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:48.945896 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:48.945851    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-67f56e-zxf8z" event={"ID":"061866b2-700e-4b08-ad03-ef19858bb2e9","Type":"ContainerStarted","Data":"932600b8e9c52ec155d619f130b52bf7234dd81ffdb6ee78ed6f5cbd255f1256"}
Oct 30 13:42:49.953405 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:49.953369    4544 generic.go:356] "Generic (PLEG): container finished" podID="061866b2-700e-4b08-ad03-ef19858bb2e9" containerID="68a1037cbd27423558f5bd476cd7ee5eec3abf2f3dc55d82eea9e426a8f45379" exitCode=0
Oct 30 13:42:49.953901 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:49.953416    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-67f56e-zxf8z" event={"ID":"061866b2-700e-4b08-ad03-ef19858bb2e9","Type":"ContainerDied","Data":"68a1037cbd27423558f5bd476cd7ee5eec3abf2f3dc55d82eea9e426a8f45379"}
Oct 30 13:42:50.974991 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:50.974946    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:51.104904 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:51.104878    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-67f56e-zxf8z"
Oct 30 13:42:51.178509 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:51.178479    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/061866b2-700e-4b08-ad03-ef19858bb2e9-kube-api-access-vzkc6\" (UniqueName: \"kubernetes.io/projected/061866b2-700e-4b08-ad03-ef19858bb2e9-kube-api-access-vzkc6\") pod \"061866b2-700e-4b08-ad03-ef19858bb2e9\" (UID: \"061866b2-700e-4b08-ad03-ef19858bb2e9\") "
Oct 30 13:42:51.178727 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:51.178522    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/061866b2-700e-4b08-ad03-ef19858bb2e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/061866b2-700e-4b08-ad03-ef19858bb2e9-maas-api-xks-ca\") pod \"061866b2-700e-4b08-ad03-ef19858bb2e9\" (UID: \"061866b2-700e-4b08-ad03-ef19858bb2e9\") "
Oct 30 13:42:51.178727 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:51.178545    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/061866b2-700e-4b08-ad03-ef19858bb2e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/061866b2-700e-4b08-ad03-ef19858bb2e9-maas-api-openshift-ca\") pod \"061866b2-700e-4b08-ad03-ef19858bb2e9\" (UID: \"061866b2-700e-4b08-ad03-ef19858bb2e9\") "
Oct 30 13:42:51.178997 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:51.178967    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/061866b2-700e-4b08-ad03-ef19858bb2e9-maas-api-openshift-ca" pod "061866b2-700e-4b08-ad03-ef19858bb2e9" (UID: "061866b2-700e-4b08-ad03-ef19858bb2e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:42:51.180680 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:51.180649    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/061866b2-700e-4b08-ad03-ef19858bb2e9-kube-api-access-vzkc6" pod "061866b2-700e-4b08-ad03-ef19858bb2e9" (UID: "061866b2-700e-4b08-ad03-ef19858bb2e9"). InnerVolumeSpecName "kube-api-access-vzkc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:42:51.180780 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:51.180747    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/061866b2-700e-4b08-ad03-ef19858bb2e9-maas-api-xks-ca" pod "061866b2-700e-4b08-ad03-ef19858bb2e9" (UID: "061866b2-700e-4b08-ad03-ef19858bb2e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:42:51.280216 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:51.280133    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzkc6\" (UniqueName: \"kubernetes.io/projected/061866b2-700e-4b08-ad03-ef19858bb2e9-kube-api-access-vzkc6\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:51.280216 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:51.280165    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/061866b2-700e-4b08-ad03-ef19858bb2e9-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:51.280216 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:51.280175    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/061866b2-700e-4b08-ad03-ef19858bb2e9-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:51.967028 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:51.966999    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-67f56e-zxf8z"
Oct 30 13:42:51.967230 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:51.966993    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-67f56e-zxf8z" event={"ID":"061866b2-700e-4b08-ad03-ef19858bb2e9","Type":"ContainerDied","Data":"932600b8e9c52ec155d619f130b52bf7234dd81ffdb6ee78ed6f5cbd255f1256"}
Oct 30 13:42:51.967230 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:51.967126    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="932600b8e9c52ec155d619f130b52bf7234dd81ffdb6ee78ed6f5cbd255f1256"
Oct 30 13:42:52.750555 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.750533    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:42:52.896044 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.896009    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/265b2410-a8b7-4d99-9cda-32565416a1d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/265b2410-a8b7-4d99-9cda-32565416a1d6-istiod-ca-cert\") pod \"265b2410-a8b7-4d99-9cda-32565416a1d6\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") "
Oct 30 13:42:52.896044 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.896046    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-istio-envoy\") pod \"265b2410-a8b7-4d99-9cda-32565416a1d6\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") "
Oct 30 13:42:52.896347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.896095    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/265b2410-a8b7-4d99-9cda-32565416a1d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/265b2410-a8b7-4d99-9cda-32565416a1d6-istio-podinfo\") pod \"265b2410-a8b7-4d99-9cda-32565416a1d6\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") "
Oct 30 13:42:52.896347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.896138    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/265b2410-a8b7-4d99-9cda-32565416a1d6-kube-api-access-7mgns\" (UniqueName: \"kubernetes.io/projected/265b2410-a8b7-4d99-9cda-32565416a1d6-kube-api-access-7mgns\") pod \"265b2410-a8b7-4d99-9cda-32565416a1d6\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") "
Oct 30 13:42:52.896347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.896180    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-istio-data\") pod \"265b2410-a8b7-4d99-9cda-32565416a1d6\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") "
Oct 30 13:42:52.896347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.896213    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-credential-socket\") pod \"265b2410-a8b7-4d99-9cda-32565416a1d6\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") "
Oct 30 13:42:52.896347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.896252    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-workload-certs\") pod \"265b2410-a8b7-4d99-9cda-32565416a1d6\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") "
Oct 30 13:42:52.896347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.896330    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/265b2410-a8b7-4d99-9cda-32565416a1d6-istio-token\" (UniqueName: \"kubernetes.io/projected/265b2410-a8b7-4d99-9cda-32565416a1d6-istio-token\") pod \"265b2410-a8b7-4d99-9cda-32565416a1d6\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") "
Oct 30 13:42:52.896682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.896390    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-workload-socket\") pod \"265b2410-a8b7-4d99-9cda-32565416a1d6\" (UID: \"265b2410-a8b7-4d99-9cda-32565416a1d6\") "
Oct 30 13:42:52.896682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.896472    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/265b2410-a8b7-4d99-9cda-32565416a1d6-istiod-ca-cert" pod "265b2410-a8b7-4d99-9cda-32565416a1d6" (UID: "265b2410-a8b7-4d99-9cda-32565416a1d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:42:52.896682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.896594    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-istio-data" pod "265b2410-a8b7-4d99-9cda-32565416a1d6" (UID: "265b2410-a8b7-4d99-9cda-32565416a1d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:52.896682 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.896620    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-workload-certs" pod "265b2410-a8b7-4d99-9cda-32565416a1d6" (UID: "265b2410-a8b7-4d99-9cda-32565416a1d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:52.896898 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.896707    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-workload-socket" pod "265b2410-a8b7-4d99-9cda-32565416a1d6" (UID: "265b2410-a8b7-4d99-9cda-32565416a1d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:52.896898 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.896812    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-credential-socket" pod "265b2410-a8b7-4d99-9cda-32565416a1d6" (UID: "265b2410-a8b7-4d99-9cda-32565416a1d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:52.897003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.896943    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:52.897003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.896963    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/265b2410-a8b7-4d99-9cda-32565416a1d6-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:52.897003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.896977    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:52.897003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.896992    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:52.897256 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.897006    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:52.898531 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.898504    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-istio-envoy" pod "265b2410-a8b7-4d99-9cda-32565416a1d6" (UID: "265b2410-a8b7-4d99-9cda-32565416a1d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:52.898632 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.898557    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/265b2410-a8b7-4d99-9cda-32565416a1d6-istio-podinfo" pod "265b2410-a8b7-4d99-9cda-32565416a1d6" (UID: "265b2410-a8b7-4d99-9cda-32565416a1d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:42:52.898632 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.898627    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/265b2410-a8b7-4d99-9cda-32565416a1d6-istio-token" pod "265b2410-a8b7-4d99-9cda-32565416a1d6" (UID: "265b2410-a8b7-4d99-9cda-32565416a1d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:42:52.898725 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.898651    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/265b2410-a8b7-4d99-9cda-32565416a1d6-kube-api-access-7mgns" pod "265b2410-a8b7-4d99-9cda-32565416a1d6" (UID: "265b2410-a8b7-4d99-9cda-32565416a1d6"). InnerVolumeSpecName "kube-api-access-7mgns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:42:52.901246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.901202    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:52.974263 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.974197    4544 generic.go:356] "Generic (PLEG): container finished" podID="265b2410-a8b7-4d99-9cda-32565416a1d6" containerID="da9dd40a34d315792924f330b490886541402d55bdab34f8150adce630fa519d" exitCode=0
Oct 30 13:42:52.974263 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.974267    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"
Oct 30 13:42:52.974485 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.974298    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5" event={"ID":"265b2410-a8b7-4d99-9cda-32565416a1d6","Type":"ContainerDied","Data":"da9dd40a34d315792924f330b490886541402d55bdab34f8150adce630fa519d"}
Oct 30 13:42:52.974485 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.974359    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5" event={"ID":"265b2410-a8b7-4d99-9cda-32565416a1d6","Type":"ContainerDied","Data":"635cc0bbccac4257950bb8cc39eae52967c1a6d9bf7c4e250890bdcb578b38a2"}
Oct 30 13:42:52.974485 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.974391    4544 scope.go:122] "RemoveContainer" containerID="da9dd40a34d315792924f330b490886541402d55bdab34f8150adce630fa519d"
Oct 30 13:42:52.985617 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.985596    4544 scope.go:122] "RemoveContainer" containerID="da9dd40a34d315792924f330b490886541402d55bdab34f8150adce630fa519d"
Oct 30 13:42:52.985931 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:42:52.985909    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da9dd40a34d315792924f330b490886541402d55bdab34f8150adce630fa519d\": container with ID starting with da9dd40a34d315792924f330b490886541402d55bdab34f8150adce630fa519d not found: ID does not exist" containerID="da9dd40a34d315792924f330b490886541402d55bdab34f8150adce630fa519d"
Oct 30 13:42:52.986012 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.985940    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da9dd40a34d315792924f330b490886541402d55bdab34f8150adce630fa519d"} err="failed to get container status \"da9dd40a34d315792924f330b490886541402d55bdab34f8150adce630fa519d\": rpc error: code = NotFound desc = could not find container \"da9dd40a34d315792924f330b490886541402d55bdab34f8150adce630fa519d\": container with ID starting with da9dd40a34d315792924f330b490886541402d55bdab34f8150adce630fa519d not found: ID does not exist"
Oct 30 13:42:52.998002 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.997982    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/265b2410-a8b7-4d99-9cda-32565416a1d6-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:52.998002 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.998002    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/265b2410-a8b7-4d99-9cda-32565416a1d6-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:52.998113 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.998011    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mgns\" (UniqueName: \"kubernetes.io/projected/265b2410-a8b7-4d99-9cda-32565416a1d6-kube-api-access-7mgns\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:52.998113 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:52.998022    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/265b2410-a8b7-4d99-9cda-32565416a1d6-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:42:53.027249 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:53.027203    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"]
Oct 30 13:42:53.034075 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:53.034049    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"]
Oct 30 13:42:53.047441 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:53.047415    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5"]
Oct 30 13:42:53.085853 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:53.085810    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="265b2410-a8b7-4d99-9cda-32565416a1d6" path="/var/lib/kubelet/pods/265b2410-a8b7-4d99-9cda-32565416a1d6/volumes"
Oct 30 13:42:53.148054 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:53.147976    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:54.693166 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:54.693125    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:56.582553 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:56.582518    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:58.252478 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:58.252427    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558" podUID="d7da56bd-b1ed-419f-b90d-c9a68a8f756d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:42:58.252885 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:58.252435    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q" podUID="b63682f2-c062-4ef4-801f-b84a3a7c10e0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:42:58.355696 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:58.355659    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:58.681691 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:58.681655    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj"]
Oct 30 13:42:58.751051 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:58.750957    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:58.858951 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:58.858918    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:42:59.007741 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:42:59.007645    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:42:59.836158 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:59.836117    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bc33589-vz7cz"]
Oct 30 13:42:59.837145 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:59.837114    4544 cpu_manager.go:414] "containerMap: removing container" podUID="061866b2-700e-4b08-ad03-ef19858bb2e9" containerName="revoke-keys"
Oct 30 13:42:59.837145 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:59.837142    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="061866b2-700e-4b08-ad03-ef19858bb2e9" containerName="revoke-keys"
Oct 30 13:42:59.837330 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:59.837163    4544 cpu_manager.go:414] "containerMap: removing container" podUID="265b2410-a8b7-4d99-9cda-32565416a1d6" containerName="istio-proxy"
Oct 30 13:42:59.837330 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:59.837169    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="265b2410-a8b7-4d99-9cda-32565416a1d6" containerName="istio-proxy"
Oct 30 13:42:59.837330 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:59.837270    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="061866b2-700e-4b08-ad03-ef19858bb2e9" containerName="revoke-keys"
Oct 30 13:42:59.837330 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:59.837280    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="265b2410-a8b7-4d99-9cda-32565416a1d6" containerName="istio-proxy"
Oct 30 13:42:59.841956 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:59.841918    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bc33589-vz7cz"
Oct 30 13:42:59.871320 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:59.871262    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:42:59.871649 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:59.871628    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:42:59.875283 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:59.875247    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bc33589-vz7cz"]
Oct 30 13:42:59.962323 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:59.962279    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ebcb91d-6f3d-4f42-b029-0affb4abbbf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bc33589-vz7cz\" (UID: \"4ebcb91d-6f3d-4f42-b029-0affb4abbbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bc33589-vz7cz"
Oct 30 13:42:59.962323 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:59.962327    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46dtx\" (UniqueName: \"kubernetes.io/projected/4ebcb91d-6f3d-4f42-b029-0affb4abbbf9-kube-api-access-46dtx\") pod \"maas-api-revoke-sub-4bc33589-vz7cz\" (UID: \"4ebcb91d-6f3d-4f42-b029-0affb4abbbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bc33589-vz7cz"
Oct 30 13:42:59.962566 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:42:59.962462    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ebcb91d-6f3d-4f42-b029-0affb4abbbf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bc33589-vz7cz\" (UID: \"4ebcb91d-6f3d-4f42-b029-0affb4abbbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bc33589-vz7cz"
Oct 30 13:43:00.063683 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:00.063635    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ebcb91d-6f3d-4f42-b029-0affb4abbbf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bc33589-vz7cz\" (UID: \"4ebcb91d-6f3d-4f42-b029-0affb4abbbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bc33589-vz7cz"
Oct 30 13:43:00.063890 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:00.063770    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ebcb91d-6f3d-4f42-b029-0affb4abbbf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bc33589-vz7cz\" (UID: \"4ebcb91d-6f3d-4f42-b029-0affb4abbbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bc33589-vz7cz"
Oct 30 13:43:00.063890 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:00.063806    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46dtx\" (UniqueName: \"kubernetes.io/projected/4ebcb91d-6f3d-4f42-b029-0affb4abbbf9-kube-api-access-46dtx\") pod \"maas-api-revoke-sub-4bc33589-vz7cz\" (UID: \"4ebcb91d-6f3d-4f42-b029-0affb4abbbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bc33589-vz7cz"
Oct 30 13:43:00.064533 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:00.064506    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ebcb91d-6f3d-4f42-b029-0affb4abbbf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bc33589-vz7cz\" (UID: \"4ebcb91d-6f3d-4f42-b029-0affb4abbbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bc33589-vz7cz"
Oct 30 13:43:00.066245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:00.066206    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ebcb91d-6f3d-4f42-b029-0affb4abbbf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bc33589-vz7cz\" (UID: \"4ebcb91d-6f3d-4f42-b029-0affb4abbbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bc33589-vz7cz"
Oct 30 13:43:00.088060 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:00.087977    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46dtx\" (UniqueName: \"kubernetes.io/projected/4ebcb91d-6f3d-4f42-b029-0affb4abbbf9-kube-api-access-46dtx\") pod \"maas-api-revoke-sub-4bc33589-vz7cz\" (UID: \"4ebcb91d-6f3d-4f42-b029-0affb4abbbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bc33589-vz7cz"
Oct 30 13:43:00.157451 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:00.157417    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bc33589-vz7cz"
Oct 30 13:43:00.468415 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:00.468330    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bc33589-vz7cz"]
Oct 30 13:43:00.472072 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:43:00.472036    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ebcb91d_6f3d_4f42_b029_0affb4abbbf9.slice/crio-017a7754c94ad9cabd6455ef27f9072ba633b01e3b5dbc108a78be73d997b1bc WatchSource:0}: Error finding container 017a7754c94ad9cabd6455ef27f9072ba633b01e3b5dbc108a78be73d997b1bc: Status 404 returned error can't find the container with id 017a7754c94ad9cabd6455ef27f9072ba633b01e3b5dbc108a78be73d997b1bc
Oct 30 13:43:01.039247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:01.039182    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bc33589-vz7cz" event={"ID":"4ebcb91d-6f3d-4f42-b029-0affb4abbbf9","Type":"ContainerStarted","Data":"c2de7b432fa7efa6a832e3aebddc87a637e0c4e6f2118477024c467febbf1c65"}
Oct 30 13:43:01.039711 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:01.039329    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bc33589-vz7cz" event={"ID":"4ebcb91d-6f3d-4f42-b029-0affb4abbbf9","Type":"ContainerStarted","Data":"017a7754c94ad9cabd6455ef27f9072ba633b01e3b5dbc108a78be73d997b1bc"}
Oct 30 13:43:01.083817 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:01.083765    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bc33589-vz7cz" podStartSLOduration=2.083750794 podStartE2EDuration="2.083750794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:43:01.080757089 +0000 UTC m=+3476.127891043" watchObservedRunningTime="2026-10-30 13:43:01.083750794 +0000 UTC m=+3476.130884746"
Oct 30 13:43:01.275374 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:01.275343    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q"
Oct 30 13:43:01.278181 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:01.278156    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558"
Oct 30 13:43:01.375636 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:01.375595    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7da56bd-b1ed-419f-b90d-c9a68a8f756d-kube-api-access-lxb5r\" (UniqueName: \"kubernetes.io/projected/d7da56bd-b1ed-419f-b90d-c9a68a8f756d-kube-api-access-lxb5r\") pod \"d7da56bd-b1ed-419f-b90d-c9a68a8f756d\" (UID: \"d7da56bd-b1ed-419f-b90d-c9a68a8f756d\") "
Oct 30 13:43:01.375827 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:01.375720    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b63682f2-c062-4ef4-801f-b84a3a7c10e0-kube-api-access-2kpfg\" (UniqueName: \"kubernetes.io/projected/b63682f2-c062-4ef4-801f-b84a3a7c10e0-kube-api-access-2kpfg\") pod \"b63682f2-c062-4ef4-801f-b84a3a7c10e0\" (UID: \"b63682f2-c062-4ef4-801f-b84a3a7c10e0\") "
Oct 30 13:43:01.375827 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:01.375789    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b63682f2-c062-4ef4-801f-b84a3a7c10e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b63682f2-c062-4ef4-801f-b84a3a7c10e0-plugins-config-volume\") pod \"b63682f2-c062-4ef4-801f-b84a3a7c10e0\" (UID: \"b63682f2-c062-4ef4-801f-b84a3a7c10e0\") "
Oct 30 13:43:01.375827 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:01.375818    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7da56bd-b1ed-419f-b90d-c9a68a8f756d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7da56bd-b1ed-419f-b90d-c9a68a8f756d-plugins-config-volume\") pod \"d7da56bd-b1ed-419f-b90d-c9a68a8f756d\" (UID: \"d7da56bd-b1ed-419f-b90d-c9a68a8f756d\") "
Oct 30 13:43:01.376209 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:01.376178    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b63682f2-c062-4ef4-801f-b84a3a7c10e0-plugins-config-volume" pod "b63682f2-c062-4ef4-801f-b84a3a7c10e0" (UID: "b63682f2-c062-4ef4-801f-b84a3a7c10e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:43:01.376374 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:01.376212    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7da56bd-b1ed-419f-b90d-c9a68a8f756d-plugins-config-volume" pod "d7da56bd-b1ed-419f-b90d-c9a68a8f756d" (UID: "d7da56bd-b1ed-419f-b90d-c9a68a8f756d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:43:01.377858 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:01.377824    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7da56bd-b1ed-419f-b90d-c9a68a8f756d-kube-api-access-lxb5r" pod "d7da56bd-b1ed-419f-b90d-c9a68a8f756d" (UID: "d7da56bd-b1ed-419f-b90d-c9a68a8f756d"). InnerVolumeSpecName "kube-api-access-lxb5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:01.377953 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:01.377857    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b63682f2-c062-4ef4-801f-b84a3a7c10e0-kube-api-access-2kpfg" pod "b63682f2-c062-4ef4-801f-b84a3a7c10e0" (UID: "b63682f2-c062-4ef4-801f-b84a3a7c10e0"). InnerVolumeSpecName "kube-api-access-2kpfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:01.477014 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:01.476974    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b63682f2-c062-4ef4-801f-b84a3a7c10e0-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:43:01.477014 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:01.477008    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7da56bd-b1ed-419f-b90d-c9a68a8f756d-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:43:01.477014 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:01.477018    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxb5r\" (UniqueName: \"kubernetes.io/projected/d7da56bd-b1ed-419f-b90d-c9a68a8f756d-kube-api-access-lxb5r\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:43:01.477296 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:01.477029    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kpfg\" (UniqueName: \"kubernetes.io/projected/b63682f2-c062-4ef4-801f-b84a3a7c10e0-kube-api-access-2kpfg\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:43:02.046888 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:02.046852    4544 generic.go:356] "Generic (PLEG): container finished" podID="4ebcb91d-6f3d-4f42-b029-0affb4abbbf9" containerID="c2de7b432fa7efa6a832e3aebddc87a637e0c4e6f2118477024c467febbf1c65" exitCode=0
Oct 30 13:43:02.047377 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:02.046936    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bc33589-vz7cz" event={"ID":"4ebcb91d-6f3d-4f42-b029-0affb4abbbf9","Type":"ContainerDied","Data":"c2de7b432fa7efa6a832e3aebddc87a637e0c4e6f2118477024c467febbf1c65"}
Oct 30 13:43:02.048535 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:02.048510    4544 generic.go:356] "Generic (PLEG): container finished" podID="b63682f2-c062-4ef4-801f-b84a3a7c10e0" containerID="c0dad761436aeb0a5652b16312563ee313497554ada9a11e09b8a647a7cd5206" exitCode=0
Oct 30 13:43:02.048667 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:02.048553    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q"
Oct 30 13:43:02.048667 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:02.048590    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q" event={"ID":"b63682f2-c062-4ef4-801f-b84a3a7c10e0","Type":"ContainerDied","Data":"c0dad761436aeb0a5652b16312563ee313497554ada9a11e09b8a647a7cd5206"}
Oct 30 13:43:02.048667 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:02.048644    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q" event={"ID":"b63682f2-c062-4ef4-801f-b84a3a7c10e0","Type":"ContainerDied","Data":"291f2b2a330606457866e60696010efed5e924766a60b6053f764666de074c3f"}
Oct 30 13:43:02.048868 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:02.048675    4544 scope.go:122] "RemoveContainer" containerID="c0dad761436aeb0a5652b16312563ee313497554ada9a11e09b8a647a7cd5206"
Oct 30 13:43:02.050029 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:02.050005    4544 generic.go:356] "Generic (PLEG): container finished" podID="d7da56bd-b1ed-419f-b90d-c9a68a8f756d" containerID="10566545545ef4c2c6cb6c768c48d6805dcb63683bb3e1bfa783b1851e71f7c2" exitCode=0
Oct 30 13:43:02.050154 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:02.050048    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558"
Oct 30 13:43:02.050154 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:02.050064    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558" event={"ID":"d7da56bd-b1ed-419f-b90d-c9a68a8f756d","Type":"ContainerDied","Data":"10566545545ef4c2c6cb6c768c48d6805dcb63683bb3e1bfa783b1851e71f7c2"}
Oct 30 13:43:02.050154 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:02.050103    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558" event={"ID":"d7da56bd-b1ed-419f-b90d-c9a68a8f756d","Type":"ContainerDied","Data":"c82d1096b6617c7f92fd0ddbcdaf1c67c2c1a930cb1c7ed305840037ed0a2e2a"}
Oct 30 13:43:02.059154 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:02.059131    4544 scope.go:122] "RemoveContainer" containerID="c0dad761436aeb0a5652b16312563ee313497554ada9a11e09b8a647a7cd5206"
Oct 30 13:43:02.059611 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:43:02.059485    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0dad761436aeb0a5652b16312563ee313497554ada9a11e09b8a647a7cd5206\": container with ID starting with c0dad761436aeb0a5652b16312563ee313497554ada9a11e09b8a647a7cd5206 not found: ID does not exist" containerID="c0dad761436aeb0a5652b16312563ee313497554ada9a11e09b8a647a7cd5206"
Oct 30 13:43:02.059611 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:02.059515    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0dad761436aeb0a5652b16312563ee313497554ada9a11e09b8a647a7cd5206"} err="failed to get container status \"c0dad761436aeb0a5652b16312563ee313497554ada9a11e09b8a647a7cd5206\": rpc error: code = NotFound desc = could not find container \"c0dad761436aeb0a5652b16312563ee313497554ada9a11e09b8a647a7cd5206\": container with ID starting with c0dad761436aeb0a5652b16312563ee313497554ada9a11e09b8a647a7cd5206 not found: ID does not exist"
Oct 30 13:43:02.059611 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:02.059535    4544 scope.go:122] "RemoveContainer" containerID="10566545545ef4c2c6cb6c768c48d6805dcb63683bb3e1bfa783b1851e71f7c2"
Oct 30 13:43:02.069885 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:02.069862    4544 scope.go:122] "RemoveContainer" containerID="10566545545ef4c2c6cb6c768c48d6805dcb63683bb3e1bfa783b1851e71f7c2"
Oct 30 13:43:02.070193 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:43:02.070176    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10566545545ef4c2c6cb6c768c48d6805dcb63683bb3e1bfa783b1851e71f7c2\": container with ID starting with 10566545545ef4c2c6cb6c768c48d6805dcb63683bb3e1bfa783b1851e71f7c2 not found: ID does not exist" containerID="10566545545ef4c2c6cb6c768c48d6805dcb63683bb3e1bfa783b1851e71f7c2"
Oct 30 13:43:02.070263 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:02.070202    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10566545545ef4c2c6cb6c768c48d6805dcb63683bb3e1bfa783b1851e71f7c2"} err="failed to get container status \"10566545545ef4c2c6cb6c768c48d6805dcb63683bb3e1bfa783b1851e71f7c2\": rpc error: code = NotFound desc = could not find container \"10566545545ef4c2c6cb6c768c48d6805dcb63683bb3e1bfa783b1851e71f7c2\": container with ID starting with 10566545545ef4c2c6cb6c768c48d6805dcb63683bb3e1bfa783b1851e71f7c2 not found: ID does not exist"
Oct 30 13:43:02.127323 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:02.127282    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q"]
Oct 30 13:43:02.133916 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:02.133872    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q"]
Oct 30 13:43:02.172974 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:02.172941    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558"]
Oct 30 13:43:02.177183 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:02.177146    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558"]
Oct 30 13:43:03.084655 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:03.084615    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b63682f2-c062-4ef4-801f-b84a3a7c10e0" path="/var/lib/kubelet/pods/b63682f2-c062-4ef4-801f-b84a3a7c10e0/volumes"
Oct 30 13:43:03.085252 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:03.085231    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7da56bd-b1ed-419f-b90d-c9a68a8f756d" path="/var/lib/kubelet/pods/d7da56bd-b1ed-419f-b90d-c9a68a8f756d/volumes"
Oct 30 13:43:03.200024 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:03.199999    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bc33589-vz7cz"
Oct 30 13:43:03.394130 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:03.394082    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ebcb91d-6f3d-4f42-b029-0affb4abbbf9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ebcb91d-6f3d-4f42-b029-0affb4abbbf9-maas-api-xks-ca\") pod \"4ebcb91d-6f3d-4f42-b029-0affb4abbbf9\" (UID: \"4ebcb91d-6f3d-4f42-b029-0affb4abbbf9\") "
Oct 30 13:43:03.394378 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:03.394150    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ebcb91d-6f3d-4f42-b029-0affb4abbbf9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ebcb91d-6f3d-4f42-b029-0affb4abbbf9-maas-api-openshift-ca\") pod \"4ebcb91d-6f3d-4f42-b029-0affb4abbbf9\" (UID: \"4ebcb91d-6f3d-4f42-b029-0affb4abbbf9\") "
Oct 30 13:43:03.394378 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:03.394193    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ebcb91d-6f3d-4f42-b029-0affb4abbbf9-kube-api-access-46dtx\" (UniqueName: \"kubernetes.io/projected/4ebcb91d-6f3d-4f42-b029-0affb4abbbf9-kube-api-access-46dtx\") pod \"4ebcb91d-6f3d-4f42-b029-0affb4abbbf9\" (UID: \"4ebcb91d-6f3d-4f42-b029-0affb4abbbf9\") "
Oct 30 13:43:03.394602 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:03.394577    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ebcb91d-6f3d-4f42-b029-0affb4abbbf9-maas-api-openshift-ca" pod "4ebcb91d-6f3d-4f42-b029-0affb4abbbf9" (UID: "4ebcb91d-6f3d-4f42-b029-0affb4abbbf9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:43:03.396417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:03.396392    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ebcb91d-6f3d-4f42-b029-0affb4abbbf9-kube-api-access-46dtx" pod "4ebcb91d-6f3d-4f42-b029-0affb4abbbf9" (UID: "4ebcb91d-6f3d-4f42-b029-0affb4abbbf9"). InnerVolumeSpecName "kube-api-access-46dtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:03.396513 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:03.396396    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ebcb91d-6f3d-4f42-b029-0affb4abbbf9-maas-api-xks-ca" pod "4ebcb91d-6f3d-4f42-b029-0affb4abbbf9" (UID: "4ebcb91d-6f3d-4f42-b029-0affb4abbbf9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:43:03.494944 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:03.494906    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ebcb91d-6f3d-4f42-b029-0affb4abbbf9-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:43:03.494944 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:03.494942    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ebcb91d-6f3d-4f42-b029-0affb4abbbf9-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:43:03.494944 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:03.494952    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46dtx\" (UniqueName: \"kubernetes.io/projected/4ebcb91d-6f3d-4f42-b029-0affb4abbbf9-kube-api-access-46dtx\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:43:03.682720 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:03.682604    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj" podUID="52424dad-1898-48d3-9529-3dc7f16cb1e5" containerName="payload-pre-processing" containerID="cri-o://529296cefffe234a9b3ab3d24164b1e366d0dcea32cce448f93f35ca2fb2a87a" gracePeriod=25
Oct 30 13:43:04.065906 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:04.065818    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bc33589-vz7cz"
Oct 30 13:43:04.065906 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:04.065840    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bc33589-vz7cz" event={"ID":"4ebcb91d-6f3d-4f42-b029-0affb4abbbf9","Type":"ContainerDied","Data":"017a7754c94ad9cabd6455ef27f9072ba633b01e3b5dbc108a78be73d997b1bc"}
Oct 30 13:43:04.065906 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:04.065890    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="017a7754c94ad9cabd6455ef27f9072ba633b01e3b5dbc108a78be73d997b1bc"
Oct 30 13:43:04.765111 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:04.765073    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj" podUID="52424dad-1898-48d3-9529-3dc7f16cb1e5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:43:08.048093 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:08.048030    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main" containerID="cri-o://09dcb1cc0ac2ab86742504cee72d627a19258fea0c0329501902ecb714d95d98" gracePeriod=45
Oct 30 13:43:08.300243 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:08.300161    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:43:08.335407 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:08.335369    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8be6bd4f-3790-489b-8a6e-edcd85f33aec-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8be6bd4f-3790-489b-8a6e-edcd85f33aec-dshm\") pod \"8be6bd4f-3790-489b-8a6e-edcd85f33aec\" (UID: \"8be6bd4f-3790-489b-8a6e-edcd85f33aec\") "
Oct 30 13:43:08.335602 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:08.335451    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8be6bd4f-3790-489b-8a6e-edcd85f33aec-kube-api-access-kjclj\" (UniqueName: \"kubernetes.io/projected/8be6bd4f-3790-489b-8a6e-edcd85f33aec-kube-api-access-kjclj\") pod \"8be6bd4f-3790-489b-8a6e-edcd85f33aec\" (UID: \"8be6bd4f-3790-489b-8a6e-edcd85f33aec\") "
Oct 30 13:43:08.335602 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:08.335504    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8be6bd4f-3790-489b-8a6e-edcd85f33aec-home\" (UniqueName: \"kubernetes.io/empty-dir/8be6bd4f-3790-489b-8a6e-edcd85f33aec-home\") pod \"8be6bd4f-3790-489b-8a6e-edcd85f33aec\" (UID: \"8be6bd4f-3790-489b-8a6e-edcd85f33aec\") "
Oct 30 13:43:08.335602 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:08.335534    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8be6bd4f-3790-489b-8a6e-edcd85f33aec-tls-certs\" (UniqueName: \"kubernetes.io/secret/8be6bd4f-3790-489b-8a6e-edcd85f33aec-tls-certs\") pod \"8be6bd4f-3790-489b-8a6e-edcd85f33aec\" (UID: \"8be6bd4f-3790-489b-8a6e-edcd85f33aec\") "
Oct 30 13:43:08.335805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:08.335672    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8be6bd4f-3790-489b-8a6e-edcd85f33aec-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8be6bd4f-3790-489b-8a6e-edcd85f33aec-model-cache\") pod \"8be6bd4f-3790-489b-8a6e-edcd85f33aec\" (UID: \"8be6bd4f-3790-489b-8a6e-edcd85f33aec\") "
Oct 30 13:43:08.335805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:08.335765    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8be6bd4f-3790-489b-8a6e-edcd85f33aec-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8be6bd4f-3790-489b-8a6e-edcd85f33aec-tmp-dir\") pod \"8be6bd4f-3790-489b-8a6e-edcd85f33aec\" (UID: \"8be6bd4f-3790-489b-8a6e-edcd85f33aec\") "
Oct 30 13:43:08.335913 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:08.335795    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be6bd4f-3790-489b-8a6e-edcd85f33aec-home" pod "8be6bd4f-3790-489b-8a6e-edcd85f33aec" (UID: "8be6bd4f-3790-489b-8a6e-edcd85f33aec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:08.335985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:08.335960    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be6bd4f-3790-489b-8a6e-edcd85f33aec-model-cache" pod "8be6bd4f-3790-489b-8a6e-edcd85f33aec" (UID: "8be6bd4f-3790-489b-8a6e-edcd85f33aec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:08.336143 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:08.336122    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be6bd4f-3790-489b-8a6e-edcd85f33aec-tmp-dir" pod "8be6bd4f-3790-489b-8a6e-edcd85f33aec" (UID: "8be6bd4f-3790-489b-8a6e-edcd85f33aec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:08.336240 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:08.336192    4544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8be6bd4f-3790-489b-8a6e-edcd85f33aec-home\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:43:08.336240 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:08.336212    4544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8be6bd4f-3790-489b-8a6e-edcd85f33aec-model-cache\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:43:08.338018 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:08.337991    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be6bd4f-3790-489b-8a6e-edcd85f33aec-dshm" pod "8be6bd4f-3790-489b-8a6e-edcd85f33aec" (UID: "8be6bd4f-3790-489b-8a6e-edcd85f33aec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:08.338244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:08.338198    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8be6bd4f-3790-489b-8a6e-edcd85f33aec-tls-certs" pod "8be6bd4f-3790-489b-8a6e-edcd85f33aec" (UID: "8be6bd4f-3790-489b-8a6e-edcd85f33aec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:43:08.338367 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:08.338285    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be6bd4f-3790-489b-8a6e-edcd85f33aec-kube-api-access-kjclj" pod "8be6bd4f-3790-489b-8a6e-edcd85f33aec" (UID: "8be6bd4f-3790-489b-8a6e-edcd85f33aec"). InnerVolumeSpecName "kube-api-access-kjclj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:08.437328 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:08.437292    4544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8be6bd4f-3790-489b-8a6e-edcd85f33aec-tls-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:43:08.437328 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:08.437326    4544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8be6bd4f-3790-489b-8a6e-edcd85f33aec-tmp-dir\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:43:08.437328 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:08.437337    4544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8be6bd4f-3790-489b-8a6e-edcd85f33aec-dshm\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:43:08.437575 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:08.437349    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjclj\" (UniqueName: \"kubernetes.io/projected/8be6bd4f-3790-489b-8a6e-edcd85f33aec-kube-api-access-kjclj\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:43:09.097978 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:09.097949    4544 generic.go:356] "Generic (PLEG): container finished" podID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerID="09dcb1cc0ac2ab86742504cee72d627a19258fea0c0329501902ecb714d95d98" exitCode=0
Oct 30 13:43:09.098563 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:09.098000    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"
Oct 30 13:43:09.098563 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:09.098026    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" event={"ID":"8be6bd4f-3790-489b-8a6e-edcd85f33aec","Type":"ContainerDied","Data":"09dcb1cc0ac2ab86742504cee72d627a19258fea0c0329501902ecb714d95d98"}
Oct 30 13:43:09.098563 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:09.098069    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs" event={"ID":"8be6bd4f-3790-489b-8a6e-edcd85f33aec","Type":"ContainerDied","Data":"9c9f4939212f95284c056da0e3737f338d2b0d1053adc89bc419aef3ec4f41b9"}
Oct 30 13:43:09.098563 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:09.098086    4544 scope.go:122] "RemoveContainer" containerID="09dcb1cc0ac2ab86742504cee72d627a19258fea0c0329501902ecb714d95d98"
Oct 30 13:43:09.107386 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:09.107369    4544 scope.go:122] "RemoveContainer" containerID="09dcb1cc0ac2ab86742504cee72d627a19258fea0c0329501902ecb714d95d98"
Oct 30 13:43:09.107692 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:43:09.107672    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09dcb1cc0ac2ab86742504cee72d627a19258fea0c0329501902ecb714d95d98\": container with ID starting with 09dcb1cc0ac2ab86742504cee72d627a19258fea0c0329501902ecb714d95d98 not found: ID does not exist" containerID="09dcb1cc0ac2ab86742504cee72d627a19258fea0c0329501902ecb714d95d98"
Oct 30 13:43:09.107752 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:09.107702    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09dcb1cc0ac2ab86742504cee72d627a19258fea0c0329501902ecb714d95d98"} err="failed to get container status \"09dcb1cc0ac2ab86742504cee72d627a19258fea0c0329501902ecb714d95d98\": rpc error: code = NotFound desc = could not find container \"09dcb1cc0ac2ab86742504cee72d627a19258fea0c0329501902ecb714d95d98\": container with ID starting with 09dcb1cc0ac2ab86742504cee72d627a19258fea0c0329501902ecb714d95d98 not found: ID does not exist"
Oct 30 13:43:09.151007 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:09.150972    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"]
Oct 30 13:43:09.163072 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:09.163040    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-67f56e/premium-w4-67f56e-kserve-5dcb66fdff-4jdgs"]
Oct 30 13:43:11.084436 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:11.084393    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" path="/var/lib/kubelet/pods/8be6bd4f-3790-489b-8a6e-edcd85f33aec/volumes"
Oct 30 13:43:14.764370 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:14.764326    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj" podUID="52424dad-1898-48d3-9529-3dc7f16cb1e5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:43:15.047176 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:15.047099    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:43:15.319195 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:15.319106    4544 scope.go:122] "RemoveContainer" containerID="26de7e75310929543da0a8bb10e9357e2fb9a70ead5c13c47387ca3ed4aefbab"
Oct 30 13:43:15.329143 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:15.329116    4544 scope.go:122] "RemoveContainer" containerID="2757fbe46f11f9f98bce167c7440ac4dc930f8af11ea09b30798b3fd608a3470"
Oct 30 13:43:15.672318 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:15.672271    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:43:15.766589 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:15.766556    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-67f56e-zxf8z"]
Oct 30 13:43:15.778664 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:15.778622    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-67f56e-zxf8z"]
Oct 30 13:43:15.797448 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:15.797412    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:43:16.293384 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:16.293349    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:43:17.084410 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:17.084377    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="061866b2-700e-4b08-ad03-ef19858bb2e9" path="/var/lib/kubelet/pods/061866b2-700e-4b08-ad03-ef19858bb2e9/volumes"
Oct 30 13:43:22.781507 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:22.781470    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:43:22.907547 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:22.907505    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:43:23.134899 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:23.134859    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"]
Oct 30 13:43:23.135129 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:23.135073    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn" podUID="7f93aa79-27fa-4f24-9b20-b6da0dee9ec2" containerName="istio-proxy" containerID="cri-o://91258d98de4e7bf522bcc4e337e678335fb6d5c3a18aa8cb6f2629fb36e640a2" gracePeriod=30
Oct 30 13:43:23.207680 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:23.207617    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:43:23.944542 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:23.944459    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj"
Oct 30 13:43:23.995339 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:23.995300    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52424dad-1898-48d3-9529-3dc7f16cb1e5-kube-api-access-5qgwk\" (UniqueName: \"kubernetes.io/projected/52424dad-1898-48d3-9529-3dc7f16cb1e5-kube-api-access-5qgwk\") pod \"52424dad-1898-48d3-9529-3dc7f16cb1e5\" (UID: \"52424dad-1898-48d3-9529-3dc7f16cb1e5\") "
Oct 30 13:43:23.995531 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:23.995423    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52424dad-1898-48d3-9529-3dc7f16cb1e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52424dad-1898-48d3-9529-3dc7f16cb1e5-plugins-config-volume\") pod \"52424dad-1898-48d3-9529-3dc7f16cb1e5\" (UID: \"52424dad-1898-48d3-9529-3dc7f16cb1e5\") "
Oct 30 13:43:23.995890 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:23.995856    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52424dad-1898-48d3-9529-3dc7f16cb1e5-plugins-config-volume" pod "52424dad-1898-48d3-9529-3dc7f16cb1e5" (UID: "52424dad-1898-48d3-9529-3dc7f16cb1e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:43:23.998455 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:23.998422    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52424dad-1898-48d3-9529-3dc7f16cb1e5-kube-api-access-5qgwk" pod "52424dad-1898-48d3-9529-3dc7f16cb1e5" (UID: "52424dad-1898-48d3-9529-3dc7f16cb1e5"). InnerVolumeSpecName "kube-api-access-5qgwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:24.096240 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:24.096194    4544 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52424dad-1898-48d3-9529-3dc7f16cb1e5-plugins-config-volume\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:43:24.096434 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:24.096251    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qgwk\" (UniqueName: \"kubernetes.io/projected/52424dad-1898-48d3-9529-3dc7f16cb1e5-kube-api-access-5qgwk\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:43:24.192493 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:24.192457    4544 generic.go:356] "Generic (PLEG): container finished" podID="52424dad-1898-48d3-9529-3dc7f16cb1e5" containerID="529296cefffe234a9b3ab3d24164b1e366d0dcea32cce448f93f35ca2fb2a87a" exitCode=0
Oct 30 13:43:24.192660 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:24.192515    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj"
Oct 30 13:43:24.192660 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:24.192548    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj" event={"ID":"52424dad-1898-48d3-9529-3dc7f16cb1e5","Type":"ContainerDied","Data":"529296cefffe234a9b3ab3d24164b1e366d0dcea32cce448f93f35ca2fb2a87a"}
Oct 30 13:43:24.192660 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:24.192609    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj" event={"ID":"52424dad-1898-48d3-9529-3dc7f16cb1e5","Type":"ContainerDied","Data":"841a05d578ac07f2668ad444398c731ee8fe72df4915f9973bfd0162fc11c16e"}
Oct 30 13:43:24.192660 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:24.192633    4544 scope.go:122] "RemoveContainer" containerID="529296cefffe234a9b3ab3d24164b1e366d0dcea32cce448f93f35ca2fb2a87a"
Oct 30 13:43:24.201999 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:24.201980    4544 scope.go:122] "RemoveContainer" containerID="529296cefffe234a9b3ab3d24164b1e366d0dcea32cce448f93f35ca2fb2a87a"
Oct 30 13:43:24.202318 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:43:24.202291    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"529296cefffe234a9b3ab3d24164b1e366d0dcea32cce448f93f35ca2fb2a87a\": container with ID starting with 529296cefffe234a9b3ab3d24164b1e366d0dcea32cce448f93f35ca2fb2a87a not found: ID does not exist" containerID="529296cefffe234a9b3ab3d24164b1e366d0dcea32cce448f93f35ca2fb2a87a"
Oct 30 13:43:24.202388 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:24.202327    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"529296cefffe234a9b3ab3d24164b1e366d0dcea32cce448f93f35ca2fb2a87a"} err="failed to get container status \"529296cefffe234a9b3ab3d24164b1e366d0dcea32cce448f93f35ca2fb2a87a\": rpc error: code = NotFound desc = could not find container \"529296cefffe234a9b3ab3d24164b1e366d0dcea32cce448f93f35ca2fb2a87a\": container with ID starting with 529296cefffe234a9b3ab3d24164b1e366d0dcea32cce448f93f35ca2fb2a87a not found: ID does not exist"
Oct 30 13:43:24.247162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:24.247124    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj"]
Oct 30 13:43:24.249186 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:24.249161    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj"]
Oct 30 13:43:25.085897 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:25.085855    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52424dad-1898-48d3-9529-3dc7f16cb1e5" path="/var/lib/kubelet/pods/52424dad-1898-48d3-9529-3dc7f16cb1e5/volumes"
Oct 30 13:43:28.250765 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.250710    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75d8bff59-xxtf4"]
Oct 30 13:43:28.251266 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.251008    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-75d8bff59-xxtf4" podUID="f9ef7c6e-9978-477c-be85-5816cf030b20" containerName="manager" containerID="cri-o://027309718f22a19a12793b4cc725756604a6144b25790a21332a3075bb524e07" gracePeriod=30
Oct 30 13:43:28.405041 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.405002    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:43:28.515426 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.515396    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75d8bff59-xxtf4"
Oct 30 13:43:28.537685 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.537648    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-kube-api-access-dtp64\" (UniqueName: \"kubernetes.io/projected/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-kube-api-access-dtp64\") pod \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") "
Oct 30 13:43:28.537922 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.537701    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istio-data\") pod \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") "
Oct 30 13:43:28.537922 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.537748    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istio-podinfo\") pod \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") "
Oct 30 13:43:28.537922 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.537808    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-workload-socket\") pod \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") "
Oct 30 13:43:28.537922 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.537862    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istiod-ca-cert\") pod \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") "
Oct 30 13:43:28.537922 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.537909    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-workload-certs\") pod \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") "
Oct 30 13:43:28.538245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.537948    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-credential-socket\") pod \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") "
Oct 30 13:43:28.538245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.537993    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istio-envoy\") pod \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") "
Oct 30 13:43:28.538245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.538045    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istio-token\" (UniqueName: \"kubernetes.io/projected/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istio-token\") pod \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\" (UID: \"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2\") "
Oct 30 13:43:28.538439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.538318    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-credential-socket" pod "7f93aa79-27fa-4f24-9b20-b6da0dee9ec2" (UID: "7f93aa79-27fa-4f24-9b20-b6da0dee9ec2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:28.539009 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.538563    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istio-data" pod "7f93aa79-27fa-4f24-9b20-b6da0dee9ec2" (UID: "7f93aa79-27fa-4f24-9b20-b6da0dee9ec2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:28.539009 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.538963    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-workload-socket" pod "7f93aa79-27fa-4f24-9b20-b6da0dee9ec2" (UID: "7f93aa79-27fa-4f24-9b20-b6da0dee9ec2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:28.539274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.539090    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-workload-certs" pod "7f93aa79-27fa-4f24-9b20-b6da0dee9ec2" (UID: "7f93aa79-27fa-4f24-9b20-b6da0dee9ec2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:28.539350 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.539277    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istiod-ca-cert" pod "7f93aa79-27fa-4f24-9b20-b6da0dee9ec2" (UID: "7f93aa79-27fa-4f24-9b20-b6da0dee9ec2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:43:28.541030 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.540991    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istio-envoy" pod "7f93aa79-27fa-4f24-9b20-b6da0dee9ec2" (UID: "7f93aa79-27fa-4f24-9b20-b6da0dee9ec2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:28.541299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.541261    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istio-podinfo" pod "7f93aa79-27fa-4f24-9b20-b6da0dee9ec2" (UID: "7f93aa79-27fa-4f24-9b20-b6da0dee9ec2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:43:28.541408 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.541314    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istio-token" pod "7f93aa79-27fa-4f24-9b20-b6da0dee9ec2" (UID: "7f93aa79-27fa-4f24-9b20-b6da0dee9ec2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:28.541747 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.541725    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-kube-api-access-dtp64" pod "7f93aa79-27fa-4f24-9b20-b6da0dee9ec2" (UID: "7f93aa79-27fa-4f24-9b20-b6da0dee9ec2"). InnerVolumeSpecName "kube-api-access-dtp64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:28.639304 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.639256    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9ef7c6e-9978-477c-be85-5816cf030b20-kube-api-access-9qjtv\" (UniqueName: \"kubernetes.io/projected/f9ef7c6e-9978-477c-be85-5816cf030b20-kube-api-access-9qjtv\") pod \"f9ef7c6e-9978-477c-be85-5816cf030b20\" (UID: \"f9ef7c6e-9978-477c-be85-5816cf030b20\") "
Oct 30 13:43:28.639502 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.639379    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9ef7c6e-9978-477c-be85-5816cf030b20-webhook-cert\" (UniqueName: \"kubernetes.io/secret/f9ef7c6e-9978-477c-be85-5816cf030b20-webhook-cert\") pod \"f9ef7c6e-9978-477c-be85-5816cf030b20\" (UID: \"f9ef7c6e-9978-477c-be85-5816cf030b20\") "
Oct 30 13:43:28.639502 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.639431    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9ef7c6e-9978-477c-be85-5816cf030b20-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9ef7c6e-9978-477c-be85-5816cf030b20-metrics-certs\") pod \"f9ef7c6e-9978-477c-be85-5816cf030b20\" (UID: \"f9ef7c6e-9978-477c-be85-5816cf030b20\") "
Oct 30 13:43:28.639713 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.639690    4544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istio-envoy\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:43:28.639824 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.639719    4544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istio-token\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:43:28.639824 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.639739    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtp64\" (UniqueName: \"kubernetes.io/projected/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-kube-api-access-dtp64\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:43:28.639824 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.639757    4544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istio-data\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:43:28.639824 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.639771    4544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istio-podinfo\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:43:28.639824 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.639783    4544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-workload-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:43:28.639824 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.639795    4544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-istiod-ca-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:43:28.639824 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.639807    4544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-workload-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:43:28.639824 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.639818    4544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2-credential-socket\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:43:28.641950 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.641917    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9ef7c6e-9978-477c-be85-5816cf030b20-webhook-cert" pod "f9ef7c6e-9978-477c-be85-5816cf030b20" (UID: "f9ef7c6e-9978-477c-be85-5816cf030b20"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:43:28.642168 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.642141    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9ef7c6e-9978-477c-be85-5816cf030b20-kube-api-access-9qjtv" pod "f9ef7c6e-9978-477c-be85-5816cf030b20" (UID: "f9ef7c6e-9978-477c-be85-5816cf030b20"). InnerVolumeSpecName "kube-api-access-9qjtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:28.642267 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.642188    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9ef7c6e-9978-477c-be85-5816cf030b20-metrics-certs" pod "f9ef7c6e-9978-477c-be85-5816cf030b20" (UID: "f9ef7c6e-9978-477c-be85-5816cf030b20"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:43:28.741065 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.740959    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qjtv\" (UniqueName: \"kubernetes.io/projected/f9ef7c6e-9978-477c-be85-5816cf030b20-kube-api-access-9qjtv\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:43:28.741065 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.741002    4544 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f9ef7c6e-9978-477c-be85-5816cf030b20-webhook-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:43:28.741065 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:28.741017    4544 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9ef7c6e-9978-477c-be85-5816cf030b20-metrics-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:43:29.007411 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:43:29.007329    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:43:29.210698 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:29.210653    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:43:29.226857 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:29.226819    4544 generic.go:356] "Generic (PLEG): container finished" podID="f9ef7c6e-9978-477c-be85-5816cf030b20" containerID="027309718f22a19a12793b4cc725756604a6144b25790a21332a3075bb524e07" exitCode=0
Oct 30 13:43:29.227049 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:29.226866    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75d8bff59-xxtf4"
Oct 30 13:43:29.227049 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:29.226917    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75d8bff59-xxtf4" event={"ID":"f9ef7c6e-9978-477c-be85-5816cf030b20","Type":"ContainerDied","Data":"027309718f22a19a12793b4cc725756604a6144b25790a21332a3075bb524e07"}
Oct 30 13:43:29.227049 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:29.226982    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75d8bff59-xxtf4" event={"ID":"f9ef7c6e-9978-477c-be85-5816cf030b20","Type":"ContainerDied","Data":"d669be218a32550c76c557b654261a8ee55282801c6567fb6524b2ceb9e9e6fe"}
Oct 30 13:43:29.227049 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:29.227010    4544 scope.go:122] "RemoveContainer" containerID="027309718f22a19a12793b4cc725756604a6144b25790a21332a3075bb524e07"
Oct 30 13:43:29.229464 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:29.229436    4544 generic.go:356] "Generic (PLEG): container finished" podID="7f93aa79-27fa-4f24-9b20-b6da0dee9ec2" containerID="91258d98de4e7bf522bcc4e337e678335fb6d5c3a18aa8cb6f2629fb36e640a2" exitCode=0
Oct 30 13:43:29.229743 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:29.229499    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"
Oct 30 13:43:29.230424 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:29.229521    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn" event={"ID":"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2","Type":"ContainerDied","Data":"91258d98de4e7bf522bcc4e337e678335fb6d5c3a18aa8cb6f2629fb36e640a2"}
Oct 30 13:43:29.230615 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:29.230588    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn" event={"ID":"7f93aa79-27fa-4f24-9b20-b6da0dee9ec2","Type":"ContainerDied","Data":"44c7f42a34037c1cb22659354bcb216a04f11b9cbe0b6048d04e7f2ce8da90fd"}
Oct 30 13:43:29.241041 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:29.241010    4544 scope.go:122] "RemoveContainer" containerID="027309718f22a19a12793b4cc725756604a6144b25790a21332a3075bb524e07"
Oct 30 13:43:29.241695 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:43:29.241661    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"027309718f22a19a12793b4cc725756604a6144b25790a21332a3075bb524e07\": container with ID starting with 027309718f22a19a12793b4cc725756604a6144b25790a21332a3075bb524e07 not found: ID does not exist" containerID="027309718f22a19a12793b4cc725756604a6144b25790a21332a3075bb524e07"
Oct 30 13:43:29.241801 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:29.241708    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"027309718f22a19a12793b4cc725756604a6144b25790a21332a3075bb524e07"} err="failed to get container status \"027309718f22a19a12793b4cc725756604a6144b25790a21332a3075bb524e07\": rpc error: code = NotFound desc = could not find container \"027309718f22a19a12793b4cc725756604a6144b25790a21332a3075bb524e07\": container with ID starting with 027309718f22a19a12793b4cc725756604a6144b25790a21332a3075bb524e07 not found: ID does not exist"
Oct 30 13:43:29.241801 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:29.241735    4544 scope.go:122] "RemoveContainer" containerID="91258d98de4e7bf522bcc4e337e678335fb6d5c3a18aa8cb6f2629fb36e640a2"
Oct 30 13:43:29.254571 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:29.254367    4544 scope.go:122] "RemoveContainer" containerID="91258d98de4e7bf522bcc4e337e678335fb6d5c3a18aa8cb6f2629fb36e640a2"
Oct 30 13:43:29.254897 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:43:29.254783    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91258d98de4e7bf522bcc4e337e678335fb6d5c3a18aa8cb6f2629fb36e640a2\": container with ID starting with 91258d98de4e7bf522bcc4e337e678335fb6d5c3a18aa8cb6f2629fb36e640a2 not found: ID does not exist" containerID="91258d98de4e7bf522bcc4e337e678335fb6d5c3a18aa8cb6f2629fb36e640a2"
Oct 30 13:43:29.254897 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:29.254828    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91258d98de4e7bf522bcc4e337e678335fb6d5c3a18aa8cb6f2629fb36e640a2"} err="failed to get container status \"91258d98de4e7bf522bcc4e337e678335fb6d5c3a18aa8cb6f2629fb36e640a2\": rpc error: code = NotFound desc = could not find container \"91258d98de4e7bf522bcc4e337e678335fb6d5c3a18aa8cb6f2629fb36e640a2\": container with ID starting with 91258d98de4e7bf522bcc4e337e678335fb6d5c3a18aa8cb6f2629fb36e640a2 not found: ID does not exist"
Oct 30 13:43:29.279337 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:29.279249    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75d8bff59-xxtf4"]
Oct 30 13:43:29.291103 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:29.291062    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-75d8bff59-xxtf4"]
Oct 30 13:43:29.319524 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:29.319485    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"]
Oct 30 13:43:29.327303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:29.327264    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn"]
Oct 30 13:43:29.343385 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:29.343349    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:43:30.009075 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.008671    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-75d8bff59-r4684"]
Oct 30 13:43:30.009335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.009214    4544 cpu_manager.go:414] "containerMap: removing container" podUID="d7da56bd-b1ed-419f-b90d-c9a68a8f756d" containerName="payload-pre-processing"
Oct 30 13:43:30.009335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.009247    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7da56bd-b1ed-419f-b90d-c9a68a8f756d" containerName="payload-pre-processing"
Oct 30 13:43:30.009335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.009264    4544 cpu_manager.go:414] "containerMap: removing container" podUID="4ebcb91d-6f3d-4f42-b029-0affb4abbbf9" containerName="revoke-keys"
Oct 30 13:43:30.009335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.009271    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ebcb91d-6f3d-4f42-b029-0affb4abbbf9" containerName="revoke-keys"
Oct 30 13:43:30.009335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.009290    4544 cpu_manager.go:414] "containerMap: removing container" podUID="b63682f2-c062-4ef4-801f-b84a3a7c10e0" containerName="payload-processing"
Oct 30 13:43:30.009335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.009297    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b63682f2-c062-4ef4-801f-b84a3a7c10e0" containerName="payload-processing"
Oct 30 13:43:30.009335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.009311    4544 cpu_manager.go:414] "containerMap: removing container" podUID="52424dad-1898-48d3-9529-3dc7f16cb1e5" containerName="payload-pre-processing"
Oct 30 13:43:30.009335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.009318    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52424dad-1898-48d3-9529-3dc7f16cb1e5" containerName="payload-pre-processing"
Oct 30 13:43:30.009335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.009330    4544 cpu_manager.go:414] "containerMap: removing container" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main"
Oct 30 13:43:30.009335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.009337    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main"
Oct 30 13:43:30.009884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.009346    4544 cpu_manager.go:414] "containerMap: removing container" podUID="7f93aa79-27fa-4f24-9b20-b6da0dee9ec2" containerName="istio-proxy"
Oct 30 13:43:30.009884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.009352    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f93aa79-27fa-4f24-9b20-b6da0dee9ec2" containerName="istio-proxy"
Oct 30 13:43:30.009884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.009372    4544 cpu_manager.go:414] "containerMap: removing container" podUID="f9ef7c6e-9978-477c-be85-5816cf030b20" containerName="manager"
Oct 30 13:43:30.009884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.009378    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9ef7c6e-9978-477c-be85-5816cf030b20" containerName="manager"
Oct 30 13:43:30.009884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.009464    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7da56bd-b1ed-419f-b90d-c9a68a8f756d" containerName="payload-pre-processing"
Oct 30 13:43:30.009884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.009473    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ebcb91d-6f3d-4f42-b029-0affb4abbbf9" containerName="revoke-keys"
Oct 30 13:43:30.009884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.009484    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="b63682f2-c062-4ef4-801f-b84a3a7c10e0" containerName="payload-processing"
Oct 30 13:43:30.009884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.009493    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="52424dad-1898-48d3-9529-3dc7f16cb1e5" containerName="payload-pre-processing"
Oct 30 13:43:30.009884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.009501    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="8be6bd4f-3790-489b-8a6e-edcd85f33aec" containerName="main"
Oct 30 13:43:30.009884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.009509    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f93aa79-27fa-4f24-9b20-b6da0dee9ec2" containerName="istio-proxy"
Oct 30 13:43:30.009884 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.009517    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9ef7c6e-9978-477c-be85-5816cf030b20" containerName="manager"
Oct 30 13:43:30.014231 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.014198    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75d8bff59-r4684"
Oct 30 13:43:30.029903 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.029877    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 13:43:30.041202 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.041173    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2z8s8\""
Oct 30 13:43:30.041410 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.041276    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 13:43:30.041410 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.041317    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 13:43:30.046793 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.046766    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75d8bff59-r4684"]
Oct 30 13:43:30.154286 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.154215    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vcz2\" (UniqueName: \"kubernetes.io/projected/db3ad103-72ab-4294-a991-7cab75502a1d-kube-api-access-7vcz2\") pod \"maas-controller-75d8bff59-r4684\" (UID: \"db3ad103-72ab-4294-a991-7cab75502a1d\") " pod="opendatahub/maas-controller-75d8bff59-r4684"
Oct 30 13:43:30.154503 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.154383    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db3ad103-72ab-4294-a991-7cab75502a1d-webhook-cert\") pod \"maas-controller-75d8bff59-r4684\" (UID: \"db3ad103-72ab-4294-a991-7cab75502a1d\") " pod="opendatahub/maas-controller-75d8bff59-r4684"
Oct 30 13:43:30.154503 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.154436    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db3ad103-72ab-4294-a991-7cab75502a1d-metrics-certs\") pod \"maas-controller-75d8bff59-r4684\" (UID: \"db3ad103-72ab-4294-a991-7cab75502a1d\") " pod="opendatahub/maas-controller-75d8bff59-r4684"
Oct 30 13:43:30.255876 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.255837    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db3ad103-72ab-4294-a991-7cab75502a1d-metrics-certs\") pod \"maas-controller-75d8bff59-r4684\" (UID: \"db3ad103-72ab-4294-a991-7cab75502a1d\") " pod="opendatahub/maas-controller-75d8bff59-r4684"
Oct 30 13:43:30.256450 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.255898    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vcz2\" (UniqueName: \"kubernetes.io/projected/db3ad103-72ab-4294-a991-7cab75502a1d-kube-api-access-7vcz2\") pod \"maas-controller-75d8bff59-r4684\" (UID: \"db3ad103-72ab-4294-a991-7cab75502a1d\") " pod="opendatahub/maas-controller-75d8bff59-r4684"
Oct 30 13:43:30.256450 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.255972    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db3ad103-72ab-4294-a991-7cab75502a1d-webhook-cert\") pod \"maas-controller-75d8bff59-r4684\" (UID: \"db3ad103-72ab-4294-a991-7cab75502a1d\") " pod="opendatahub/maas-controller-75d8bff59-r4684"
Oct 30 13:43:30.258524 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.258496    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db3ad103-72ab-4294-a991-7cab75502a1d-webhook-cert\") pod \"maas-controller-75d8bff59-r4684\" (UID: \"db3ad103-72ab-4294-a991-7cab75502a1d\") " pod="opendatahub/maas-controller-75d8bff59-r4684"
Oct 30 13:43:30.258524 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.258518    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db3ad103-72ab-4294-a991-7cab75502a1d-metrics-certs\") pod \"maas-controller-75d8bff59-r4684\" (UID: \"db3ad103-72ab-4294-a991-7cab75502a1d\") " pod="opendatahub/maas-controller-75d8bff59-r4684"
Oct 30 13:43:30.280230 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.280134    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vcz2\" (UniqueName: \"kubernetes.io/projected/db3ad103-72ab-4294-a991-7cab75502a1d-kube-api-access-7vcz2\") pod \"maas-controller-75d8bff59-r4684\" (UID: \"db3ad103-72ab-4294-a991-7cab75502a1d\") " pod="opendatahub/maas-controller-75d8bff59-r4684"
Oct 30 13:43:30.326937 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.326902    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75d8bff59-r4684"
Oct 30 13:43:30.655060 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:30.655020    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75d8bff59-r4684"]
Oct 30 13:43:30.658296 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:43:30.658265    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb3ad103_72ab_4294_a991_7cab75502a1d.slice/crio-c7c23c1ee165c3d5a1fff371af07ca2d9eb105d650b25aeec9ab0d077cf03130 WatchSource:0}: Error finding container c7c23c1ee165c3d5a1fff371af07ca2d9eb105d650b25aeec9ab0d077cf03130: Status 404 returned error can't find the container with id c7c23c1ee165c3d5a1fff371af07ca2d9eb105d650b25aeec9ab0d077cf03130
Oct 30 13:43:31.084611 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:31.084574    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f93aa79-27fa-4f24-9b20-b6da0dee9ec2" path="/var/lib/kubelet/pods/7f93aa79-27fa-4f24-9b20-b6da0dee9ec2/volumes"
Oct 30 13:43:31.084972 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:31.084958    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9ef7c6e-9978-477c-be85-5816cf030b20" path="/var/lib/kubelet/pods/f9ef7c6e-9978-477c-be85-5816cf030b20/volumes"
Oct 30 13:43:31.244821 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:31.244791    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75d8bff59-r4684" event={"ID":"db3ad103-72ab-4294-a991-7cab75502a1d","Type":"ContainerStarted","Data":"c7c23c1ee165c3d5a1fff371af07ca2d9eb105d650b25aeec9ab0d077cf03130"}
Oct 30 13:43:32.252730 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:32.252690    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75d8bff59-r4684" event={"ID":"db3ad103-72ab-4294-a991-7cab75502a1d","Type":"ContainerStarted","Data":"608c3d0b1558e1cd95f8e995f65e985618430c40ce458d552b12a5c9afbddcc4"}
Oct 30 13:43:32.253179 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:32.252868    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-75d8bff59-r4684"
Oct 30 13:43:32.302904 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:32.302847    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-75d8bff59-r4684" podStartSLOduration=2.7193913629999997 podStartE2EDuration="3.302830719s" totalImagesPullingTime="583.439356ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:43:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:43:32.302433565 +0000 UTC m=+3507.349567517" watchObservedRunningTime="2026-10-30 13:43:32.302830719 +0000 UTC m=+3507.349964673"
Oct 30 13:43:32.386605 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:32.386567    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:43:38.274747 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:38.274711    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:43:43.260914 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:43.260879    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-75d8bff59-r4684"
Oct 30 13:43:51.892413 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:51.892362    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-66xvf"]
Oct 30 13:43:51.898901 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:51.898869    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-66xvf"
Oct 30 13:43:51.915749 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:51.915718    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:43:51.928065 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:51.928029    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:43:51.930586 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:51.930545    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-66xvf"]
Oct 30 13:43:51.956235 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:51.956183    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8fafc3f-2651-4647-9cb8-56a984441c97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-66xvf\" (UID: \"c8fafc3f-2651-4647-9cb8-56a984441c97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-66xvf"
Oct 30 13:43:51.956425 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:51.956258    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8fafc3f-2651-4647-9cb8-56a984441c97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-66xvf\" (UID: \"c8fafc3f-2651-4647-9cb8-56a984441c97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-66xvf"
Oct 30 13:43:51.956478 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:51.956429    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nslns\" (UniqueName: \"kubernetes.io/projected/c8fafc3f-2651-4647-9cb8-56a984441c97-kube-api-access-nslns\") pod \"maas-api-revoke-sub-85237e45-66xvf\" (UID: \"c8fafc3f-2651-4647-9cb8-56a984441c97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-66xvf"
Oct 30 13:43:52.056913 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:52.056875    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nslns\" (UniqueName: \"kubernetes.io/projected/c8fafc3f-2651-4647-9cb8-56a984441c97-kube-api-access-nslns\") pod \"maas-api-revoke-sub-85237e45-66xvf\" (UID: \"c8fafc3f-2651-4647-9cb8-56a984441c97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-66xvf"
Oct 30 13:43:52.057125 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:52.057062    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8fafc3f-2651-4647-9cb8-56a984441c97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-66xvf\" (UID: \"c8fafc3f-2651-4647-9cb8-56a984441c97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-66xvf"
Oct 30 13:43:52.057125 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:52.057105    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8fafc3f-2651-4647-9cb8-56a984441c97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-66xvf\" (UID: \"c8fafc3f-2651-4647-9cb8-56a984441c97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-66xvf"
Oct 30 13:43:52.057765 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:52.057736    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8fafc3f-2651-4647-9cb8-56a984441c97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-66xvf\" (UID: \"c8fafc3f-2651-4647-9cb8-56a984441c97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-66xvf"
Oct 30 13:43:52.059445 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:52.059419    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8fafc3f-2651-4647-9cb8-56a984441c97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-66xvf\" (UID: \"c8fafc3f-2651-4647-9cb8-56a984441c97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-66xvf"
Oct 30 13:43:52.080651 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:52.080620    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nslns\" (UniqueName: \"kubernetes.io/projected/c8fafc3f-2651-4647-9cb8-56a984441c97-kube-api-access-nslns\") pod \"maas-api-revoke-sub-85237e45-66xvf\" (UID: \"c8fafc3f-2651-4647-9cb8-56a984441c97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-66xvf"
Oct 30 13:43:52.214715 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:52.214614    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-66xvf"
Oct 30 13:43:52.526338 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:52.526257    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-66xvf"]
Oct 30 13:43:52.532859 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:43:52.532824    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8fafc3f_2651_4647_9cb8_56a984441c97.slice/crio-08b43c9f568f241384a21cb1ce8afaacc64913a22a23df680b0320b2b740e7b6 WatchSource:0}: Error finding container 08b43c9f568f241384a21cb1ce8afaacc64913a22a23df680b0320b2b740e7b6: Status 404 returned error can't find the container with id 08b43c9f568f241384a21cb1ce8afaacc64913a22a23df680b0320b2b740e7b6
Oct 30 13:43:53.370446 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:53.370405    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-66xvf" event={"ID":"c8fafc3f-2651-4647-9cb8-56a984441c97","Type":"ContainerStarted","Data":"7df008c94d72b07ed1f2e5ef297bb6d020196d254c18e536c0189d6245a6f935"}
Oct 30 13:43:53.370941 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:53.370463    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-66xvf" event={"ID":"c8fafc3f-2651-4647-9cb8-56a984441c97","Type":"ContainerStarted","Data":"08b43c9f568f241384a21cb1ce8afaacc64913a22a23df680b0320b2b740e7b6"}
Oct 30 13:43:53.421970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:53.421913    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-66xvf" podStartSLOduration=2.421894971 podStartE2EDuration="2.421894971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:43:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:43:53.420930958 +0000 UTC m=+3528.468064911" watchObservedRunningTime="2026-10-30 13:43:53.421894971 +0000 UTC m=+3528.469028926"
Oct 30 13:43:54.384573 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:54.384532    4544 generic.go:356] "Generic (PLEG): container finished" podID="c8fafc3f-2651-4647-9cb8-56a984441c97" containerID="7df008c94d72b07ed1f2e5ef297bb6d020196d254c18e536c0189d6245a6f935" exitCode=0
Oct 30 13:43:54.385031 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:54.384614    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-66xvf" event={"ID":"c8fafc3f-2651-4647-9cb8-56a984441c97","Type":"ContainerDied","Data":"7df008c94d72b07ed1f2e5ef297bb6d020196d254c18e536c0189d6245a6f935"}
Oct 30 13:43:55.514199 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:55.514173    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-66xvf"
Oct 30 13:43:55.593644 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:55.593611    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8fafc3f-2651-4647-9cb8-56a984441c97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8fafc3f-2651-4647-9cb8-56a984441c97-maas-api-xks-ca\") pod \"c8fafc3f-2651-4647-9cb8-56a984441c97\" (UID: \"c8fafc3f-2651-4647-9cb8-56a984441c97\") "
Oct 30 13:43:55.593809 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:55.593673    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8fafc3f-2651-4647-9cb8-56a984441c97-kube-api-access-nslns\" (UniqueName: \"kubernetes.io/projected/c8fafc3f-2651-4647-9cb8-56a984441c97-kube-api-access-nslns\") pod \"c8fafc3f-2651-4647-9cb8-56a984441c97\" (UID: \"c8fafc3f-2651-4647-9cb8-56a984441c97\") "
Oct 30 13:43:55.593809 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:55.593720    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8fafc3f-2651-4647-9cb8-56a984441c97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8fafc3f-2651-4647-9cb8-56a984441c97-maas-api-openshift-ca\") pod \"c8fafc3f-2651-4647-9cb8-56a984441c97\" (UID: \"c8fafc3f-2651-4647-9cb8-56a984441c97\") "
Oct 30 13:43:55.594105 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:55.594083    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8fafc3f-2651-4647-9cb8-56a984441c97-maas-api-openshift-ca" pod "c8fafc3f-2651-4647-9cb8-56a984441c97" (UID: "c8fafc3f-2651-4647-9cb8-56a984441c97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:43:55.595864 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:55.595836    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8fafc3f-2651-4647-9cb8-56a984441c97-maas-api-xks-ca" pod "c8fafc3f-2651-4647-9cb8-56a984441c97" (UID: "c8fafc3f-2651-4647-9cb8-56a984441c97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:43:55.595977 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:55.595860    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8fafc3f-2651-4647-9cb8-56a984441c97-kube-api-access-nslns" pod "c8fafc3f-2651-4647-9cb8-56a984441c97" (UID: "c8fafc3f-2651-4647-9cb8-56a984441c97"). InnerVolumeSpecName "kube-api-access-nslns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:55.694454 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:55.694360    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8fafc3f-2651-4647-9cb8-56a984441c97-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:43:55.694454 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:55.694394    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nslns\" (UniqueName: \"kubernetes.io/projected/c8fafc3f-2651-4647-9cb8-56a984441c97-kube-api-access-nslns\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:43:55.694454 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:55.694408    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8fafc3f-2651-4647-9cb8-56a984441c97-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:43:56.397062 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:56.397023    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-66xvf"
Oct 30 13:43:56.397062 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:56.397036    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-66xvf" event={"ID":"c8fafc3f-2651-4647-9cb8-56a984441c97","Type":"ContainerDied","Data":"08b43c9f568f241384a21cb1ce8afaacc64913a22a23df680b0320b2b740e7b6"}
Oct 30 13:43:56.397342 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:56.397090    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08b43c9f568f241384a21cb1ce8afaacc64913a22a23df680b0320b2b740e7b6"
Oct 30 13:43:58.905958 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:58.905924    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:43:59.006162 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:43:59.006125    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:43:59.190480 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:43:59.190393    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:44:03.678236 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:03.678183    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:44:03.810963 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:03.810928    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75d8bff59-r4684"]
Oct 30 13:44:03.811233 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:03.811176    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-75d8bff59-r4684" podUID="db3ad103-72ab-4294-a991-7cab75502a1d" containerName="manager" containerID="cri-o://608c3d0b1558e1cd95f8e995f65e985618430c40ce458d552b12a5c9afbddcc4" gracePeriod=30
Oct 30 13:44:04.061790 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:04.061756    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75d8bff59-r4684"
Oct 30 13:44:04.173474 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:04.173441    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db3ad103-72ab-4294-a991-7cab75502a1d-kube-api-access-7vcz2\" (UniqueName: \"kubernetes.io/projected/db3ad103-72ab-4294-a991-7cab75502a1d-kube-api-access-7vcz2\") pod \"db3ad103-72ab-4294-a991-7cab75502a1d\" (UID: \"db3ad103-72ab-4294-a991-7cab75502a1d\") "
Oct 30 13:44:04.173709 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:04.173543    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db3ad103-72ab-4294-a991-7cab75502a1d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/db3ad103-72ab-4294-a991-7cab75502a1d-metrics-certs\") pod \"db3ad103-72ab-4294-a991-7cab75502a1d\" (UID: \"db3ad103-72ab-4294-a991-7cab75502a1d\") "
Oct 30 13:44:04.173709 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:04.173609    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db3ad103-72ab-4294-a991-7cab75502a1d-webhook-cert\" (UniqueName: \"kubernetes.io/secret/db3ad103-72ab-4294-a991-7cab75502a1d-webhook-cert\") pod \"db3ad103-72ab-4294-a991-7cab75502a1d\" (UID: \"db3ad103-72ab-4294-a991-7cab75502a1d\") "
Oct 30 13:44:04.175652 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:04.175616    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db3ad103-72ab-4294-a991-7cab75502a1d-metrics-certs" pod "db3ad103-72ab-4294-a991-7cab75502a1d" (UID: "db3ad103-72ab-4294-a991-7cab75502a1d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:44:04.175765 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:04.175626    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db3ad103-72ab-4294-a991-7cab75502a1d-kube-api-access-7vcz2" pod "db3ad103-72ab-4294-a991-7cab75502a1d" (UID: "db3ad103-72ab-4294-a991-7cab75502a1d"). InnerVolumeSpecName "kube-api-access-7vcz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:04.175765 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:04.175697    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db3ad103-72ab-4294-a991-7cab75502a1d-webhook-cert" pod "db3ad103-72ab-4294-a991-7cab75502a1d" (UID: "db3ad103-72ab-4294-a991-7cab75502a1d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:44:04.274840 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:04.274805    4544 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db3ad103-72ab-4294-a991-7cab75502a1d-metrics-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:44:04.274840 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:04.274833    4544 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db3ad103-72ab-4294-a991-7cab75502a1d-webhook-cert\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:44:04.274840 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:04.274842    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vcz2\" (UniqueName: \"kubernetes.io/projected/db3ad103-72ab-4294-a991-7cab75502a1d-kube-api-access-7vcz2\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:44:04.442548 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:04.442459    4544 generic.go:356] "Generic (PLEG): container finished" podID="db3ad103-72ab-4294-a991-7cab75502a1d" containerID="608c3d0b1558e1cd95f8e995f65e985618430c40ce458d552b12a5c9afbddcc4" exitCode=0
Oct 30 13:44:04.442548 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:04.442515    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75d8bff59-r4684"
Oct 30 13:44:04.442776 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:04.442547    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75d8bff59-r4684" event={"ID":"db3ad103-72ab-4294-a991-7cab75502a1d","Type":"ContainerDied","Data":"608c3d0b1558e1cd95f8e995f65e985618430c40ce458d552b12a5c9afbddcc4"}
Oct 30 13:44:04.442776 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:04.442609    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75d8bff59-r4684" event={"ID":"db3ad103-72ab-4294-a991-7cab75502a1d","Type":"ContainerDied","Data":"c7c23c1ee165c3d5a1fff371af07ca2d9eb105d650b25aeec9ab0d077cf03130"}
Oct 30 13:44:04.442776 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:04.442641    4544 scope.go:122] "RemoveContainer" containerID="608c3d0b1558e1cd95f8e995f65e985618430c40ce458d552b12a5c9afbddcc4"
Oct 30 13:44:04.452546 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:04.452527    4544 scope.go:122] "RemoveContainer" containerID="608c3d0b1558e1cd95f8e995f65e985618430c40ce458d552b12a5c9afbddcc4"
Oct 30 13:44:04.452825 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:44:04.452804    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"608c3d0b1558e1cd95f8e995f65e985618430c40ce458d552b12a5c9afbddcc4\": container with ID starting with 608c3d0b1558e1cd95f8e995f65e985618430c40ce458d552b12a5c9afbddcc4 not found: ID does not exist" containerID="608c3d0b1558e1cd95f8e995f65e985618430c40ce458d552b12a5c9afbddcc4"
Oct 30 13:44:04.452923 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:04.452834    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"608c3d0b1558e1cd95f8e995f65e985618430c40ce458d552b12a5c9afbddcc4"} err="failed to get container status \"608c3d0b1558e1cd95f8e995f65e985618430c40ce458d552b12a5c9afbddcc4\": rpc error: code = NotFound desc = could not find container \"608c3d0b1558e1cd95f8e995f65e985618430c40ce458d552b12a5c9afbddcc4\": container with ID starting with 608c3d0b1558e1cd95f8e995f65e985618430c40ce458d552b12a5c9afbddcc4 not found: ID does not exist"
Oct 30 13:44:04.495173 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:04.495143    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75d8bff59-r4684"]
Oct 30 13:44:04.500446 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:04.500423    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-75d8bff59-r4684"]
Oct 30 13:44:05.084961 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:05.084924    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db3ad103-72ab-4294-a991-7cab75502a1d" path="/var/lib/kubelet/pods/db3ad103-72ab-4294-a991-7cab75502a1d/volumes"
Oct 30 13:44:05.642702 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:05.642663    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:44:05.857057 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:05.857023    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-75d8bff59-8hz8s"]
Oct 30 13:44:05.857417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:05.857403    4544 cpu_manager.go:414] "containerMap: removing container" podUID="db3ad103-72ab-4294-a991-7cab75502a1d" containerName="manager"
Oct 30 13:44:05.857471 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:05.857419    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db3ad103-72ab-4294-a991-7cab75502a1d" containerName="manager"
Oct 30 13:44:05.857471 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:05.857438    4544 cpu_manager.go:414] "containerMap: removing container" podUID="c8fafc3f-2651-4647-9cb8-56a984441c97" containerName="revoke-keys"
Oct 30 13:44:05.857471 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:05.857443    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8fafc3f-2651-4647-9cb8-56a984441c97" containerName="revoke-keys"
Oct 30 13:44:05.857586 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:05.857509    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8fafc3f-2651-4647-9cb8-56a984441c97" containerName="revoke-keys"
Oct 30 13:44:05.857586 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:05.857522    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="db3ad103-72ab-4294-a991-7cab75502a1d" containerName="manager"
Oct 30 13:44:05.861858 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:05.861838    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75d8bff59-8hz8s"
Oct 30 13:44:05.891340 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:05.891320    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 13:44:05.891340 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:05.891321    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 13:44:05.891501 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:05.891409    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 13:44:05.891624 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:05.891610    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2z8s8\""
Oct 30 13:44:05.893805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:05.893751    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75d8bff59-8hz8s"]
Oct 30 13:44:05.991036 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:05.991007    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af0033c-76ac-4cc4-852f-3a3c190778bb-metrics-certs\") pod \"maas-controller-75d8bff59-8hz8s\" (UID: \"9af0033c-76ac-4cc4-852f-3a3c190778bb\") " pod="opendatahub/maas-controller-75d8bff59-8hz8s"
Oct 30 13:44:05.991203 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:05.991045    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9af0033c-76ac-4cc4-852f-3a3c190778bb-webhook-cert\") pod \"maas-controller-75d8bff59-8hz8s\" (UID: \"9af0033c-76ac-4cc4-852f-3a3c190778bb\") " pod="opendatahub/maas-controller-75d8bff59-8hz8s"
Oct 30 13:44:05.991203 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:05.991074    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpxwj\" (UniqueName: \"kubernetes.io/projected/9af0033c-76ac-4cc4-852f-3a3c190778bb-kube-api-access-zpxwj\") pod \"maas-controller-75d8bff59-8hz8s\" (UID: \"9af0033c-76ac-4cc4-852f-3a3c190778bb\") " pod="opendatahub/maas-controller-75d8bff59-8hz8s"
Oct 30 13:44:06.092273 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:06.092238    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af0033c-76ac-4cc4-852f-3a3c190778bb-metrics-certs\") pod \"maas-controller-75d8bff59-8hz8s\" (UID: \"9af0033c-76ac-4cc4-852f-3a3c190778bb\") " pod="opendatahub/maas-controller-75d8bff59-8hz8s"
Oct 30 13:44:06.092273 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:06.092281    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9af0033c-76ac-4cc4-852f-3a3c190778bb-webhook-cert\") pod \"maas-controller-75d8bff59-8hz8s\" (UID: \"9af0033c-76ac-4cc4-852f-3a3c190778bb\") " pod="opendatahub/maas-controller-75d8bff59-8hz8s"
Oct 30 13:44:06.092747 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:06.092310    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpxwj\" (UniqueName: \"kubernetes.io/projected/9af0033c-76ac-4cc4-852f-3a3c190778bb-kube-api-access-zpxwj\") pod \"maas-controller-75d8bff59-8hz8s\" (UID: \"9af0033c-76ac-4cc4-852f-3a3c190778bb\") " pod="opendatahub/maas-controller-75d8bff59-8hz8s"
Oct 30 13:44:06.095051 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:06.095016    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af0033c-76ac-4cc4-852f-3a3c190778bb-metrics-certs\") pod \"maas-controller-75d8bff59-8hz8s\" (UID: \"9af0033c-76ac-4cc4-852f-3a3c190778bb\") " pod="opendatahub/maas-controller-75d8bff59-8hz8s"
Oct 30 13:44:06.095181 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:06.095016    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9af0033c-76ac-4cc4-852f-3a3c190778bb-webhook-cert\") pod \"maas-controller-75d8bff59-8hz8s\" (UID: \"9af0033c-76ac-4cc4-852f-3a3c190778bb\") " pod="opendatahub/maas-controller-75d8bff59-8hz8s"
Oct 30 13:44:06.116432 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:06.116407    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpxwj\" (UniqueName: \"kubernetes.io/projected/9af0033c-76ac-4cc4-852f-3a3c190778bb-kube-api-access-zpxwj\") pod \"maas-controller-75d8bff59-8hz8s\" (UID: \"9af0033c-76ac-4cc4-852f-3a3c190778bb\") " pod="opendatahub/maas-controller-75d8bff59-8hz8s"
Oct 30 13:44:06.174534 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:06.174438    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75d8bff59-8hz8s"
Oct 30 13:44:06.482657 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:06.482565    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75d8bff59-8hz8s"]
Oct 30 13:44:06.486938 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:44:06.486905    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af0033c_76ac_4cc4_852f_3a3c190778bb.slice/crio-b3327c6571faebceb23b1d346ba4a06d115dc24933cd5aaf4066bae4cd76d1a5 WatchSource:0}: Error finding container b3327c6571faebceb23b1d346ba4a06d115dc24933cd5aaf4066bae4cd76d1a5: Status 404 returned error can't find the container with id b3327c6571faebceb23b1d346ba4a06d115dc24933cd5aaf4066bae4cd76d1a5
Oct 30 13:44:07.470539 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:07.470497    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75d8bff59-8hz8s" event={"ID":"9af0033c-76ac-4cc4-852f-3a3c190778bb","Type":"ContainerStarted","Data":"f29093d2606d3a298cbba962c0a2c002740456c6bb9a819e3170ffa9685e63af"}
Oct 30 13:44:07.471003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:07.470575    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75d8bff59-8hz8s" event={"ID":"9af0033c-76ac-4cc4-852f-3a3c190778bb","Type":"ContainerStarted","Data":"b3327c6571faebceb23b1d346ba4a06d115dc24933cd5aaf4066bae4cd76d1a5"}
Oct 30 13:44:07.471003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:07.470657    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-75d8bff59-8hz8s"
Oct 30 13:44:07.513154 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:07.513100    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-75d8bff59-8hz8s" podStartSLOduration=2.173305178 podStartE2EDuration="2.513086947s" totalImagesPullingTime="339.781769ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:07.512259168 +0000 UTC m=+3542.559393117" watchObservedRunningTime="2026-10-30 13:44:07.513086947 +0000 UTC m=+3542.560220899"
Oct 30 13:44:15.392303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:15.392270    4544 scope.go:122] "RemoveContainer" containerID="51960c74027d772c41a8d22a93196f755b2bf7653912db1fdbf3b1f4cb0865f6"
Oct 30 13:44:15.400963 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:15.400940    4544 scope.go:122] "RemoveContainer" containerID="c4b29abc4bba9dc8b6e8cce9e20baeaa9a42b17bdf61b7deed106bf581f606dc"
Oct 30 13:44:15.408647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:15.408627    4544 scope.go:122] "RemoveContainer" containerID="4feccb12d9e729a8befd620d4094d93676e9c29b6ec3b5b21ca1b76b59d6ea64"
Oct 30 13:44:15.416552 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:15.416531    4544 scope.go:122] "RemoveContainer" containerID="e50f322507ae05231c350075560556cc697d72b5ea682bcd0932ebc4a7662456"
Oct 30 13:44:15.426197 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:15.426179    4544 scope.go:122] "RemoveContainer" containerID="c3dd8a4783604edbf2573dabe8d9519cdefd2c7c38036800dd3490d7cb7e120b"
Oct 30 13:44:18.477980 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:18.477896    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-75d8bff59-8hz8s"
Oct 30 13:44:29.004701 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:44:29.004660    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:44:30.753490 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:30.753457    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:44:41.792468 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:41.792429    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8c4l4"]
Oct 30 13:44:41.795992 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:41.795961    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8c4l4"
Oct 30 13:44:41.812441 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:41.812407    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:44:41.824129 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:41.824097    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:44:41.827092 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:41.827054    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8c4l4"]
Oct 30 13:44:41.830791 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:41.830761    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9hpr\" (UniqueName: \"kubernetes.io/projected/43d7da87-2345-40ed-a5cb-96c2438eaf5a-kube-api-access-f9hpr\") pod \"maas-api-revoke-sub-c1e271a9-8c4l4\" (UID: \"43d7da87-2345-40ed-a5cb-96c2438eaf5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8c4l4"
Oct 30 13:44:41.830975 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:41.830821    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d7da87-2345-40ed-a5cb-96c2438eaf5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-8c4l4\" (UID: \"43d7da87-2345-40ed-a5cb-96c2438eaf5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8c4l4"
Oct 30 13:44:41.830975 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:41.830923    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d7da87-2345-40ed-a5cb-96c2438eaf5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-8c4l4\" (UID: \"43d7da87-2345-40ed-a5cb-96c2438eaf5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8c4l4"
Oct 30 13:44:41.932282 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:41.932239    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9hpr\" (UniqueName: \"kubernetes.io/projected/43d7da87-2345-40ed-a5cb-96c2438eaf5a-kube-api-access-f9hpr\") pod \"maas-api-revoke-sub-c1e271a9-8c4l4\" (UID: \"43d7da87-2345-40ed-a5cb-96c2438eaf5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8c4l4"
Oct 30 13:44:41.932497 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:41.932321    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d7da87-2345-40ed-a5cb-96c2438eaf5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-8c4l4\" (UID: \"43d7da87-2345-40ed-a5cb-96c2438eaf5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8c4l4"
Oct 30 13:44:41.932497 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:41.932357    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d7da87-2345-40ed-a5cb-96c2438eaf5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-8c4l4\" (UID: \"43d7da87-2345-40ed-a5cb-96c2438eaf5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8c4l4"
Oct 30 13:44:41.932958 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:41.932936    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d7da87-2345-40ed-a5cb-96c2438eaf5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-8c4l4\" (UID: \"43d7da87-2345-40ed-a5cb-96c2438eaf5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8c4l4"
Oct 30 13:44:41.934735 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:41.934710    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d7da87-2345-40ed-a5cb-96c2438eaf5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-8c4l4\" (UID: \"43d7da87-2345-40ed-a5cb-96c2438eaf5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8c4l4"
Oct 30 13:44:41.955519 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:41.955490    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9hpr\" (UniqueName: \"kubernetes.io/projected/43d7da87-2345-40ed-a5cb-96c2438eaf5a-kube-api-access-f9hpr\") pod \"maas-api-revoke-sub-c1e271a9-8c4l4\" (UID: \"43d7da87-2345-40ed-a5cb-96c2438eaf5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8c4l4"
Oct 30 13:44:42.109400 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:42.109360    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8c4l4"
Oct 30 13:44:42.412630 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:42.412526    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8c4l4"]
Oct 30 13:44:42.417559 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:44:42.417521    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d7da87_2345_40ed_a5cb_96c2438eaf5a.slice/crio-6d181346b604f81a962402174351e03b287e77254d6202c729315d6c077a2124 WatchSource:0}: Error finding container 6d181346b604f81a962402174351e03b287e77254d6202c729315d6c077a2124: Status 404 returned error can't find the container with id 6d181346b604f81a962402174351e03b287e77254d6202c729315d6c077a2124
Oct 30 13:44:42.666926 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:42.666837    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8c4l4" event={"ID":"43d7da87-2345-40ed-a5cb-96c2438eaf5a","Type":"ContainerStarted","Data":"6d181346b604f81a962402174351e03b287e77254d6202c729315d6c077a2124"}
Oct 30 13:44:43.674301 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:43.674266    4544 generic.go:356] "Generic (PLEG): container finished" podID="43d7da87-2345-40ed-a5cb-96c2438eaf5a" containerID="fc3d77339479044de1242eafb71000ec238d759941526d6bd6e404a8e6c57bed" exitCode=0
Oct 30 13:44:43.674792 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:43.674351    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8c4l4" event={"ID":"43d7da87-2345-40ed-a5cb-96c2438eaf5a","Type":"ContainerDied","Data":"fc3d77339479044de1242eafb71000ec238d759941526d6bd6e404a8e6c57bed"}
Oct 30 13:44:44.817876 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:44.817845    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8c4l4"
Oct 30 13:44:44.862281 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:44.862244    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43d7da87-2345-40ed-a5cb-96c2438eaf5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d7da87-2345-40ed-a5cb-96c2438eaf5a-maas-api-openshift-ca\") pod \"43d7da87-2345-40ed-a5cb-96c2438eaf5a\" (UID: \"43d7da87-2345-40ed-a5cb-96c2438eaf5a\") "
Oct 30 13:44:44.862492 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:44.862319    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43d7da87-2345-40ed-a5cb-96c2438eaf5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d7da87-2345-40ed-a5cb-96c2438eaf5a-maas-api-xks-ca\") pod \"43d7da87-2345-40ed-a5cb-96c2438eaf5a\" (UID: \"43d7da87-2345-40ed-a5cb-96c2438eaf5a\") "
Oct 30 13:44:44.862492 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:44.862363    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43d7da87-2345-40ed-a5cb-96c2438eaf5a-kube-api-access-f9hpr\" (UniqueName: \"kubernetes.io/projected/43d7da87-2345-40ed-a5cb-96c2438eaf5a-kube-api-access-f9hpr\") pod \"43d7da87-2345-40ed-a5cb-96c2438eaf5a\" (UID: \"43d7da87-2345-40ed-a5cb-96c2438eaf5a\") "
Oct 30 13:44:44.862701 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:44.862669    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43d7da87-2345-40ed-a5cb-96c2438eaf5a-maas-api-openshift-ca" pod "43d7da87-2345-40ed-a5cb-96c2438eaf5a" (UID: "43d7da87-2345-40ed-a5cb-96c2438eaf5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:44.865002 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:44.864978    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43d7da87-2345-40ed-a5cb-96c2438eaf5a-maas-api-xks-ca" pod "43d7da87-2345-40ed-a5cb-96c2438eaf5a" (UID: "43d7da87-2345-40ed-a5cb-96c2438eaf5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:44:44.865214 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:44.865182    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d7da87-2345-40ed-a5cb-96c2438eaf5a-kube-api-access-f9hpr" pod "43d7da87-2345-40ed-a5cb-96c2438eaf5a" (UID: "43d7da87-2345-40ed-a5cb-96c2438eaf5a"). InnerVolumeSpecName "kube-api-access-f9hpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:44.963348 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:44.963255    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d7da87-2345-40ed-a5cb-96c2438eaf5a-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:44:44.963348 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:44.963294    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d7da87-2345-40ed-a5cb-96c2438eaf5a-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:44:44.963348 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:44.963304    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9hpr\" (UniqueName: \"kubernetes.io/projected/43d7da87-2345-40ed-a5cb-96c2438eaf5a-kube-api-access-f9hpr\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:44:45.686234 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:45.686179    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8c4l4"
Oct 30 13:44:45.686234 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:45.686194    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8c4l4" event={"ID":"43d7da87-2345-40ed-a5cb-96c2438eaf5a","Type":"ContainerDied","Data":"6d181346b604f81a962402174351e03b287e77254d6202c729315d6c077a2124"}
Oct 30 13:44:45.686486 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:45.686273    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d181346b604f81a962402174351e03b287e77254d6202c729315d6c077a2124"
Oct 30 13:44:59.005459 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:44:59.005426    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:44:59.199923 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:59.199845    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:44:59.367937 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:44:59.367887    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:45:00.154407 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:00.154362    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889465-7wnjs"]
Oct 30 13:45:00.154896 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:00.154764    4544 cpu_manager.go:414] "containerMap: removing container" podUID="43d7da87-2345-40ed-a5cb-96c2438eaf5a" containerName="revoke-keys"
Oct 30 13:45:00.154896 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:00.154777    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43d7da87-2345-40ed-a5cb-96c2438eaf5a" containerName="revoke-keys"
Oct 30 13:45:00.154896 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:00.154873    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="43d7da87-2345-40ed-a5cb-96c2438eaf5a" containerName="revoke-keys"
Oct 30 13:45:00.157634 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:00.157617    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-7wnjs"
Oct 30 13:45:00.179854 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:00.179830    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:45:00.194737 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:00.194645    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889465-7wnjs"]
Oct 30 13:45:00.302700 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:00.302659    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st65p\" (UniqueName: \"kubernetes.io/projected/5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2-kube-api-access-st65p\") pod \"maas-api-key-cleanup-29889465-7wnjs\" (UID: \"5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-7wnjs"
Oct 30 13:45:00.302907 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:00.302736    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889465-7wnjs\" (UID: \"5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-7wnjs"
Oct 30 13:45:00.403961 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:00.403926    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889465-7wnjs\" (UID: \"5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-7wnjs"
Oct 30 13:45:00.404147 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:00.403990    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st65p\" (UniqueName: \"kubernetes.io/projected/5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2-kube-api-access-st65p\") pod \"maas-api-key-cleanup-29889465-7wnjs\" (UID: \"5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-7wnjs"
Oct 30 13:45:00.404601 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:00.404556    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889465-7wnjs\" (UID: \"5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-7wnjs"
Oct 30 13:45:00.427461 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:00.427434    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st65p\" (UniqueName: \"kubernetes.io/projected/5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2-kube-api-access-st65p\") pod \"maas-api-key-cleanup-29889465-7wnjs\" (UID: \"5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-7wnjs"
Oct 30 13:45:00.470813 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:00.470786    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-7wnjs"
Oct 30 13:45:00.784629 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:00.784547    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889465-7wnjs"]
Oct 30 13:45:00.788482 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:45:00.788456    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e6b2edf_8a61_4500_a758_9b1ff0b1b8e2.slice/crio-ab8ca4ef2aaf78b1dd68b749fa34c29a9c9a9b2011dcd54cb1c911171fa9b676 WatchSource:0}: Error finding container ab8ca4ef2aaf78b1dd68b749fa34c29a9c9a9b2011dcd54cb1c911171fa9b676: Status 404 returned error can't find the container with id ab8ca4ef2aaf78b1dd68b749fa34c29a9c9a9b2011dcd54cb1c911171fa9b676
Oct 30 13:45:01.771040 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:01.770957    4544 generic.go:356] "Generic (PLEG): container finished" podID="5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2" containerID="a856e273378a6588630fc7bc1c2510eaa69f3fb2bc1767a1bf4f09429928a59f" exitCode=60
Oct 30 13:45:01.771040 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:01.771032    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-7wnjs" event={"ID":"5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2","Type":"ContainerDied","Data":"a856e273378a6588630fc7bc1c2510eaa69f3fb2bc1767a1bf4f09429928a59f"}
Oct 30 13:45:01.771473 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:01.771072    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-7wnjs" event={"ID":"5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2","Type":"ContainerStarted","Data":"ab8ca4ef2aaf78b1dd68b749fa34c29a9c9a9b2011dcd54cb1c911171fa9b676"}
Oct 30 13:45:01.771473 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:01.771290    4544 scope.go:122] "RemoveContainer" containerID="a856e273378a6588630fc7bc1c2510eaa69f3fb2bc1767a1bf4f09429928a59f"
Oct 30 13:45:02.778075 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:02.777985    4544 generic.go:356] "Generic (PLEG): container finished" podID="5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2" containerID="0af8363d4d0c8062f859ad7d3be0d703d3a87c6060fe73b6de2bd1cdc9d99af3" exitCode=60
Oct 30 13:45:02.778579 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:02.778075    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-7wnjs" event={"ID":"5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2","Type":"ContainerDied","Data":"0af8363d4d0c8062f859ad7d3be0d703d3a87c6060fe73b6de2bd1cdc9d99af3"}
Oct 30 13:45:02.778579 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:02.778147    4544 scope.go:122] "RemoveContainer" containerID="a856e273378a6588630fc7bc1c2510eaa69f3fb2bc1767a1bf4f09429928a59f"
Oct 30 13:45:02.778579 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:02.778326    4544 scope.go:122] "RemoveContainer" containerID="0af8363d4d0c8062f859ad7d3be0d703d3a87c6060fe73b6de2bd1cdc9d99af3"
Oct 30 13:45:02.778756 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:45:02.778585    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889465-7wnjs_odh-ai-gateway-infra(5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-7wnjs" podUID="5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2"
Oct 30 13:45:03.785074 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:03.785043    4544 scope.go:122] "RemoveContainer" containerID="0af8363d4d0c8062f859ad7d3be0d703d3a87c6060fe73b6de2bd1cdc9d99af3"
Oct 30 13:45:03.785583 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:45:03.785307    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889465-7wnjs_odh-ai-gateway-infra(5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-7wnjs" podUID="5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2"
Oct 30 13:45:03.898697 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:03.898647    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"]
Oct 30 13:45:03.903504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:03.903480    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"
Oct 30 13:45:03.934045 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:03.934017    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"]
Oct 30 13:45:04.038554 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.038459    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61006021-7c0a-4e04-bb5a-c8b2079b3a3e-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-9cf5j\" (UID: \"61006021-7c0a-4e04-bb5a-c8b2079b3a3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"
Oct 30 13:45:04.038554 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.038502    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61006021-7c0a-4e04-bb5a-c8b2079b3a3e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-9cf5j\" (UID: \"61006021-7c0a-4e04-bb5a-c8b2079b3a3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"
Oct 30 13:45:04.038763 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.038585    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61006021-7c0a-4e04-bb5a-c8b2079b3a3e-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-9cf5j\" (UID: \"61006021-7c0a-4e04-bb5a-c8b2079b3a3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"
Oct 30 13:45:04.038763 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.038609    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8ktr\" (UniqueName: \"kubernetes.io/projected/61006021-7c0a-4e04-bb5a-c8b2079b3a3e-kube-api-access-h8ktr\") pod \"maas-default-gateway-openshift-default-58c6585447-9cf5j\" (UID: \"61006021-7c0a-4e04-bb5a-c8b2079b3a3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"
Oct 30 13:45:04.038763 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.038652    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61006021-7c0a-4e04-bb5a-c8b2079b3a3e-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-9cf5j\" (UID: \"61006021-7c0a-4e04-bb5a-c8b2079b3a3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"
Oct 30 13:45:04.038763 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.038728    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61006021-7c0a-4e04-bb5a-c8b2079b3a3e-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-9cf5j\" (UID: \"61006021-7c0a-4e04-bb5a-c8b2079b3a3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"
Oct 30 13:45:04.038906 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.038778    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61006021-7c0a-4e04-bb5a-c8b2079b3a3e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-9cf5j\" (UID: \"61006021-7c0a-4e04-bb5a-c8b2079b3a3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"
Oct 30 13:45:04.038906 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.038795    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61006021-7c0a-4e04-bb5a-c8b2079b3a3e-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-9cf5j\" (UID: \"61006021-7c0a-4e04-bb5a-c8b2079b3a3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"
Oct 30 13:45:04.038906 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.038843    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61006021-7c0a-4e04-bb5a-c8b2079b3a3e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-9cf5j\" (UID: \"61006021-7c0a-4e04-bb5a-c8b2079b3a3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"
Oct 30 13:45:04.139614 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.139570    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61006021-7c0a-4e04-bb5a-c8b2079b3a3e-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-9cf5j\" (UID: \"61006021-7c0a-4e04-bb5a-c8b2079b3a3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"
Oct 30 13:45:04.139614 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.139617    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61006021-7c0a-4e04-bb5a-c8b2079b3a3e-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-9cf5j\" (UID: \"61006021-7c0a-4e04-bb5a-c8b2079b3a3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"
Oct 30 13:45:04.139883 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.139659    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61006021-7c0a-4e04-bb5a-c8b2079b3a3e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-9cf5j\" (UID: \"61006021-7c0a-4e04-bb5a-c8b2079b3a3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"
Oct 30 13:45:04.139883 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.139686    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61006021-7c0a-4e04-bb5a-c8b2079b3a3e-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-9cf5j\" (UID: \"61006021-7c0a-4e04-bb5a-c8b2079b3a3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"
Oct 30 13:45:04.139883 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.139708    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61006021-7c0a-4e04-bb5a-c8b2079b3a3e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-9cf5j\" (UID: \"61006021-7c0a-4e04-bb5a-c8b2079b3a3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"
Oct 30 13:45:04.140064 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.139877    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61006021-7c0a-4e04-bb5a-c8b2079b3a3e-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-9cf5j\" (UID: \"61006021-7c0a-4e04-bb5a-c8b2079b3a3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"
Oct 30 13:45:04.140064 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.139938    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61006021-7c0a-4e04-bb5a-c8b2079b3a3e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-9cf5j\" (UID: \"61006021-7c0a-4e04-bb5a-c8b2079b3a3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"
Oct 30 13:45:04.140064 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.140006    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61006021-7c0a-4e04-bb5a-c8b2079b3a3e-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-9cf5j\" (UID: \"61006021-7c0a-4e04-bb5a-c8b2079b3a3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"
Oct 30 13:45:04.140064 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.140045    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8ktr\" (UniqueName: \"kubernetes.io/projected/61006021-7c0a-4e04-bb5a-c8b2079b3a3e-kube-api-access-h8ktr\") pod \"maas-default-gateway-openshift-default-58c6585447-9cf5j\" (UID: \"61006021-7c0a-4e04-bb5a-c8b2079b3a3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"
Oct 30 13:45:04.140312 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.140127    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61006021-7c0a-4e04-bb5a-c8b2079b3a3e-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-9cf5j\" (UID: \"61006021-7c0a-4e04-bb5a-c8b2079b3a3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"
Oct 30 13:45:04.140312 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.140047    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61006021-7c0a-4e04-bb5a-c8b2079b3a3e-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-9cf5j\" (UID: \"61006021-7c0a-4e04-bb5a-c8b2079b3a3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"
Oct 30 13:45:04.140312 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.140216    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61006021-7c0a-4e04-bb5a-c8b2079b3a3e-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-9cf5j\" (UID: \"61006021-7c0a-4e04-bb5a-c8b2079b3a3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"
Oct 30 13:45:04.140480 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.140319    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61006021-7c0a-4e04-bb5a-c8b2079b3a3e-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-9cf5j\" (UID: \"61006021-7c0a-4e04-bb5a-c8b2079b3a3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"
Oct 30 13:45:04.140770 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.140746    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61006021-7c0a-4e04-bb5a-c8b2079b3a3e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-9cf5j\" (UID: \"61006021-7c0a-4e04-bb5a-c8b2079b3a3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"
Oct 30 13:45:04.142067 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.142043    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61006021-7c0a-4e04-bb5a-c8b2079b3a3e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-9cf5j\" (UID: \"61006021-7c0a-4e04-bb5a-c8b2079b3a3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"
Oct 30 13:45:04.142344 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.142314    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61006021-7c0a-4e04-bb5a-c8b2079b3a3e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-9cf5j\" (UID: \"61006021-7c0a-4e04-bb5a-c8b2079b3a3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"
Oct 30 13:45:04.163926 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.163886    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61006021-7c0a-4e04-bb5a-c8b2079b3a3e-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-9cf5j\" (UID: \"61006021-7c0a-4e04-bb5a-c8b2079b3a3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"
Oct 30 13:45:04.164080 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.163964    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8ktr\" (UniqueName: \"kubernetes.io/projected/61006021-7c0a-4e04-bb5a-c8b2079b3a3e-kube-api-access-h8ktr\") pod \"maas-default-gateway-openshift-default-58c6585447-9cf5j\" (UID: \"61006021-7c0a-4e04-bb5a-c8b2079b3a3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"
Oct 30 13:45:04.217349 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.217313    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"
Oct 30 13:45:04.543504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.543465    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"]
Oct 30 13:45:04.549881 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:45:04.549841    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61006021_7c0a_4e04_bb5a_c8b2079b3a3e.slice/crio-f2eb63cfca1e69b8c5c54687922159e429fbde539cc8f3761830c64a86d0a6a4 WatchSource:0}: Error finding container f2eb63cfca1e69b8c5c54687922159e429fbde539cc8f3761830c64a86d0a6a4: Status 404 returned error can't find the container with id f2eb63cfca1e69b8c5c54687922159e429fbde539cc8f3761830c64a86d0a6a4
Oct 30 13:45:04.552154 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.552115    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:45:04.552291 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.552204    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:45:04.552291 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.552266    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:45:04.794296 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.794139    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j" event={"ID":"61006021-7c0a-4e04-bb5a-c8b2079b3a3e","Type":"ContainerStarted","Data":"bb93c6009b2f1d10b62b235ea0cf56d2d53afa8a5efcf2de02304bf71a829218"}
Oct 30 13:45:04.794692 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.794338    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j" event={"ID":"61006021-7c0a-4e04-bb5a-c8b2079b3a3e","Type":"ContainerStarted","Data":"f2eb63cfca1e69b8c5c54687922159e429fbde539cc8f3761830c64a86d0a6a4"}
Oct 30 13:45:04.850295 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:04.850229    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j" podStartSLOduration=1.8501997719999999 podStartE2EDuration="1.850199772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:04.847950832 +0000 UTC m=+3599.895084785" watchObservedRunningTime="2026-10-30 13:45:04.850199772 +0000 UTC m=+3599.897333725"
Oct 30 13:45:05.237876 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:05.237707    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"
Oct 30 13:45:05.239599 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:05.239568    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-9cf5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.95:15021/healthz/ready\": dial tcp 10.133.1.95:15021: connect: connection refused" start-of-body=
Oct 30 13:45:05.239780 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:05.239626    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j" podUID="61006021-7c0a-4e04-bb5a-c8b2079b3a3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.95:15021/healthz/ready\": dial tcp 10.133.1.95:15021: connect: connection refused"
Oct 30 13:45:06.226247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:06.222661    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-9cf5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.95:15021/healthz/ready\": dial tcp 10.133.1.95:15021: connect: connection refused" start-of-body=
Oct 30 13:45:06.226247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:06.222726    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j" podUID="61006021-7c0a-4e04-bb5a-c8b2079b3a3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.95:15021/healthz/ready\": dial tcp 10.133.1.95:15021: connect: connection refused"
Oct 30 13:45:07.232246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:07.226734    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-9cf5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.95:15021/healthz/ready\": dial tcp 10.133.1.95:15021: connect: connection refused" start-of-body=
Oct 30 13:45:07.232246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:07.226802    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j" podUID="61006021-7c0a-4e04-bb5a-c8b2079b3a3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.95:15021/healthz/ready\": dial tcp 10.133.1.95:15021: connect: connection refused"
Oct 30 13:45:08.222246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:08.218717    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-9cf5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.95:15021/healthz/ready\": dial tcp 10.133.1.95:15021: connect: connection refused" start-of-body=
Oct 30 13:45:08.222246 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:08.218808    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j" podUID="61006021-7c0a-4e04-bb5a-c8b2079b3a3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.95:15021/healthz/ready\": dial tcp 10.133.1.95:15021: connect: connection refused"
Oct 30 13:45:09.226247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:09.221708    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-9cf5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.95:15021/healthz/ready\": dial tcp 10.133.1.95:15021: connect: connection refused" start-of-body=
Oct 30 13:45:09.226247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:09.221792    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j" podUID="61006021-7c0a-4e04-bb5a-c8b2079b3a3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.95:15021/healthz/ready\": dial tcp 10.133.1.95:15021: connect: connection refused"
Oct 30 13:45:10.235244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:10.234678    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-9cf5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.95:15021/healthz/ready\": dial tcp 10.133.1.95:15021: connect: connection refused" start-of-body=
Oct 30 13:45:10.235244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:10.234751    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j" podUID="61006021-7c0a-4e04-bb5a-c8b2079b3a3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.95:15021/healthz/ready\": dial tcp 10.133.1.95:15021: connect: connection refused"
Oct 30 13:45:11.217933 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:11.217892    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-9cf5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.95:15021/healthz/ready\": dial tcp 10.133.1.95:15021: connect: connection refused" start-of-body=
Oct 30 13:45:11.218126 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:11.217956    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j" podUID="61006021-7c0a-4e04-bb5a-c8b2079b3a3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.95:15021/healthz/ready\": dial tcp 10.133.1.95:15021: connect: connection refused"
Oct 30 13:45:12.797107 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:12.797068    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"
Oct 30 13:45:12.797995 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:12.797973    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"
Oct 30 13:45:12.798806 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:12.798786    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j"
Oct 30 13:45:15.081452 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:15.081421    4544 scope.go:122] "RemoveContainer" containerID="0af8363d4d0c8062f859ad7d3be0d703d3a87c6060fe73b6de2bd1cdc9d99af3"
Oct 30 13:45:15.130529 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:15.130488    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:45:15.879451 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:15.879415    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-7wnjs" event={"ID":"5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2","Type":"ContainerStarted","Data":"786392183e8eb251a99cd2a36d3535bd7be14e31c1d2eb0163ab995f9f976b4b"}
Oct 30 13:45:15.940496 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:15.940446    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-7wnjs" podStartSLOduration=15.940431078 podStartE2EDuration="15.940431078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:15.939568902 +0000 UTC m=+3610.986702856" watchObservedRunningTime="2026-10-30 13:45:15.940431078 +0000 UTC m=+3610.987565031"
Oct 30 13:45:16.889537 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:16.889503    4544 generic.go:356] "Generic (PLEG): container finished" podID="5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2" containerID="786392183e8eb251a99cd2a36d3535bd7be14e31c1d2eb0163ab995f9f976b4b" exitCode=60
Oct 30 13:45:16.889997 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:16.889583    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-7wnjs" event={"ID":"5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2","Type":"ContainerDied","Data":"786392183e8eb251a99cd2a36d3535bd7be14e31c1d2eb0163ab995f9f976b4b"}
Oct 30 13:45:16.889997 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:16.889667    4544 scope.go:122] "RemoveContainer" containerID="0af8363d4d0c8062f859ad7d3be0d703d3a87c6060fe73b6de2bd1cdc9d99af3"
Oct 30 13:45:16.890125 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:16.890029    4544 scope.go:122] "RemoveContainer" containerID="786392183e8eb251a99cd2a36d3535bd7be14e31c1d2eb0163ab995f9f976b4b"
Oct 30 13:45:16.890296 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:45:16.890273    4544 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889465-7wnjs_odh-ai-gateway-infra(5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-7wnjs" podUID="5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2"
Oct 30 13:45:16.953587 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:16.953552    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889465-7wnjs"]
Oct 30 13:45:18.095196 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:18.095168    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-7wnjs"
Oct 30 13:45:18.192585 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:18.192174    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2-kube-api-access-st65p\" (UniqueName: \"kubernetes.io/projected/5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2-kube-api-access-st65p\") pod \"5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2\" (UID: \"5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2\") "
Oct 30 13:45:18.192585 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:18.192250    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2-maas-api-service-ca\") pod \"5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2\" (UID: \"5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2\") "
Oct 30 13:45:18.192855 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:18.192582    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2-maas-api-service-ca" pod "5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2" (UID: "5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:18.194269 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:18.194240    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2-kube-api-access-st65p" pod "5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2" (UID: "5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2"). InnerVolumeSpecName "kube-api-access-st65p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:18.293722 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:18.293683    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st65p\" (UniqueName: \"kubernetes.io/projected/5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2-kube-api-access-st65p\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:45:18.293722 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:18.293716    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2-maas-api-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:45:18.904718 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:18.904686    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-7wnjs"
Oct 30 13:45:18.904950 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:18.904725    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-7wnjs" event={"ID":"5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2","Type":"ContainerDied","Data":"ab8ca4ef2aaf78b1dd68b749fa34c29a9c9a9b2011dcd54cb1c911171fa9b676"}
Oct 30 13:45:18.904950 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:18.904786    4544 scope.go:122] "RemoveContainer" containerID="786392183e8eb251a99cd2a36d3535bd7be14e31c1d2eb0163ab995f9f976b4b"
Oct 30 13:45:18.962992 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:18.962960    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"]
Oct 30 13:45:18.963514 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:18.963489    4544 cpu_manager.go:414] "containerMap: removing container" podUID="5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2" containerName="cleanup"
Oct 30 13:45:18.963514 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:18.963512    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2" containerName="cleanup"
Oct 30 13:45:18.963793 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:18.963529    4544 cpu_manager.go:414] "containerMap: removing container" podUID="5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2" containerName="cleanup"
Oct 30 13:45:18.963793 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:18.963537    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2" containerName="cleanup"
Oct 30 13:45:18.963793 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:18.963550    4544 cpu_manager.go:414] "containerMap: removing container" podUID="5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2" containerName="cleanup"
Oct 30 13:45:18.963793 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:18.963557    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2" containerName="cleanup"
Oct 30 13:45:18.963793 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:18.963703    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2" containerName="cleanup"
Oct 30 13:45:18.963793 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:18.963714    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2" containerName="cleanup"
Oct 30 13:45:18.968500 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:18.968473    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"
Oct 30 13:45:18.982537 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:18.982508    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889465-7wnjs"]
Oct 30 13:45:18.989344 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:18.989313    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889465-7wnjs"]
Oct 30 13:45:18.996197 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:18.996163    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"]
Oct 30 13:45:19.001929 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.001896    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfb6ceaf-36c4-4a7e-8d4d-72efeecef675-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-fdnx5\" (UID: \"bfb6ceaf-36c4-4a7e-8d4d-72efeecef675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"
Oct 30 13:45:19.002122 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.001947    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfb6ceaf-36c4-4a7e-8d4d-72efeecef675-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-fdnx5\" (UID: \"bfb6ceaf-36c4-4a7e-8d4d-72efeecef675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"
Oct 30 13:45:19.002122 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.002069    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfb6ceaf-36c4-4a7e-8d4d-72efeecef675-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-fdnx5\" (UID: \"bfb6ceaf-36c4-4a7e-8d4d-72efeecef675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"
Oct 30 13:45:19.002122 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.002115    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfb6ceaf-36c4-4a7e-8d4d-72efeecef675-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-fdnx5\" (UID: \"bfb6ceaf-36c4-4a7e-8d4d-72efeecef675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"
Oct 30 13:45:19.002360 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.002170    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb6ceaf-36c4-4a7e-8d4d-72efeecef675-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-fdnx5\" (UID: \"bfb6ceaf-36c4-4a7e-8d4d-72efeecef675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"
Oct 30 13:45:19.002360 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.002205    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj755\" (UniqueName: \"kubernetes.io/projected/bfb6ceaf-36c4-4a7e-8d4d-72efeecef675-kube-api-access-bj755\") pod \"maas-default-gateway-openshift-default-58c6585447-fdnx5\" (UID: \"bfb6ceaf-36c4-4a7e-8d4d-72efeecef675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"
Oct 30 13:45:19.002360 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.002254    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfb6ceaf-36c4-4a7e-8d4d-72efeecef675-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-fdnx5\" (UID: \"bfb6ceaf-36c4-4a7e-8d4d-72efeecef675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"
Oct 30 13:45:19.002360 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.002295    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfb6ceaf-36c4-4a7e-8d4d-72efeecef675-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-fdnx5\" (UID: \"bfb6ceaf-36c4-4a7e-8d4d-72efeecef675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"
Oct 30 13:45:19.002360 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.002331    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb6ceaf-36c4-4a7e-8d4d-72efeecef675-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-fdnx5\" (UID: \"bfb6ceaf-36c4-4a7e-8d4d-72efeecef675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"
Oct 30 13:45:19.084450 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.084411    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2" path="/var/lib/kubelet/pods/5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2/volumes"
Oct 30 13:45:19.103170 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.103133    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb6ceaf-36c4-4a7e-8d4d-72efeecef675-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-fdnx5\" (UID: \"bfb6ceaf-36c4-4a7e-8d4d-72efeecef675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"
Oct 30 13:45:19.103170 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.103174    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj755\" (UniqueName: \"kubernetes.io/projected/bfb6ceaf-36c4-4a7e-8d4d-72efeecef675-kube-api-access-bj755\") pod \"maas-default-gateway-openshift-default-58c6585447-fdnx5\" (UID: \"bfb6ceaf-36c4-4a7e-8d4d-72efeecef675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"
Oct 30 13:45:19.103781 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.103194    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfb6ceaf-36c4-4a7e-8d4d-72efeecef675-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-fdnx5\" (UID: \"bfb6ceaf-36c4-4a7e-8d4d-72efeecef675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"
Oct 30 13:45:19.103781 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.103236    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfb6ceaf-36c4-4a7e-8d4d-72efeecef675-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-fdnx5\" (UID: \"bfb6ceaf-36c4-4a7e-8d4d-72efeecef675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"
Oct 30 13:45:19.103781 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.103270    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb6ceaf-36c4-4a7e-8d4d-72efeecef675-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-fdnx5\" (UID: \"bfb6ceaf-36c4-4a7e-8d4d-72efeecef675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"
Oct 30 13:45:19.103781 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.103327    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfb6ceaf-36c4-4a7e-8d4d-72efeecef675-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-fdnx5\" (UID: \"bfb6ceaf-36c4-4a7e-8d4d-72efeecef675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"
Oct 30 13:45:19.103781 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.103362    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfb6ceaf-36c4-4a7e-8d4d-72efeecef675-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-fdnx5\" (UID: \"bfb6ceaf-36c4-4a7e-8d4d-72efeecef675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"
Oct 30 13:45:19.103781 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.103399    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfb6ceaf-36c4-4a7e-8d4d-72efeecef675-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-fdnx5\" (UID: \"bfb6ceaf-36c4-4a7e-8d4d-72efeecef675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"
Oct 30 13:45:19.103781 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.103433    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfb6ceaf-36c4-4a7e-8d4d-72efeecef675-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-fdnx5\" (UID: \"bfb6ceaf-36c4-4a7e-8d4d-72efeecef675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"
Oct 30 13:45:19.103781 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.103733    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb6ceaf-36c4-4a7e-8d4d-72efeecef675-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-fdnx5\" (UID: \"bfb6ceaf-36c4-4a7e-8d4d-72efeecef675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"
Oct 30 13:45:19.103781 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.103771    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb6ceaf-36c4-4a7e-8d4d-72efeecef675-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-fdnx5\" (UID: \"bfb6ceaf-36c4-4a7e-8d4d-72efeecef675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"
Oct 30 13:45:19.104286 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.104136    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfb6ceaf-36c4-4a7e-8d4d-72efeecef675-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-fdnx5\" (UID: \"bfb6ceaf-36c4-4a7e-8d4d-72efeecef675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"
Oct 30 13:45:19.104286 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.104150    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfb6ceaf-36c4-4a7e-8d4d-72efeecef675-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-fdnx5\" (UID: \"bfb6ceaf-36c4-4a7e-8d4d-72efeecef675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"
Oct 30 13:45:19.104286 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.104137    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfb6ceaf-36c4-4a7e-8d4d-72efeecef675-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-fdnx5\" (UID: \"bfb6ceaf-36c4-4a7e-8d4d-72efeecef675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"
Oct 30 13:45:19.105926 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.105910    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfb6ceaf-36c4-4a7e-8d4d-72efeecef675-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-fdnx5\" (UID: \"bfb6ceaf-36c4-4a7e-8d4d-72efeecef675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"
Oct 30 13:45:19.106360 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.106342    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfb6ceaf-36c4-4a7e-8d4d-72efeecef675-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-fdnx5\" (UID: \"bfb6ceaf-36c4-4a7e-8d4d-72efeecef675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"
Oct 30 13:45:19.127630 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.127594    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfb6ceaf-36c4-4a7e-8d4d-72efeecef675-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-fdnx5\" (UID: \"bfb6ceaf-36c4-4a7e-8d4d-72efeecef675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"
Oct 30 13:45:19.127777 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.127663    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj755\" (UniqueName: \"kubernetes.io/projected/bfb6ceaf-36c4-4a7e-8d4d-72efeecef675-kube-api-access-bj755\") pod \"maas-default-gateway-openshift-default-58c6585447-fdnx5\" (UID: \"bfb6ceaf-36c4-4a7e-8d4d-72efeecef675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"
Oct 30 13:45:19.283445 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.283345    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"
Oct 30 13:45:19.611549 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.611510    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"]
Oct 30 13:45:19.615874 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:45:19.615844    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb6ceaf_36c4_4a7e_8d4d_72efeecef675.slice/crio-15cb2938465cf5bb62152f64d12dc582648ca964d37f4de86d6a9c3b32125dfd WatchSource:0}: Error finding container 15cb2938465cf5bb62152f64d12dc582648ca964d37f4de86d6a9c3b32125dfd: Status 404 returned error can't find the container with id 15cb2938465cf5bb62152f64d12dc582648ca964d37f4de86d6a9c3b32125dfd
Oct 30 13:45:19.618269 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.618216    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:45:19.618382 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.618327    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:45:19.618382 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.618369    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:45:19.914053 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.913965    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5" event={"ID":"bfb6ceaf-36c4-4a7e-8d4d-72efeecef675","Type":"ContainerStarted","Data":"8e01ed803f743cb78858ba324353ff07d13e9612a11fff46a21506c0448b0b68"}
Oct 30 13:45:19.914053 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.914018    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5" event={"ID":"bfb6ceaf-36c4-4a7e-8d4d-72efeecef675","Type":"ContainerStarted","Data":"15cb2938465cf5bb62152f64d12dc582648ca964d37f4de86d6a9c3b32125dfd"}
Oct 30 13:45:19.966822 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:19.966765    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5" podStartSLOduration=1.966750068 podStartE2EDuration="1.966750068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:19.965627909 +0000 UTC m=+3615.012761862" watchObservedRunningTime="2026-10-30 13:45:19.966750068 +0000 UTC m=+3615.013884017"
Oct 30 13:45:20.284253 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:20.284115    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"
Oct 30 13:45:20.285968 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:20.285937    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-fdnx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" start-of-body=
Oct 30 13:45:20.286096 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:20.285994    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5" podUID="bfb6ceaf-36c4-4a7e-8d4d-72efeecef675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused"
Oct 30 13:45:21.284157 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:21.284112    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-fdnx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" start-of-body=
Oct 30 13:45:21.284365 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:21.284172    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5" podUID="bfb6ceaf-36c4-4a7e-8d4d-72efeecef675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused"
Oct 30 13:45:22.283932 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:22.283894    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-fdnx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" start-of-body=
Oct 30 13:45:22.284158 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:22.283958    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5" podUID="bfb6ceaf-36c4-4a7e-8d4d-72efeecef675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused"
Oct 30 13:45:23.039969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:23.039927    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16e62535-gwl8v"]
Oct 30 13:45:23.051770 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:23.051732    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16e62535-gwl8v"]
Oct 30 13:45:23.085152 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:23.085114    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84aeefc2-4217-4779-904d-e081cca145bc" path="/var/lib/kubelet/pods/84aeefc2-4217-4779-904d-e081cca145bc/volumes"
Oct 30 13:45:23.284079 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:23.284037    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-fdnx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" start-of-body=
Oct 30 13:45:23.284298 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:23.284094    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5" podUID="bfb6ceaf-36c4-4a7e-8d4d-72efeecef675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused"
Oct 30 13:45:24.033255 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:24.031846    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b45652a-nxdb5"]
Oct 30 13:45:24.041818 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:24.041788    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13fe37db-b66zr"]
Oct 30 13:45:24.044055 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:24.044031    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b45652a-nxdb5"]
Oct 30 13:45:24.045577 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:24.045557    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13fe37db-b66zr"]
Oct 30 13:45:24.284277 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:24.284155    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-fdnx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" start-of-body=
Oct 30 13:45:24.284277 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:24.284243    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5" podUID="bfb6ceaf-36c4-4a7e-8d4d-72efeecef675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused"
Oct 30 13:45:25.092330 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:25.092296    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19afa216-d512-4110-8641-bdfa5e42c7fd" path="/var/lib/kubelet/pods/19afa216-d512-4110-8641-bdfa5e42c7fd/volumes"
Oct 30 13:45:25.092916 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:25.092766    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9df75e04-9e0b-46b5-8a95-6b4be84e3314" path="/var/lib/kubelet/pods/9df75e04-9e0b-46b5-8a95-6b4be84e3314/volumes"
Oct 30 13:45:25.284255 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:25.284188    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-fdnx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" start-of-body=
Oct 30 13:45:25.284484 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:25.284295    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5" podUID="bfb6ceaf-36c4-4a7e-8d4d-72efeecef675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused"
Oct 30 13:45:26.284814 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:26.284773    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-fdnx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" start-of-body=
Oct 30 13:45:26.285347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:26.284839    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5" podUID="bfb6ceaf-36c4-4a7e-8d4d-72efeecef675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused"
Oct 30 13:45:27.955729 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:27.955690    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"
Oct 30 13:45:27.956306 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:27.955999    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"
Oct 30 13:45:27.957012 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:27.956993    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5"
Oct 30 13:45:29.006453 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:45:29.006413    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:45:41.524518 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:41.524475    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xwvvp"]
Oct 30 13:45:41.525312 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:41.525283    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e6b2edf-8a61-4500-a758-9b1ff0b1b8e2" containerName="cleanup"
Oct 30 13:45:41.528585 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:41.528559    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xwvvp"
Oct 30 13:45:41.543933 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:41.543902    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:45:41.555566 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:41.555531    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:45:41.558395 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:41.558359    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xwvvp"]
Oct 30 13:45:41.574632 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:41.574593    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwpx4\" (UniqueName: \"kubernetes.io/projected/819366a4-ff59-41e8-a51a-51dba3404a76-kube-api-access-rwpx4\") pod \"maas-api-revoke-sub-046e1343-xwvvp\" (UID: \"819366a4-ff59-41e8-a51a-51dba3404a76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xwvvp"
Oct 30 13:45:41.574837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:41.574649    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/819366a4-ff59-41e8-a51a-51dba3404a76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-xwvvp\" (UID: \"819366a4-ff59-41e8-a51a-51dba3404a76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xwvvp"
Oct 30 13:45:41.574837 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:41.574778    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/819366a4-ff59-41e8-a51a-51dba3404a76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-xwvvp\" (UID: \"819366a4-ff59-41e8-a51a-51dba3404a76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xwvvp"
Oct 30 13:45:41.676117 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:41.676073    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/819366a4-ff59-41e8-a51a-51dba3404a76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-xwvvp\" (UID: \"819366a4-ff59-41e8-a51a-51dba3404a76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xwvvp"
Oct 30 13:45:41.676389 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:41.676135    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/819366a4-ff59-41e8-a51a-51dba3404a76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-xwvvp\" (UID: \"819366a4-ff59-41e8-a51a-51dba3404a76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xwvvp"
Oct 30 13:45:41.676389 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:41.676201    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwpx4\" (UniqueName: \"kubernetes.io/projected/819366a4-ff59-41e8-a51a-51dba3404a76-kube-api-access-rwpx4\") pod \"maas-api-revoke-sub-046e1343-xwvvp\" (UID: \"819366a4-ff59-41e8-a51a-51dba3404a76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xwvvp"
Oct 30 13:45:41.676885 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:41.676857    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/819366a4-ff59-41e8-a51a-51dba3404a76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-xwvvp\" (UID: \"819366a4-ff59-41e8-a51a-51dba3404a76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xwvvp"
Oct 30 13:45:41.678608 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:41.678582    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/819366a4-ff59-41e8-a51a-51dba3404a76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-xwvvp\" (UID: \"819366a4-ff59-41e8-a51a-51dba3404a76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xwvvp"
Oct 30 13:45:41.699956 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:41.699920    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwpx4\" (UniqueName: \"kubernetes.io/projected/819366a4-ff59-41e8-a51a-51dba3404a76-kube-api-access-rwpx4\") pod \"maas-api-revoke-sub-046e1343-xwvvp\" (UID: \"819366a4-ff59-41e8-a51a-51dba3404a76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xwvvp"
Oct 30 13:45:41.842039 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:41.841988    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xwvvp"
Oct 30 13:45:42.141194 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:42.141081    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xwvvp"]
Oct 30 13:45:42.146679 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:45:42.146649    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod819366a4_ff59_41e8_a51a_51dba3404a76.slice/crio-46bab3f36e820f9ae2260601b40cb42af163a0a700dc0fe32616c6cfc0829843 WatchSource:0}: Error finding container 46bab3f36e820f9ae2260601b40cb42af163a0a700dc0fe32616c6cfc0829843: Status 404 returned error can't find the container with id 46bab3f36e820f9ae2260601b40cb42af163a0a700dc0fe32616c6cfc0829843
Oct 30 13:45:43.060402 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:43.060366    4544 generic.go:356] "Generic (PLEG): container finished" podID="819366a4-ff59-41e8-a51a-51dba3404a76" containerID="b3c40ecaea0831886f4cc1d14da88e5be80e1f9273c3ebdbe07352e99550dac3" exitCode=0
Oct 30 13:45:43.060793 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:43.060448    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xwvvp" event={"ID":"819366a4-ff59-41e8-a51a-51dba3404a76","Type":"ContainerDied","Data":"b3c40ecaea0831886f4cc1d14da88e5be80e1f9273c3ebdbe07352e99550dac3"}
Oct 30 13:45:43.060793 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:43.060500    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xwvvp" event={"ID":"819366a4-ff59-41e8-a51a-51dba3404a76","Type":"ContainerStarted","Data":"46bab3f36e820f9ae2260601b40cb42af163a0a700dc0fe32616c6cfc0829843"}
Oct 30 13:45:44.201396 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:44.201369    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xwvvp"
Oct 30 13:45:44.299319 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:44.299281    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/819366a4-ff59-41e8-a51a-51dba3404a76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/819366a4-ff59-41e8-a51a-51dba3404a76-maas-api-openshift-ca\") pod \"819366a4-ff59-41e8-a51a-51dba3404a76\" (UID: \"819366a4-ff59-41e8-a51a-51dba3404a76\") "
Oct 30 13:45:44.299515 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:44.299383    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/819366a4-ff59-41e8-a51a-51dba3404a76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/819366a4-ff59-41e8-a51a-51dba3404a76-maas-api-xks-ca\") pod \"819366a4-ff59-41e8-a51a-51dba3404a76\" (UID: \"819366a4-ff59-41e8-a51a-51dba3404a76\") "
Oct 30 13:45:44.299515 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:44.299436    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/819366a4-ff59-41e8-a51a-51dba3404a76-kube-api-access-rwpx4\" (UniqueName: \"kubernetes.io/projected/819366a4-ff59-41e8-a51a-51dba3404a76-kube-api-access-rwpx4\") pod \"819366a4-ff59-41e8-a51a-51dba3404a76\" (UID: \"819366a4-ff59-41e8-a51a-51dba3404a76\") "
Oct 30 13:45:44.299687 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:44.299662    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/819366a4-ff59-41e8-a51a-51dba3404a76-maas-api-openshift-ca" pod "819366a4-ff59-41e8-a51a-51dba3404a76" (UID: "819366a4-ff59-41e8-a51a-51dba3404a76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:44.301478 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:44.301445    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/819366a4-ff59-41e8-a51a-51dba3404a76-maas-api-xks-ca" pod "819366a4-ff59-41e8-a51a-51dba3404a76" (UID: "819366a4-ff59-41e8-a51a-51dba3404a76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:45:44.301579 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:44.301494    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/819366a4-ff59-41e8-a51a-51dba3404a76-kube-api-access-rwpx4" pod "819366a4-ff59-41e8-a51a-51dba3404a76" (UID: "819366a4-ff59-41e8-a51a-51dba3404a76"). InnerVolumeSpecName "kube-api-access-rwpx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:44.401032 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:44.400996    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/819366a4-ff59-41e8-a51a-51dba3404a76-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:45:44.401032 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:44.401029    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwpx4\" (UniqueName: \"kubernetes.io/projected/819366a4-ff59-41e8-a51a-51dba3404a76-kube-api-access-rwpx4\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:45:44.401269 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:44.401045    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/819366a4-ff59-41e8-a51a-51dba3404a76-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:45:45.080458 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:45.080425    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xwvvp"
Oct 30 13:45:45.085040 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:45.084989    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xwvvp" event={"ID":"819366a4-ff59-41e8-a51a-51dba3404a76","Type":"ContainerDied","Data":"46bab3f36e820f9ae2260601b40cb42af163a0a700dc0fe32616c6cfc0829843"}
Oct 30 13:45:45.085040 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:45.085048    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46bab3f36e820f9ae2260601b40cb42af163a0a700dc0fe32616c6cfc0829843"
Oct 30 13:45:47.045446 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:47.045359    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7ee8148-77kd8"]
Oct 30 13:45:47.059369 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:47.059329    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7ee8148-77kd8"]
Oct 30 13:45:47.085660 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:47.085623    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e781781-4614-4cb3-84c9-987ca6719f12" path="/var/lib/kubelet/pods/3e781781-4614-4cb3-84c9-987ca6719f12/volumes"
Oct 30 13:45:49.071328 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.071286    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"]
Oct 30 13:45:49.071877 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.071855    4544 cpu_manager.go:414] "containerMap: removing container" podUID="819366a4-ff59-41e8-a51a-51dba3404a76" containerName="revoke-keys"
Oct 30 13:45:49.071928 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.071879    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="819366a4-ff59-41e8-a51a-51dba3404a76" containerName="revoke-keys"
Oct 30 13:45:49.072096 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.072069    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="819366a4-ff59-41e8-a51a-51dba3404a76" containerName="revoke-keys"
Oct 30 13:45:49.075745 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.075718    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"
Oct 30 13:45:49.108595 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.108557    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"]
Oct 30 13:45:49.146000 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.145951    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e9c9db5-5873-40d8-94bb-d827c7698714-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-52t9q\" (UID: \"7e9c9db5-5873-40d8-94bb-d827c7698714\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"
Oct 30 13:45:49.146207 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.146024    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e9c9db5-5873-40d8-94bb-d827c7698714-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-52t9q\" (UID: \"7e9c9db5-5873-40d8-94bb-d827c7698714\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"
Oct 30 13:45:49.146290 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.146256    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e9c9db5-5873-40d8-94bb-d827c7698714-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-52t9q\" (UID: \"7e9c9db5-5873-40d8-94bb-d827c7698714\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"
Oct 30 13:45:49.146330 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.146317    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e9c9db5-5873-40d8-94bb-d827c7698714-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-52t9q\" (UID: \"7e9c9db5-5873-40d8-94bb-d827c7698714\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"
Oct 30 13:45:49.146382 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.146359    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e9c9db5-5873-40d8-94bb-d827c7698714-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-52t9q\" (UID: \"7e9c9db5-5873-40d8-94bb-d827c7698714\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"
Oct 30 13:45:49.146441 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.146415    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9c9db5-5873-40d8-94bb-d827c7698714-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-52t9q\" (UID: \"7e9c9db5-5873-40d8-94bb-d827c7698714\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"
Oct 30 13:45:49.146506 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.146462    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e9c9db5-5873-40d8-94bb-d827c7698714-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-52t9q\" (UID: \"7e9c9db5-5873-40d8-94bb-d827c7698714\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"
Oct 30 13:45:49.146564 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.146505    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9c9db5-5873-40d8-94bb-d827c7698714-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-52t9q\" (UID: \"7e9c9db5-5873-40d8-94bb-d827c7698714\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"
Oct 30 13:45:49.146564 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.146551    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmf6q\" (UniqueName: \"kubernetes.io/projected/7e9c9db5-5873-40d8-94bb-d827c7698714-kube-api-access-nmf6q\") pod \"maas-default-gateway-openshift-default-58c6585447-52t9q\" (UID: \"7e9c9db5-5873-40d8-94bb-d827c7698714\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"
Oct 30 13:45:49.248110 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.248064    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e9c9db5-5873-40d8-94bb-d827c7698714-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-52t9q\" (UID: \"7e9c9db5-5873-40d8-94bb-d827c7698714\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"
Oct 30 13:45:49.248385 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.248123    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9c9db5-5873-40d8-94bb-d827c7698714-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-52t9q\" (UID: \"7e9c9db5-5873-40d8-94bb-d827c7698714\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"
Oct 30 13:45:49.248385 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.248165    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e9c9db5-5873-40d8-94bb-d827c7698714-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-52t9q\" (UID: \"7e9c9db5-5873-40d8-94bb-d827c7698714\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"
Oct 30 13:45:49.248385 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.248200    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9c9db5-5873-40d8-94bb-d827c7698714-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-52t9q\" (UID: \"7e9c9db5-5873-40d8-94bb-d827c7698714\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"
Oct 30 13:45:49.248385 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.248262    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmf6q\" (UniqueName: \"kubernetes.io/projected/7e9c9db5-5873-40d8-94bb-d827c7698714-kube-api-access-nmf6q\") pod \"maas-default-gateway-openshift-default-58c6585447-52t9q\" (UID: \"7e9c9db5-5873-40d8-94bb-d827c7698714\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"
Oct 30 13:45:49.248385 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.248300    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e9c9db5-5873-40d8-94bb-d827c7698714-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-52t9q\" (UID: \"7e9c9db5-5873-40d8-94bb-d827c7698714\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"
Oct 30 13:45:49.248683 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.248539    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e9c9db5-5873-40d8-94bb-d827c7698714-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-52t9q\" (UID: \"7e9c9db5-5873-40d8-94bb-d827c7698714\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"
Oct 30 13:45:49.248683 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.248647    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e9c9db5-5873-40d8-94bb-d827c7698714-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-52t9q\" (UID: \"7e9c9db5-5873-40d8-94bb-d827c7698714\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"
Oct 30 13:45:49.248793 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.248678    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9c9db5-5873-40d8-94bb-d827c7698714-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-52t9q\" (UID: \"7e9c9db5-5873-40d8-94bb-d827c7698714\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"
Oct 30 13:45:49.248793 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.248681    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e9c9db5-5873-40d8-94bb-d827c7698714-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-52t9q\" (UID: \"7e9c9db5-5873-40d8-94bb-d827c7698714\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"
Oct 30 13:45:49.248793 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.248647    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9c9db5-5873-40d8-94bb-d827c7698714-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-52t9q\" (UID: \"7e9c9db5-5873-40d8-94bb-d827c7698714\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"
Oct 30 13:45:49.248970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.248869    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e9c9db5-5873-40d8-94bb-d827c7698714-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-52t9q\" (UID: \"7e9c9db5-5873-40d8-94bb-d827c7698714\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"
Oct 30 13:45:49.248970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.248957    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e9c9db5-5873-40d8-94bb-d827c7698714-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-52t9q\" (UID: \"7e9c9db5-5873-40d8-94bb-d827c7698714\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"
Oct 30 13:45:49.249349 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.249326    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e9c9db5-5873-40d8-94bb-d827c7698714-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-52t9q\" (UID: \"7e9c9db5-5873-40d8-94bb-d827c7698714\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"
Oct 30 13:45:49.251264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.251216    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e9c9db5-5873-40d8-94bb-d827c7698714-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-52t9q\" (UID: \"7e9c9db5-5873-40d8-94bb-d827c7698714\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"
Oct 30 13:45:49.251578 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.251554    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e9c9db5-5873-40d8-94bb-d827c7698714-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-52t9q\" (UID: \"7e9c9db5-5873-40d8-94bb-d827c7698714\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"
Oct 30 13:45:49.272651 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.272617    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e9c9db5-5873-40d8-94bb-d827c7698714-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-52t9q\" (UID: \"7e9c9db5-5873-40d8-94bb-d827c7698714\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"
Oct 30 13:45:49.272829 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.272766    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmf6q\" (UniqueName: \"kubernetes.io/projected/7e9c9db5-5873-40d8-94bb-d827c7698714-kube-api-access-nmf6q\") pod \"maas-default-gateway-openshift-default-58c6585447-52t9q\" (UID: \"7e9c9db5-5873-40d8-94bb-d827c7698714\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"
Oct 30 13:45:49.389531 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.389491    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"
Oct 30 13:45:49.716744 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.716128    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"]
Oct 30 13:45:49.720402 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:45:49.720371    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e9c9db5_5873_40d8_94bb_d827c7698714.slice/crio-ee277e6abb8dc83816b65cbebd5dc90a06ea36b7aaa3866330ab4c13aa5c8477 WatchSource:0}: Error finding container ee277e6abb8dc83816b65cbebd5dc90a06ea36b7aaa3866330ab4c13aa5c8477: Status 404 returned error can't find the container with id ee277e6abb8dc83816b65cbebd5dc90a06ea36b7aaa3866330ab4c13aa5c8477
Oct 30 13:45:49.722648 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.722613    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:45:49.722730 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.722683    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:45:49.722730 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:49.722718    4544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 13:45:50.109799 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:50.109754    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q" event={"ID":"7e9c9db5-5873-40d8-94bb-d827c7698714","Type":"ContainerStarted","Data":"cabc2c95b655a3b0418038da87e19991917ae8921a4a975a28744d795647acc6"}
Oct 30 13:45:50.110259 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:50.109821    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q" event={"ID":"7e9c9db5-5873-40d8-94bb-d827c7698714","Type":"ContainerStarted","Data":"ee277e6abb8dc83816b65cbebd5dc90a06ea36b7aaa3866330ab4c13aa5c8477"}
Oct 30 13:45:50.195003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:50.194943    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q" podStartSLOduration=1.194927756 podStartE2EDuration="1.194927756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:50.182677369 +0000 UTC m=+3645.229811322" watchObservedRunningTime="2026-10-30 13:45:50.194927756 +0000 UTC m=+3645.242061710"
Oct 30 13:45:50.390747 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:50.390645    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"
Oct 30 13:45:50.392353 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:50.392328    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-52t9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.98:15021/healthz/ready\": dial tcp 10.133.1.98:15021: connect: connection refused" start-of-body=
Oct 30 13:45:50.392430 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:50.392381    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q" podUID="7e9c9db5-5873-40d8-94bb-d827c7698714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.98:15021/healthz/ready\": dial tcp 10.133.1.98:15021: connect: connection refused"
Oct 30 13:45:51.390865 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:51.390816    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-52t9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.98:15021/healthz/ready\": dial tcp 10.133.1.98:15021: connect: connection refused" start-of-body=
Oct 30 13:45:51.391329 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:51.390882    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q" podUID="7e9c9db5-5873-40d8-94bb-d827c7698714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.98:15021/healthz/ready\": dial tcp 10.133.1.98:15021: connect: connection refused"
Oct 30 13:45:51.879962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:51.879913    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6jqc6"]
Oct 30 13:45:51.883855 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:51.883826    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6jqc6"
Oct 30 13:45:51.907829 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:51.907791    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6jqc6"]
Oct 30 13:45:51.954763 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:51.954725    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:45:51.954979 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:51.954867    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:45:51.978450 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:51.978407    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4hvv\" (UniqueName: \"kubernetes.io/projected/365c34a6-53b4-4f52-bc89-3408a579937a-kube-api-access-l4hvv\") pod \"maas-api-revoke-sub-d3fba056-6jqc6\" (UID: \"365c34a6-53b4-4f52-bc89-3408a579937a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6jqc6"
Oct 30 13:45:51.978450 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:51.978454    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/365c34a6-53b4-4f52-bc89-3408a579937a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-6jqc6\" (UID: \"365c34a6-53b4-4f52-bc89-3408a579937a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6jqc6"
Oct 30 13:45:51.978689 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:51.978607    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/365c34a6-53b4-4f52-bc89-3408a579937a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-6jqc6\" (UID: \"365c34a6-53b4-4f52-bc89-3408a579937a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6jqc6"
Oct 30 13:45:52.079671 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:52.079634    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4hvv\" (UniqueName: \"kubernetes.io/projected/365c34a6-53b4-4f52-bc89-3408a579937a-kube-api-access-l4hvv\") pod \"maas-api-revoke-sub-d3fba056-6jqc6\" (UID: \"365c34a6-53b4-4f52-bc89-3408a579937a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6jqc6"
Oct 30 13:45:52.079671 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:52.079672    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/365c34a6-53b4-4f52-bc89-3408a579937a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-6jqc6\" (UID: \"365c34a6-53b4-4f52-bc89-3408a579937a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6jqc6"
Oct 30 13:45:52.079980 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:52.079797    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/365c34a6-53b4-4f52-bc89-3408a579937a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-6jqc6\" (UID: \"365c34a6-53b4-4f52-bc89-3408a579937a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6jqc6"
Oct 30 13:45:52.080423 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:52.080401    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/365c34a6-53b4-4f52-bc89-3408a579937a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-6jqc6\" (UID: \"365c34a6-53b4-4f52-bc89-3408a579937a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6jqc6"
Oct 30 13:45:52.082201 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:52.082174    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/365c34a6-53b4-4f52-bc89-3408a579937a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-6jqc6\" (UID: \"365c34a6-53b4-4f52-bc89-3408a579937a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6jqc6"
Oct 30 13:45:52.102863 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:52.102825    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4hvv\" (UniqueName: \"kubernetes.io/projected/365c34a6-53b4-4f52-bc89-3408a579937a-kube-api-access-l4hvv\") pod \"maas-api-revoke-sub-d3fba056-6jqc6\" (UID: \"365c34a6-53b4-4f52-bc89-3408a579937a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6jqc6"
Oct 30 13:45:52.198086 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:52.197990    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6jqc6"
Oct 30 13:45:52.389962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:52.389919    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-52t9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.98:15021/healthz/ready\": dial tcp 10.133.1.98:15021: connect: connection refused" start-of-body=
Oct 30 13:45:52.390155 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:52.389988    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q" podUID="7e9c9db5-5873-40d8-94bb-d827c7698714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.98:15021/healthz/ready\": dial tcp 10.133.1.98:15021: connect: connection refused"
Oct 30 13:45:52.534299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:52.530044    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6jqc6"]
Oct 30 13:45:52.542558 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:45:52.542526    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod365c34a6_53b4_4f52_bc89_3408a579937a.slice/crio-bdb977642b270df97124e3013ad3c6992fa5b93a4c9b03787d8cfe510f27bca5 WatchSource:0}: Error finding container bdb977642b270df97124e3013ad3c6992fa5b93a4c9b03787d8cfe510f27bca5: Status 404 returned error can't find the container with id bdb977642b270df97124e3013ad3c6992fa5b93a4c9b03787d8cfe510f27bca5
Oct 30 13:45:53.173415 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:53.173309    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6jqc6" event={"ID":"365c34a6-53b4-4f52-bc89-3408a579937a","Type":"ContainerStarted","Data":"52320b68fd5bfcb4c81647212b3376634c319d43af9553461d13742d1d48f524"}
Oct 30 13:45:53.173715 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:53.173691    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6jqc6" event={"ID":"365c34a6-53b4-4f52-bc89-3408a579937a","Type":"ContainerStarted","Data":"bdb977642b270df97124e3013ad3c6992fa5b93a4c9b03787d8cfe510f27bca5"}
Oct 30 13:45:53.221268 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:53.221172    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6jqc6" podStartSLOduration=2.221147629 podStartE2EDuration="2.221147629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:53.217897795 +0000 UTC m=+3648.265031741" watchObservedRunningTime="2026-10-30 13:45:53.221147629 +0000 UTC m=+3648.268281580"
Oct 30 13:45:53.390299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:53.390248    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-52t9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.98:15021/healthz/ready\": dial tcp 10.133.1.98:15021: connect: connection refused" start-of-body=
Oct 30 13:45:53.390488 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:53.390332    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q" podUID="7e9c9db5-5873-40d8-94bb-d827c7698714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.98:15021/healthz/ready\": dial tcp 10.133.1.98:15021: connect: connection refused"
Oct 30 13:45:54.184827 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:54.184783    4544 generic.go:356] "Generic (PLEG): container finished" podID="365c34a6-53b4-4f52-bc89-3408a579937a" containerID="52320b68fd5bfcb4c81647212b3376634c319d43af9553461d13742d1d48f524" exitCode=0
Oct 30 13:45:54.185504 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:54.184901    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6jqc6" event={"ID":"365c34a6-53b4-4f52-bc89-3408a579937a","Type":"ContainerDied","Data":"52320b68fd5bfcb4c81647212b3376634c319d43af9553461d13742d1d48f524"}
Oct 30 13:45:54.394168 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:54.394130    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-52t9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.98:15021/healthz/ready\": dial tcp 10.133.1.98:15021: connect: connection refused" start-of-body=
Oct 30 13:45:54.394618 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:54.394475    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q" podUID="7e9c9db5-5873-40d8-94bb-d827c7698714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.98:15021/healthz/ready\": dial tcp 10.133.1.98:15021: connect: connection refused"
Oct 30 13:45:55.398842 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:55.395690    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-52t9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.98:15021/healthz/ready\": dial tcp 10.133.1.98:15021: connect: connection refused" start-of-body=
Oct 30 13:45:55.398842 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:55.395758    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q" podUID="7e9c9db5-5873-40d8-94bb-d827c7698714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.98:15021/healthz/ready\": dial tcp 10.133.1.98:15021: connect: connection refused"
Oct 30 13:45:55.966374 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:55.966021    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6jqc6"
Oct 30 13:45:56.103565 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:56.103496    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/365c34a6-53b4-4f52-bc89-3408a579937a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/365c34a6-53b4-4f52-bc89-3408a579937a-maas-api-openshift-ca\") pod \"365c34a6-53b4-4f52-bc89-3408a579937a\" (UID: \"365c34a6-53b4-4f52-bc89-3408a579937a\") "
Oct 30 13:45:56.103969 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:56.103941    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/365c34a6-53b4-4f52-bc89-3408a579937a-kube-api-access-l4hvv\" (UniqueName: \"kubernetes.io/projected/365c34a6-53b4-4f52-bc89-3408a579937a-kube-api-access-l4hvv\") pod \"365c34a6-53b4-4f52-bc89-3408a579937a\" (UID: \"365c34a6-53b4-4f52-bc89-3408a579937a\") "
Oct 30 13:45:56.104162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:56.104142    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/365c34a6-53b4-4f52-bc89-3408a579937a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/365c34a6-53b4-4f52-bc89-3408a579937a-maas-api-xks-ca\") pod \"365c34a6-53b4-4f52-bc89-3408a579937a\" (UID: \"365c34a6-53b4-4f52-bc89-3408a579937a\") "
Oct 30 13:45:56.105370 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:56.105345    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/365c34a6-53b4-4f52-bc89-3408a579937a-maas-api-openshift-ca" pod "365c34a6-53b4-4f52-bc89-3408a579937a" (UID: "365c34a6-53b4-4f52-bc89-3408a579937a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:56.110661 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:56.110626    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/365c34a6-53b4-4f52-bc89-3408a579937a-maas-api-xks-ca" pod "365c34a6-53b4-4f52-bc89-3408a579937a" (UID: "365c34a6-53b4-4f52-bc89-3408a579937a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:45:56.112447 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:56.112406    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/365c34a6-53b4-4f52-bc89-3408a579937a-kube-api-access-l4hvv" pod "365c34a6-53b4-4f52-bc89-3408a579937a" (UID: "365c34a6-53b4-4f52-bc89-3408a579937a"). InnerVolumeSpecName "kube-api-access-l4hvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:56.205161 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:56.205118    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/365c34a6-53b4-4f52-bc89-3408a579937a-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:45:56.205161 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:56.205158    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4hvv\" (UniqueName: \"kubernetes.io/projected/365c34a6-53b4-4f52-bc89-3408a579937a-kube-api-access-l4hvv\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:45:56.205161 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:56.205169    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/365c34a6-53b4-4f52-bc89-3408a579937a-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:45:56.205457 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:56.205393    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6jqc6"
Oct 30 13:45:56.205497 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:56.205384    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6jqc6" event={"ID":"365c34a6-53b4-4f52-bc89-3408a579937a","Type":"ContainerDied","Data":"bdb977642b270df97124e3013ad3c6992fa5b93a4c9b03787d8cfe510f27bca5"}
Oct 30 13:45:56.205551 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:56.205533    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdb977642b270df97124e3013ad3c6992fa5b93a4c9b03787d8cfe510f27bca5"
Oct 30 13:45:56.390801 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:56.390709    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-52t9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.98:15021/healthz/ready\": dial tcp 10.133.1.98:15021: connect: connection refused" start-of-body=
Oct 30 13:45:56.390801 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:56.390784    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q" podUID="7e9c9db5-5873-40d8-94bb-d827c7698714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.98:15021/healthz/ready\": dial tcp 10.133.1.98:15021: connect: connection refused"
Oct 30 13:45:57.390734 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:57.390700    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-52t9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.98:15021/healthz/ready\": dial tcp 10.133.1.98:15021: connect: connection refused" start-of-body=
Oct 30 13:45:57.391242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:57.390768    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q" podUID="7e9c9db5-5873-40d8-94bb-d827c7698714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.98:15021/healthz/ready\": dial tcp 10.133.1.98:15021: connect: connection refused"
Oct 30 13:45:59.006609 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:45:59.006568    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:45:59.180401 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:59.180352    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:45:59.391333 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:59.391291    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-52t9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:45:59.391546 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:45:59.391369    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q" podUID="7e9c9db5-5873-40d8-94bb-d827c7698714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:46:00.432445 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:00.432394    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-52t9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:46:00.432985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:00.432455    4544 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q" podUID="7e9c9db5-5873-40d8-94bb-d827c7698714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:46:00.537496 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:00.537462    4544 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"
Oct 30 13:46:00.537791 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:00.537766    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"
Oct 30 13:46:00.538505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:00.538479    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q"
Oct 30 13:46:06.039414 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:06.039367    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a25d83d-7pm25"]
Oct 30 13:46:06.050592 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:06.050549    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a25d83d-7pm25"]
Oct 30 13:46:07.086042 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:07.086007    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="364bad41-bbc0-47e1-9f68-a66e5c73be65" path="/var/lib/kubelet/pods/364bad41-bbc0-47e1-9f68-a66e5c73be65/volumes"
Oct 30 13:46:13.594804 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:13.594763    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-t8skv"]
Oct 30 13:46:13.595719 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:13.595690    4544 cpu_manager.go:414] "containerMap: removing container" podUID="365c34a6-53b4-4f52-bc89-3408a579937a" containerName="revoke-keys"
Oct 30 13:46:13.595719 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:13.595716    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="365c34a6-53b4-4f52-bc89-3408a579937a" containerName="revoke-keys"
Oct 30 13:46:13.595902 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:13.595795    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="365c34a6-53b4-4f52-bc89-3408a579937a" containerName="revoke-keys"
Oct 30 13:46:13.598898 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:13.598880    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-t8skv"
Oct 30 13:46:13.613558 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:13.613531    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:46:13.625423 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:13.625389    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:46:13.628155 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:13.628123    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-t8skv"]
Oct 30 13:46:13.710859 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:13.710815    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jls6m\" (UniqueName: \"kubernetes.io/projected/a0344ce9-437c-4203-ac7a-dd3036521b2c-kube-api-access-jls6m\") pod \"maas-api-revoke-sub-ae594979-t8skv\" (UID: \"a0344ce9-437c-4203-ac7a-dd3036521b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-t8skv"
Oct 30 13:46:13.711082 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:13.710879    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0344ce9-437c-4203-ac7a-dd3036521b2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-t8skv\" (UID: \"a0344ce9-437c-4203-ac7a-dd3036521b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-t8skv"
Oct 30 13:46:13.711082 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:13.711040    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0344ce9-437c-4203-ac7a-dd3036521b2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-t8skv\" (UID: \"a0344ce9-437c-4203-ac7a-dd3036521b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-t8skv"
Oct 30 13:46:13.812211 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:13.812160    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0344ce9-437c-4203-ac7a-dd3036521b2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-t8skv\" (UID: \"a0344ce9-437c-4203-ac7a-dd3036521b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-t8skv"
Oct 30 13:46:13.812465 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:13.812301    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jls6m\" (UniqueName: \"kubernetes.io/projected/a0344ce9-437c-4203-ac7a-dd3036521b2c-kube-api-access-jls6m\") pod \"maas-api-revoke-sub-ae594979-t8skv\" (UID: \"a0344ce9-437c-4203-ac7a-dd3036521b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-t8skv"
Oct 30 13:46:13.812465 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:13.812362    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0344ce9-437c-4203-ac7a-dd3036521b2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-t8skv\" (UID: \"a0344ce9-437c-4203-ac7a-dd3036521b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-t8skv"
Oct 30 13:46:13.812874 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:13.812854    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0344ce9-437c-4203-ac7a-dd3036521b2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-t8skv\" (UID: \"a0344ce9-437c-4203-ac7a-dd3036521b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-t8skv"
Oct 30 13:46:13.814774 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:13.814754    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0344ce9-437c-4203-ac7a-dd3036521b2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-t8skv\" (UID: \"a0344ce9-437c-4203-ac7a-dd3036521b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-t8skv"
Oct 30 13:46:13.836310 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:13.836269    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jls6m\" (UniqueName: \"kubernetes.io/projected/a0344ce9-437c-4203-ac7a-dd3036521b2c-kube-api-access-jls6m\") pod \"maas-api-revoke-sub-ae594979-t8skv\" (UID: \"a0344ce9-437c-4203-ac7a-dd3036521b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-t8skv"
Oct 30 13:46:13.911673 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:13.911577    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-t8skv"
Oct 30 13:46:14.214793 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:14.214708    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-t8skv"]
Oct 30 13:46:14.219124 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:46:14.219091    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0344ce9_437c_4203_ac7a_dd3036521b2c.slice/crio-0a45b119aa31b07e72cbb3947987cbec37503bc0dd818b4d0c0cd2b6850eba68 WatchSource:0}: Error finding container 0a45b119aa31b07e72cbb3947987cbec37503bc0dd818b4d0c0cd2b6850eba68: Status 404 returned error can't find the container with id 0a45b119aa31b07e72cbb3947987cbec37503bc0dd818b4d0c0cd2b6850eba68
Oct 30 13:46:14.335316 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:14.335272    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-t8skv" event={"ID":"a0344ce9-437c-4203-ac7a-dd3036521b2c","Type":"ContainerStarted","Data":"0a45b119aa31b07e72cbb3947987cbec37503bc0dd818b4d0c0cd2b6850eba68"}
Oct 30 13:46:15.342061 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:15.342020    4544 generic.go:356] "Generic (PLEG): container finished" podID="a0344ce9-437c-4203-ac7a-dd3036521b2c" containerID="65eb362ec3fabb22525be56d1cb3259fb6be095fba87b9159af99cbb25198230" exitCode=0
Oct 30 13:46:15.342503 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:15.342102    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-t8skv" event={"ID":"a0344ce9-437c-4203-ac7a-dd3036521b2c","Type":"ContainerDied","Data":"65eb362ec3fabb22525be56d1cb3259fb6be095fba87b9159af99cbb25198230"}
Oct 30 13:46:15.485565 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:15.485482    4544 scope.go:122] "RemoveContainer" containerID="3f2a690746eb91a6c4ead430def72b3300465fef87a56025ff2ee8c6a77f0af4"
Oct 30 13:46:15.494746 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:15.494717    4544 scope.go:122] "RemoveContainer" containerID="4fdbb923ca53a5dae50a6c46d1958622057861b7907a7ec0305f0f0a93d12b7f"
Oct 30 13:46:15.504041 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:15.504012    4544 scope.go:122] "RemoveContainer" containerID="c280c7467610ee828ee7416e8c6536981431eadcb7606938c6c41253c020b8c5"
Oct 30 13:46:15.512507 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:15.512483    4544 scope.go:122] "RemoveContainer" containerID="7eadaf6403d31da67a65ca693ff1942e6fccea406a06711467f2f9131088e3d3"
Oct 30 13:46:15.520892 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:15.520868    4544 scope.go:122] "RemoveContainer" containerID="5c2051dc7e4f8fd8e2770fdf391d6c4f9e61c3de44aa4d165b4dfb13ace5445d"
Oct 30 13:46:16.494254 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:16.494211    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-t8skv"
Oct 30 13:46:16.639489 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:16.639390    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0344ce9-437c-4203-ac7a-dd3036521b2c-kube-api-access-jls6m\" (UniqueName: \"kubernetes.io/projected/a0344ce9-437c-4203-ac7a-dd3036521b2c-kube-api-access-jls6m\") pod \"a0344ce9-437c-4203-ac7a-dd3036521b2c\" (UID: \"a0344ce9-437c-4203-ac7a-dd3036521b2c\") "
Oct 30 13:46:16.639667 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:16.639542    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0344ce9-437c-4203-ac7a-dd3036521b2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0344ce9-437c-4203-ac7a-dd3036521b2c-maas-api-xks-ca\") pod \"a0344ce9-437c-4203-ac7a-dd3036521b2c\" (UID: \"a0344ce9-437c-4203-ac7a-dd3036521b2c\") "
Oct 30 13:46:16.639667 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:16.639575    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0344ce9-437c-4203-ac7a-dd3036521b2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0344ce9-437c-4203-ac7a-dd3036521b2c-maas-api-openshift-ca\") pod \"a0344ce9-437c-4203-ac7a-dd3036521b2c\" (UID: \"a0344ce9-437c-4203-ac7a-dd3036521b2c\") "
Oct 30 13:46:16.640009 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:16.639986    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0344ce9-437c-4203-ac7a-dd3036521b2c-maas-api-openshift-ca" pod "a0344ce9-437c-4203-ac7a-dd3036521b2c" (UID: "a0344ce9-437c-4203-ac7a-dd3036521b2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:46:16.641592 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:16.641557    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0344ce9-437c-4203-ac7a-dd3036521b2c-kube-api-access-jls6m" pod "a0344ce9-437c-4203-ac7a-dd3036521b2c" (UID: "a0344ce9-437c-4203-ac7a-dd3036521b2c"). InnerVolumeSpecName "kube-api-access-jls6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:16.641712 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:16.641649    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0344ce9-437c-4203-ac7a-dd3036521b2c-maas-api-xks-ca" pod "a0344ce9-437c-4203-ac7a-dd3036521b2c" (UID: "a0344ce9-437c-4203-ac7a-dd3036521b2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:46:16.740495 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:16.740458    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0344ce9-437c-4203-ac7a-dd3036521b2c-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:46:16.740495 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:16.740488    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0344ce9-437c-4203-ac7a-dd3036521b2c-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:46:16.740495 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:16.740500    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jls6m\" (UniqueName: \"kubernetes.io/projected/a0344ce9-437c-4203-ac7a-dd3036521b2c-kube-api-access-jls6m\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:46:17.353866 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:17.353781    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-t8skv"
Oct 30 13:46:17.353866 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:17.353811    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-t8skv" event={"ID":"a0344ce9-437c-4203-ac7a-dd3036521b2c","Type":"ContainerDied","Data":"0a45b119aa31b07e72cbb3947987cbec37503bc0dd818b4d0c0cd2b6850eba68"}
Oct 30 13:46:17.353866 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:17.353865    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a45b119aa31b07e72cbb3947987cbec37503bc0dd818b4d0c0cd2b6850eba68"
Oct 30 13:46:29.135245 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:46:29.134701    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:46:35.201361 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:35.201318    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bp7f9"]
Oct 30 13:46:35.202034 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:35.202012    4544 cpu_manager.go:414] "containerMap: removing container" podUID="a0344ce9-437c-4203-ac7a-dd3036521b2c" containerName="revoke-keys"
Oct 30 13:46:35.202083 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:35.202038    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0344ce9-437c-4203-ac7a-dd3036521b2c" containerName="revoke-keys"
Oct 30 13:46:35.202168 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:35.202155    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0344ce9-437c-4203-ac7a-dd3036521b2c" containerName="revoke-keys"
Oct 30 13:46:35.207071 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:35.207044    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bp7f9"
Oct 30 13:46:35.226037 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:35.225994    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bp7f9"]
Oct 30 13:46:35.235552 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:35.235516    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:46:35.235806 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:35.235598    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:46:35.237983 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:35.237955    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e287f86b-364b-4a4f-bcd1-4a99c695d1da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-bp7f9\" (UID: \"e287f86b-364b-4a4f-bcd1-4a99c695d1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bp7f9"
Oct 30 13:46:35.238145 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:35.237992    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e287f86b-364b-4a4f-bcd1-4a99c695d1da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-bp7f9\" (UID: \"e287f86b-364b-4a4f-bcd1-4a99c695d1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bp7f9"
Oct 30 13:46:35.238145 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:35.238020    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8djjx\" (UniqueName: \"kubernetes.io/projected/e287f86b-364b-4a4f-bcd1-4a99c695d1da-kube-api-access-8djjx\") pod \"maas-api-revoke-sub-a4aef7d7-bp7f9\" (UID: \"e287f86b-364b-4a4f-bcd1-4a99c695d1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bp7f9"
Oct 30 13:46:35.338919 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:35.338882    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8djjx\" (UniqueName: \"kubernetes.io/projected/e287f86b-364b-4a4f-bcd1-4a99c695d1da-kube-api-access-8djjx\") pod \"maas-api-revoke-sub-a4aef7d7-bp7f9\" (UID: \"e287f86b-364b-4a4f-bcd1-4a99c695d1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bp7f9"
Oct 30 13:46:35.339102 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:35.339012    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e287f86b-364b-4a4f-bcd1-4a99c695d1da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-bp7f9\" (UID: \"e287f86b-364b-4a4f-bcd1-4a99c695d1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bp7f9"
Oct 30 13:46:35.339102 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:35.339050    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e287f86b-364b-4a4f-bcd1-4a99c695d1da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-bp7f9\" (UID: \"e287f86b-364b-4a4f-bcd1-4a99c695d1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bp7f9"
Oct 30 13:46:35.339791 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:35.339771    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e287f86b-364b-4a4f-bcd1-4a99c695d1da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-bp7f9\" (UID: \"e287f86b-364b-4a4f-bcd1-4a99c695d1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bp7f9"
Oct 30 13:46:35.341346 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:35.341327    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e287f86b-364b-4a4f-bcd1-4a99c695d1da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-bp7f9\" (UID: \"e287f86b-364b-4a4f-bcd1-4a99c695d1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bp7f9"
Oct 30 13:46:35.362793 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:35.362753    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8djjx\" (UniqueName: \"kubernetes.io/projected/e287f86b-364b-4a4f-bcd1-4a99c695d1da-kube-api-access-8djjx\") pod \"maas-api-revoke-sub-a4aef7d7-bp7f9\" (UID: \"e287f86b-364b-4a4f-bcd1-4a99c695d1da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bp7f9"
Oct 30 13:46:35.520302 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:35.520186    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bp7f9"
Oct 30 13:46:35.820893 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:35.820806    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bp7f9"]
Oct 30 13:46:35.824946 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:46:35.824907    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode287f86b_364b_4a4f_bcd1_4a99c695d1da.slice/crio-d28e8a55cf7b95fc17a580eedc838cdb20202371509f748f8910ef36e5e6eca2 WatchSource:0}: Error finding container d28e8a55cf7b95fc17a580eedc838cdb20202371509f748f8910ef36e5e6eca2: Status 404 returned error can't find the container with id d28e8a55cf7b95fc17a580eedc838cdb20202371509f748f8910ef36e5e6eca2
Oct 30 13:46:36.488157 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:36.488058    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bp7f9" event={"ID":"e287f86b-364b-4a4f-bcd1-4a99c695d1da","Type":"ContainerStarted","Data":"60df2f8573c538cbf6f695db502a8415207f53c4ef96bc94086b083fe7b25e10"}
Oct 30 13:46:36.488157 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:36.488125    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bp7f9" event={"ID":"e287f86b-364b-4a4f-bcd1-4a99c695d1da","Type":"ContainerStarted","Data":"d28e8a55cf7b95fc17a580eedc838cdb20202371509f748f8910ef36e5e6eca2"}
Oct 30 13:46:36.530425 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:36.530366    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bp7f9" podStartSLOduration=1.530349433 podStartE2EDuration="1.530349433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:46:36.528112377 +0000 UTC m=+3691.575246332" watchObservedRunningTime="2026-10-30 13:46:36.530349433 +0000 UTC m=+3691.577483385"
Oct 30 13:46:37.495200 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:37.495153    4544 generic.go:356] "Generic (PLEG): container finished" podID="e287f86b-364b-4a4f-bcd1-4a99c695d1da" containerID="60df2f8573c538cbf6f695db502a8415207f53c4ef96bc94086b083fe7b25e10" exitCode=0
Oct 30 13:46:37.495619 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:37.495214    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bp7f9" event={"ID":"e287f86b-364b-4a4f-bcd1-4a99c695d1da","Type":"ContainerDied","Data":"60df2f8573c538cbf6f695db502a8415207f53c4ef96bc94086b083fe7b25e10"}
Oct 30 13:46:38.646638 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:38.646599    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bp7f9"
Oct 30 13:46:38.669462 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:38.669422    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e287f86b-364b-4a4f-bcd1-4a99c695d1da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e287f86b-364b-4a4f-bcd1-4a99c695d1da-maas-api-xks-ca\") pod \"e287f86b-364b-4a4f-bcd1-4a99c695d1da\" (UID: \"e287f86b-364b-4a4f-bcd1-4a99c695d1da\") "
Oct 30 13:46:38.669462 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:38.669464    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e287f86b-364b-4a4f-bcd1-4a99c695d1da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e287f86b-364b-4a4f-bcd1-4a99c695d1da-maas-api-openshift-ca\") pod \"e287f86b-364b-4a4f-bcd1-4a99c695d1da\" (UID: \"e287f86b-364b-4a4f-bcd1-4a99c695d1da\") "
Oct 30 13:46:38.669732 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:38.669532    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e287f86b-364b-4a4f-bcd1-4a99c695d1da-kube-api-access-8djjx\" (UniqueName: \"kubernetes.io/projected/e287f86b-364b-4a4f-bcd1-4a99c695d1da-kube-api-access-8djjx\") pod \"e287f86b-364b-4a4f-bcd1-4a99c695d1da\" (UID: \"e287f86b-364b-4a4f-bcd1-4a99c695d1da\") "
Oct 30 13:46:38.669899 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:38.669868    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e287f86b-364b-4a4f-bcd1-4a99c695d1da-maas-api-openshift-ca" pod "e287f86b-364b-4a4f-bcd1-4a99c695d1da" (UID: "e287f86b-364b-4a4f-bcd1-4a99c695d1da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:46:38.671699 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:38.671669    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e287f86b-364b-4a4f-bcd1-4a99c695d1da-maas-api-xks-ca" pod "e287f86b-364b-4a4f-bcd1-4a99c695d1da" (UID: "e287f86b-364b-4a4f-bcd1-4a99c695d1da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:46:38.671813 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:38.671785    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e287f86b-364b-4a4f-bcd1-4a99c695d1da-kube-api-access-8djjx" pod "e287f86b-364b-4a4f-bcd1-4a99c695d1da" (UID: "e287f86b-364b-4a4f-bcd1-4a99c695d1da"). InnerVolumeSpecName "kube-api-access-8djjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:38.770726 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:38.770626    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e287f86b-364b-4a4f-bcd1-4a99c695d1da-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:46:38.770726 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:38.770663    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e287f86b-364b-4a4f-bcd1-4a99c695d1da-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:46:38.770726 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:38.770673    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8djjx\" (UniqueName: \"kubernetes.io/projected/e287f86b-364b-4a4f-bcd1-4a99c695d1da-kube-api-access-8djjx\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:46:39.506569 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:39.506490    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bp7f9"
Oct 30 13:46:39.506569 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:39.506501    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bp7f9" event={"ID":"e287f86b-364b-4a4f-bcd1-4a99c695d1da","Type":"ContainerDied","Data":"d28e8a55cf7b95fc17a580eedc838cdb20202371509f748f8910ef36e5e6eca2"}
Oct 30 13:46:39.506569 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:39.506547    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d28e8a55cf7b95fc17a580eedc838cdb20202371509f748f8910ef36e5e6eca2"
Oct 30 13:46:55.508641 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:55.508604    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xc4b8"]
Oct 30 13:46:55.509162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:55.509032    4544 cpu_manager.go:414] "containerMap: removing container" podUID="e287f86b-364b-4a4f-bcd1-4a99c695d1da" containerName="revoke-keys"
Oct 30 13:46:55.509162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:55.509044    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e287f86b-364b-4a4f-bcd1-4a99c695d1da" containerName="revoke-keys"
Oct 30 13:46:55.509162 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:55.509132    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="e287f86b-364b-4a4f-bcd1-4a99c695d1da" containerName="revoke-keys"
Oct 30 13:46:55.512507 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:55.512482    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xc4b8"
Oct 30 13:46:55.532703 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:55.532667    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xc4b8"]
Oct 30 13:46:55.541450 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:55.541407    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:46:55.541450 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:55.541436    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:46:55.576696 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:55.576663    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w5cl\" (UniqueName: \"kubernetes.io/projected/f42cebea-cc55-4430-b6ff-c1abf0b0c47c-kube-api-access-2w5cl\") pod \"maas-api-revoke-sub-79805ca9-xc4b8\" (UID: \"f42cebea-cc55-4430-b6ff-c1abf0b0c47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xc4b8"
Oct 30 13:46:55.576846 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:55.576723    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42cebea-cc55-4430-b6ff-c1abf0b0c47c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-xc4b8\" (UID: \"f42cebea-cc55-4430-b6ff-c1abf0b0c47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xc4b8"
Oct 30 13:46:55.576846 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:55.576802    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42cebea-cc55-4430-b6ff-c1abf0b0c47c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-xc4b8\" (UID: \"f42cebea-cc55-4430-b6ff-c1abf0b0c47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xc4b8"
Oct 30 13:46:55.677672 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:55.677633    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42cebea-cc55-4430-b6ff-c1abf0b0c47c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-xc4b8\" (UID: \"f42cebea-cc55-4430-b6ff-c1abf0b0c47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xc4b8"
Oct 30 13:46:55.677868 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:55.677688    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42cebea-cc55-4430-b6ff-c1abf0b0c47c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-xc4b8\" (UID: \"f42cebea-cc55-4430-b6ff-c1abf0b0c47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xc4b8"
Oct 30 13:46:55.677868 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:55.677764    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w5cl\" (UniqueName: \"kubernetes.io/projected/f42cebea-cc55-4430-b6ff-c1abf0b0c47c-kube-api-access-2w5cl\") pod \"maas-api-revoke-sub-79805ca9-xc4b8\" (UID: \"f42cebea-cc55-4430-b6ff-c1abf0b0c47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xc4b8"
Oct 30 13:46:55.678489 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:55.678467    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42cebea-cc55-4430-b6ff-c1abf0b0c47c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-xc4b8\" (UID: \"f42cebea-cc55-4430-b6ff-c1abf0b0c47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xc4b8"
Oct 30 13:46:55.680044 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:55.680026    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42cebea-cc55-4430-b6ff-c1abf0b0c47c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-xc4b8\" (UID: \"f42cebea-cc55-4430-b6ff-c1abf0b0c47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xc4b8"
Oct 30 13:46:55.700812 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:55.700782    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w5cl\" (UniqueName: \"kubernetes.io/projected/f42cebea-cc55-4430-b6ff-c1abf0b0c47c-kube-api-access-2w5cl\") pod \"maas-api-revoke-sub-79805ca9-xc4b8\" (UID: \"f42cebea-cc55-4430-b6ff-c1abf0b0c47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xc4b8"
Oct 30 13:46:55.825132 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:55.825100    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xc4b8"
Oct 30 13:46:56.129258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:56.129165    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xc4b8"]
Oct 30 13:46:56.132919 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:46:56.132881    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf42cebea_cc55_4430_b6ff_c1abf0b0c47c.slice/crio-b7952a5393f1a975362cd185ecd372d0d7de334e76006337d26d80f0d42f2fba WatchSource:0}: Error finding container b7952a5393f1a975362cd185ecd372d0d7de334e76006337d26d80f0d42f2fba: Status 404 returned error can't find the container with id b7952a5393f1a975362cd185ecd372d0d7de334e76006337d26d80f0d42f2fba
Oct 30 13:46:56.610680 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:56.610635    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xc4b8" event={"ID":"f42cebea-cc55-4430-b6ff-c1abf0b0c47c","Type":"ContainerStarted","Data":"904c33f8901ba551fc1114948cdbaaa610ad009c0e22acdcf50aa9743735202f"}
Oct 30 13:46:56.611171 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:56.610703    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xc4b8" event={"ID":"f42cebea-cc55-4430-b6ff-c1abf0b0c47c","Type":"ContainerStarted","Data":"b7952a5393f1a975362cd185ecd372d0d7de334e76006337d26d80f0d42f2fba"}
Oct 30 13:46:56.652970 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:56.652907    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xc4b8" podStartSLOduration=1.652891582 podStartE2EDuration="1.652891582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:46:56.650440531 +0000 UTC m=+3711.697574484" watchObservedRunningTime="2026-10-30 13:46:56.652891582 +0000 UTC m=+3711.700025535"
Oct 30 13:46:57.617704 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:57.617665    4544 generic.go:356] "Generic (PLEG): container finished" podID="f42cebea-cc55-4430-b6ff-c1abf0b0c47c" containerID="904c33f8901ba551fc1114948cdbaaa610ad009c0e22acdcf50aa9743735202f" exitCode=0
Oct 30 13:46:57.618116 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:57.617750    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xc4b8" event={"ID":"f42cebea-cc55-4430-b6ff-c1abf0b0c47c","Type":"ContainerDied","Data":"904c33f8901ba551fc1114948cdbaaa610ad009c0e22acdcf50aa9743735202f"}
Oct 30 13:46:58.765090 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:58.765065    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xc4b8"
Oct 30 13:46:58.801352 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:58.801318    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f42cebea-cc55-4430-b6ff-c1abf0b0c47c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42cebea-cc55-4430-b6ff-c1abf0b0c47c-maas-api-xks-ca\") pod \"f42cebea-cc55-4430-b6ff-c1abf0b0c47c\" (UID: \"f42cebea-cc55-4430-b6ff-c1abf0b0c47c\") "
Oct 30 13:46:58.801547 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:58.801364    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f42cebea-cc55-4430-b6ff-c1abf0b0c47c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42cebea-cc55-4430-b6ff-c1abf0b0c47c-maas-api-openshift-ca\") pod \"f42cebea-cc55-4430-b6ff-c1abf0b0c47c\" (UID: \"f42cebea-cc55-4430-b6ff-c1abf0b0c47c\") "
Oct 30 13:46:58.801787 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:58.801755    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f42cebea-cc55-4430-b6ff-c1abf0b0c47c-maas-api-openshift-ca" pod "f42cebea-cc55-4430-b6ff-c1abf0b0c47c" (UID: "f42cebea-cc55-4430-b6ff-c1abf0b0c47c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:46:58.803417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:58.803394    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f42cebea-cc55-4430-b6ff-c1abf0b0c47c-maas-api-xks-ca" pod "f42cebea-cc55-4430-b6ff-c1abf0b0c47c" (UID: "f42cebea-cc55-4430-b6ff-c1abf0b0c47c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:46:58.902125 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:58.902022    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f42cebea-cc55-4430-b6ff-c1abf0b0c47c-kube-api-access-2w5cl\" (UniqueName: \"kubernetes.io/projected/f42cebea-cc55-4430-b6ff-c1abf0b0c47c-kube-api-access-2w5cl\") pod \"f42cebea-cc55-4430-b6ff-c1abf0b0c47c\" (UID: \"f42cebea-cc55-4430-b6ff-c1abf0b0c47c\") "
Oct 30 13:46:58.902412 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:58.902393    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42cebea-cc55-4430-b6ff-c1abf0b0c47c-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:46:58.902490 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:58.902417    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42cebea-cc55-4430-b6ff-c1abf0b0c47c-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:46:58.904138 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:58.904113    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f42cebea-cc55-4430-b6ff-c1abf0b0c47c-kube-api-access-2w5cl" pod "f42cebea-cc55-4430-b6ff-c1abf0b0c47c" (UID: "f42cebea-cc55-4430-b6ff-c1abf0b0c47c"). InnerVolumeSpecName "kube-api-access-2w5cl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:59.003016 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:59.002986    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w5cl\" (UniqueName: \"kubernetes.io/projected/f42cebea-cc55-4430-b6ff-c1abf0b0c47c-kube-api-access-2w5cl\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:46:59.009057 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:46:59.009023    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:46:59.271836 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:59.271750    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:46:59.630166 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:59.630129    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xc4b8"
Oct 30 13:46:59.630166 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:59.630154    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xc4b8" event={"ID":"f42cebea-cc55-4430-b6ff-c1abf0b0c47c","Type":"ContainerDied","Data":"b7952a5393f1a975362cd185ecd372d0d7de334e76006337d26d80f0d42f2fba"}
Oct 30 13:46:59.630413 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:46:59.630196    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7952a5393f1a975362cd185ecd372d0d7de334e76006337d26d80f0d42f2fba"
Oct 30 13:47:13.459941 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:13.459889    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pdgdk"]
Oct 30 13:47:13.460489 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:13.460399    4544 cpu_manager.go:414] "containerMap: removing container" podUID="f42cebea-cc55-4430-b6ff-c1abf0b0c47c" containerName="revoke-keys"
Oct 30 13:47:13.460489 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:13.460415    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f42cebea-cc55-4430-b6ff-c1abf0b0c47c" containerName="revoke-keys"
Oct 30 13:47:13.460583 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:13.460499    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="f42cebea-cc55-4430-b6ff-c1abf0b0c47c" containerName="revoke-keys"
Oct 30 13:47:13.463695 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:13.463670    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pdgdk"
Oct 30 13:47:13.481814 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:13.481783    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:47:13.490620 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:13.490580    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:47:13.493446 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:13.493407    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pdgdk"]
Oct 30 13:47:13.602349 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:13.602306    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxv96\" (UniqueName: \"kubernetes.io/projected/b5f2b7a6-bdb6-4096-92c9-ed10b209c297-kube-api-access-rxv96\") pod \"maas-api-revoke-sub-0b7e8edb-pdgdk\" (UID: \"b5f2b7a6-bdb6-4096-92c9-ed10b209c297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pdgdk"
Oct 30 13:47:13.602349 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:13.602343    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5f2b7a6-bdb6-4096-92c9-ed10b209c297-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-pdgdk\" (UID: \"b5f2b7a6-bdb6-4096-92c9-ed10b209c297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pdgdk"
Oct 30 13:47:13.602600 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:13.602487    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5f2b7a6-bdb6-4096-92c9-ed10b209c297-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-pdgdk\" (UID: \"b5f2b7a6-bdb6-4096-92c9-ed10b209c297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pdgdk"
Oct 30 13:47:13.703063 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:13.703016    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5f2b7a6-bdb6-4096-92c9-ed10b209c297-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-pdgdk\" (UID: \"b5f2b7a6-bdb6-4096-92c9-ed10b209c297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pdgdk"
Oct 30 13:47:13.703307 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:13.703088    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxv96\" (UniqueName: \"kubernetes.io/projected/b5f2b7a6-bdb6-4096-92c9-ed10b209c297-kube-api-access-rxv96\") pod \"maas-api-revoke-sub-0b7e8edb-pdgdk\" (UID: \"b5f2b7a6-bdb6-4096-92c9-ed10b209c297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pdgdk"
Oct 30 13:47:13.703307 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:13.703106    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5f2b7a6-bdb6-4096-92c9-ed10b209c297-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-pdgdk\" (UID: \"b5f2b7a6-bdb6-4096-92c9-ed10b209c297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pdgdk"
Oct 30 13:47:13.703820 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:13.703796    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5f2b7a6-bdb6-4096-92c9-ed10b209c297-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-pdgdk\" (UID: \"b5f2b7a6-bdb6-4096-92c9-ed10b209c297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pdgdk"
Oct 30 13:47:13.705460 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:13.705436    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5f2b7a6-bdb6-4096-92c9-ed10b209c297-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-pdgdk\" (UID: \"b5f2b7a6-bdb6-4096-92c9-ed10b209c297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pdgdk"
Oct 30 13:47:13.727371 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:13.727299    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxv96\" (UniqueName: \"kubernetes.io/projected/b5f2b7a6-bdb6-4096-92c9-ed10b209c297-kube-api-access-rxv96\") pod \"maas-api-revoke-sub-0b7e8edb-pdgdk\" (UID: \"b5f2b7a6-bdb6-4096-92c9-ed10b209c297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pdgdk"
Oct 30 13:47:13.777345 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:13.777301    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pdgdk"
Oct 30 13:47:14.077199 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:14.077164    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pdgdk"]
Oct 30 13:47:14.081926 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:47:14.081891    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f2b7a6_bdb6_4096_92c9_ed10b209c297.slice/crio-aeeb1f855a206f759819d2c7366169c05a999b43f63140effce4205b2a91299a WatchSource:0}: Error finding container aeeb1f855a206f759819d2c7366169c05a999b43f63140effce4205b2a91299a: Status 404 returned error can't find the container with id aeeb1f855a206f759819d2c7366169c05a999b43f63140effce4205b2a91299a
Oct 30 13:47:14.740782 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:14.740683    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pdgdk" event={"ID":"b5f2b7a6-bdb6-4096-92c9-ed10b209c297","Type":"ContainerStarted","Data":"29b963f73c49aca65547d01f31a028404a3116e500cc21d99097aa1c583d54f6"}
Oct 30 13:47:14.740782 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:14.740755    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pdgdk" event={"ID":"b5f2b7a6-bdb6-4096-92c9-ed10b209c297","Type":"ContainerStarted","Data":"aeeb1f855a206f759819d2c7366169c05a999b43f63140effce4205b2a91299a"}
Oct 30 13:47:14.782455 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:14.782399    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pdgdk" podStartSLOduration=1.78238291 podStartE2EDuration="1.78238291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:47:14.779960111 +0000 UTC m=+3729.827094065" watchObservedRunningTime="2026-10-30 13:47:14.78238291 +0000 UTC m=+3729.829516862"
Oct 30 13:47:15.565318 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:15.565283    4544 scope.go:122] "RemoveContainer" containerID="544c5378f2f0780f40f13b42d38d6df4de0b2653e5e61e5b221ff8eedfe04c4b"
Oct 30 13:47:15.574790 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:15.574760    4544 scope.go:122] "RemoveContainer" containerID="26091745f2ca1da4f54bcb4d5edac6b4238dd72665a941e14a425fa662133a11"
Oct 30 13:47:15.583814 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:15.583789    4544 scope.go:122] "RemoveContainer" containerID="969d03d92ff38fb0e817d29c5a6985961f59fd377dcdf3e4565568404f3dc2a8"
Oct 30 13:47:15.592512 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:15.592490    4544 scope.go:122] "RemoveContainer" containerID="e9a683ec2b08978bdb04768f18f79843217b622048936d0a2cd4f47bad42b7ae"
Oct 30 13:47:15.600901 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:15.600879    4544 scope.go:122] "RemoveContainer" containerID="119be7047d43cc5c4e3c5561db2e655faf273f826cf8b0afed664b22449e163d"
Oct 30 13:47:15.747264 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:15.747215    4544 generic.go:356] "Generic (PLEG): container finished" podID="b5f2b7a6-bdb6-4096-92c9-ed10b209c297" containerID="29b963f73c49aca65547d01f31a028404a3116e500cc21d99097aa1c583d54f6" exitCode=0
Oct 30 13:47:15.747669 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:15.747320    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pdgdk" event={"ID":"b5f2b7a6-bdb6-4096-92c9-ed10b209c297","Type":"ContainerDied","Data":"29b963f73c49aca65547d01f31a028404a3116e500cc21d99097aa1c583d54f6"}
Oct 30 13:47:16.892194 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:16.892160    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pdgdk"
Oct 30 13:47:17.037782 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:17.037662    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5f2b7a6-bdb6-4096-92c9-ed10b209c297-kube-api-access-rxv96\" (UniqueName: \"kubernetes.io/projected/b5f2b7a6-bdb6-4096-92c9-ed10b209c297-kube-api-access-rxv96\") pod \"b5f2b7a6-bdb6-4096-92c9-ed10b209c297\" (UID: \"b5f2b7a6-bdb6-4096-92c9-ed10b209c297\") "
Oct 30 13:47:17.038002 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:17.037818    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5f2b7a6-bdb6-4096-92c9-ed10b209c297-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5f2b7a6-bdb6-4096-92c9-ed10b209c297-maas-api-xks-ca\") pod \"b5f2b7a6-bdb6-4096-92c9-ed10b209c297\" (UID: \"b5f2b7a6-bdb6-4096-92c9-ed10b209c297\") "
Oct 30 13:47:17.038002 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:17.037910    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5f2b7a6-bdb6-4096-92c9-ed10b209c297-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5f2b7a6-bdb6-4096-92c9-ed10b209c297-maas-api-openshift-ca\") pod \"b5f2b7a6-bdb6-4096-92c9-ed10b209c297\" (UID: \"b5f2b7a6-bdb6-4096-92c9-ed10b209c297\") "
Oct 30 13:47:17.038354 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:17.038329    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5f2b7a6-bdb6-4096-92c9-ed10b209c297-maas-api-openshift-ca" pod "b5f2b7a6-bdb6-4096-92c9-ed10b209c297" (UID: "b5f2b7a6-bdb6-4096-92c9-ed10b209c297"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:47:17.039892 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:17.039862    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5f2b7a6-bdb6-4096-92c9-ed10b209c297-kube-api-access-rxv96" pod "b5f2b7a6-bdb6-4096-92c9-ed10b209c297" (UID: "b5f2b7a6-bdb6-4096-92c9-ed10b209c297"). InnerVolumeSpecName "kube-api-access-rxv96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:47:17.040002 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:17.039947    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5f2b7a6-bdb6-4096-92c9-ed10b209c297-maas-api-xks-ca" pod "b5f2b7a6-bdb6-4096-92c9-ed10b209c297" (UID: "b5f2b7a6-bdb6-4096-92c9-ed10b209c297"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:47:17.139486 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:17.139442    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5f2b7a6-bdb6-4096-92c9-ed10b209c297-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:47:17.139486 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:17.139481    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxv96\" (UniqueName: \"kubernetes.io/projected/b5f2b7a6-bdb6-4096-92c9-ed10b209c297-kube-api-access-rxv96\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:47:17.139486 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:17.139492    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5f2b7a6-bdb6-4096-92c9-ed10b209c297-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:47:17.759065 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:17.759028    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pdgdk"
Oct 30 13:47:17.759065 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:17.759052    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pdgdk" event={"ID":"b5f2b7a6-bdb6-4096-92c9-ed10b209c297","Type":"ContainerDied","Data":"aeeb1f855a206f759819d2c7366169c05a999b43f63140effce4205b2a91299a"}
Oct 30 13:47:17.759336 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:17.759102    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aeeb1f855a206f759819d2c7366169c05a999b43f63140effce4205b2a91299a"
Oct 30 13:47:24.215463 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:24.215423    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd7dc"]
Oct 30 13:47:24.217030 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:24.216999    4544 cpu_manager.go:414] "containerMap: removing container" podUID="b5f2b7a6-bdb6-4096-92c9-ed10b209c297" containerName="revoke-keys"
Oct 30 13:47:24.217242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:24.217209    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5f2b7a6-bdb6-4096-92c9-ed10b209c297" containerName="revoke-keys"
Oct 30 13:47:24.217564 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:24.217547    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5f2b7a6-bdb6-4096-92c9-ed10b209c297" containerName="revoke-keys"
Oct 30 13:47:24.221987 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:24.221960    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd7dc"
Oct 30 13:47:24.243426 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:24.243394    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:47:24.254913 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:24.254877    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:47:24.266498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:24.266461    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd7dc"]
Oct 30 13:47:24.311193 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:24.311160    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gngvn\" (UniqueName: \"kubernetes.io/projected/2f3d011b-3daa-4050-a42c-7d014717857d-kube-api-access-gngvn\") pod \"maas-api-revoke-sub-3d4156f1-fd7dc\" (UID: \"2f3d011b-3daa-4050-a42c-7d014717857d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd7dc"
Oct 30 13:47:24.311193 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:24.311207    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f3d011b-3daa-4050-a42c-7d014717857d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fd7dc\" (UID: \"2f3d011b-3daa-4050-a42c-7d014717857d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd7dc"
Oct 30 13:47:24.311436 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:24.311330    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f3d011b-3daa-4050-a42c-7d014717857d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fd7dc\" (UID: \"2f3d011b-3daa-4050-a42c-7d014717857d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd7dc"
Oct 30 13:47:24.412037 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:24.411997    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gngvn\" (UniqueName: \"kubernetes.io/projected/2f3d011b-3daa-4050-a42c-7d014717857d-kube-api-access-gngvn\") pod \"maas-api-revoke-sub-3d4156f1-fd7dc\" (UID: \"2f3d011b-3daa-4050-a42c-7d014717857d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd7dc"
Oct 30 13:47:24.412272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:24.412046    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f3d011b-3daa-4050-a42c-7d014717857d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fd7dc\" (UID: \"2f3d011b-3daa-4050-a42c-7d014717857d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd7dc"
Oct 30 13:47:24.412272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:24.412094    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f3d011b-3daa-4050-a42c-7d014717857d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fd7dc\" (UID: \"2f3d011b-3daa-4050-a42c-7d014717857d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd7dc"
Oct 30 13:47:24.412854 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:24.412825    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f3d011b-3daa-4050-a42c-7d014717857d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fd7dc\" (UID: \"2f3d011b-3daa-4050-a42c-7d014717857d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd7dc"
Oct 30 13:47:24.414633 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:24.414610    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f3d011b-3daa-4050-a42c-7d014717857d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fd7dc\" (UID: \"2f3d011b-3daa-4050-a42c-7d014717857d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd7dc"
Oct 30 13:47:24.435387 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:24.435356    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gngvn\" (UniqueName: \"kubernetes.io/projected/2f3d011b-3daa-4050-a42c-7d014717857d-kube-api-access-gngvn\") pod \"maas-api-revoke-sub-3d4156f1-fd7dc\" (UID: \"2f3d011b-3daa-4050-a42c-7d014717857d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd7dc"
Oct 30 13:47:24.536022 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:24.535927    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd7dc"
Oct 30 13:47:24.837169 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:24.837098    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd7dc"]
Oct 30 13:47:24.840920 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:47:24.840893    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f3d011b_3daa_4050_a42c_7d014717857d.slice/crio-1f19b8be3f2a9b1012eb8e8b344e57a10e1440871e4991b8d27177c4419e6839 WatchSource:0}: Error finding container 1f19b8be3f2a9b1012eb8e8b344e57a10e1440871e4991b8d27177c4419e6839: Status 404 returned error can't find the container with id 1f19b8be3f2a9b1012eb8e8b344e57a10e1440871e4991b8d27177c4419e6839
Oct 30 13:47:25.803607 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:25.803572    4544 generic.go:356] "Generic (PLEG): container finished" podID="2f3d011b-3daa-4050-a42c-7d014717857d" containerID="10e5d3c768fe0360acf0c36235b1259c8b55f99ddfed74c55262d15f29e6f7a3" exitCode=0
Oct 30 13:47:25.804101 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:25.803665    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd7dc" event={"ID":"2f3d011b-3daa-4050-a42c-7d014717857d","Type":"ContainerDied","Data":"10e5d3c768fe0360acf0c36235b1259c8b55f99ddfed74c55262d15f29e6f7a3"}
Oct 30 13:47:25.804101 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:25.803719    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd7dc" event={"ID":"2f3d011b-3daa-4050-a42c-7d014717857d","Type":"ContainerStarted","Data":"1f19b8be3f2a9b1012eb8e8b344e57a10e1440871e4991b8d27177c4419e6839"}
Oct 30 13:47:26.952771 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:26.952739    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd7dc"
Oct 30 13:47:27.141586 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:27.141546    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f3d011b-3daa-4050-a42c-7d014717857d-kube-api-access-gngvn\" (UniqueName: \"kubernetes.io/projected/2f3d011b-3daa-4050-a42c-7d014717857d-kube-api-access-gngvn\") pod \"2f3d011b-3daa-4050-a42c-7d014717857d\" (UID: \"2f3d011b-3daa-4050-a42c-7d014717857d\") "
Oct 30 13:47:27.141586 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:27.141589    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f3d011b-3daa-4050-a42c-7d014717857d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f3d011b-3daa-4050-a42c-7d014717857d-maas-api-openshift-ca\") pod \"2f3d011b-3daa-4050-a42c-7d014717857d\" (UID: \"2f3d011b-3daa-4050-a42c-7d014717857d\") "
Oct 30 13:47:27.141901 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:27.141649    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f3d011b-3daa-4050-a42c-7d014717857d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f3d011b-3daa-4050-a42c-7d014717857d-maas-api-xks-ca\") pod \"2f3d011b-3daa-4050-a42c-7d014717857d\" (UID: \"2f3d011b-3daa-4050-a42c-7d014717857d\") "
Oct 30 13:47:27.142051 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:27.142025    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f3d011b-3daa-4050-a42c-7d014717857d-maas-api-openshift-ca" pod "2f3d011b-3daa-4050-a42c-7d014717857d" (UID: "2f3d011b-3daa-4050-a42c-7d014717857d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:47:27.143793 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:27.143764    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f3d011b-3daa-4050-a42c-7d014717857d-maas-api-xks-ca" pod "2f3d011b-3daa-4050-a42c-7d014717857d" (UID: "2f3d011b-3daa-4050-a42c-7d014717857d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:47:27.143860 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:27.143824    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f3d011b-3daa-4050-a42c-7d014717857d-kube-api-access-gngvn" pod "2f3d011b-3daa-4050-a42c-7d014717857d" (UID: "2f3d011b-3daa-4050-a42c-7d014717857d"). InnerVolumeSpecName "kube-api-access-gngvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:47:27.242735 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:27.242688    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f3d011b-3daa-4050-a42c-7d014717857d-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:47:27.242735 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:27.242724    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gngvn\" (UniqueName: \"kubernetes.io/projected/2f3d011b-3daa-4050-a42c-7d014717857d-kube-api-access-gngvn\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:47:27.242735 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:27.242735    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f3d011b-3daa-4050-a42c-7d014717857d-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:47:27.817306 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:27.817268    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd7dc" event={"ID":"2f3d011b-3daa-4050-a42c-7d014717857d","Type":"ContainerDied","Data":"1f19b8be3f2a9b1012eb8e8b344e57a10e1440871e4991b8d27177c4419e6839"}
Oct 30 13:47:27.817306 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:27.817303    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd7dc"
Oct 30 13:47:27.817601 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:27.817320    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f19b8be3f2a9b1012eb8e8b344e57a10e1440871e4991b8d27177c4419e6839"
Oct 30 13:47:29.006709 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:47:29.006675    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:47:45.635272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:45.635230    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd7dc"]
Oct 30 13:47:45.647193 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:45.647154    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fd7dc"]
Oct 30 13:47:45.682823 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:45.682791    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rqchz"]
Oct 30 13:47:45.683214 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:45.683201    4544 cpu_manager.go:414] "containerMap: removing container" podUID="2f3d011b-3daa-4050-a42c-7d014717857d" containerName="revoke-keys"
Oct 30 13:47:45.683214 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:45.683215    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f3d011b-3daa-4050-a42c-7d014717857d" containerName="revoke-keys"
Oct 30 13:47:45.683338 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:45.683314    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f3d011b-3daa-4050-a42c-7d014717857d" containerName="revoke-keys"
Oct 30 13:47:45.686798 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:45.686771    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rqchz"
Oct 30 13:47:45.704107 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:45.704063    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:47:45.704364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:45.704327    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:47:45.707404 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:45.707368    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rqchz"]
Oct 30 13:47:45.792047 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:45.792011    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f25527-b950-4ba5-b02e-c991b116e4db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rqchz\" (UID: \"32f25527-b950-4ba5-b02e-c991b116e4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rqchz"
Oct 30 13:47:45.792291 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:45.792069    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66xfg\" (UniqueName: \"kubernetes.io/projected/32f25527-b950-4ba5-b02e-c991b116e4db-kube-api-access-66xfg\") pod \"maas-api-revoke-sub-3d4156f1-rqchz\" (UID: \"32f25527-b950-4ba5-b02e-c991b116e4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rqchz"
Oct 30 13:47:45.792291 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:45.792183    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f25527-b950-4ba5-b02e-c991b116e4db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rqchz\" (UID: \"32f25527-b950-4ba5-b02e-c991b116e4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rqchz"
Oct 30 13:47:45.893647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:45.893542    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66xfg\" (UniqueName: \"kubernetes.io/projected/32f25527-b950-4ba5-b02e-c991b116e4db-kube-api-access-66xfg\") pod \"maas-api-revoke-sub-3d4156f1-rqchz\" (UID: \"32f25527-b950-4ba5-b02e-c991b116e4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rqchz"
Oct 30 13:47:45.893647 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:45.893617    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f25527-b950-4ba5-b02e-c991b116e4db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rqchz\" (UID: \"32f25527-b950-4ba5-b02e-c991b116e4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rqchz"
Oct 30 13:47:45.893865 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:45.893685    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f25527-b950-4ba5-b02e-c991b116e4db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rqchz\" (UID: \"32f25527-b950-4ba5-b02e-c991b116e4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rqchz"
Oct 30 13:47:45.894377 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:45.894349    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f25527-b950-4ba5-b02e-c991b116e4db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rqchz\" (UID: \"32f25527-b950-4ba5-b02e-c991b116e4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rqchz"
Oct 30 13:47:45.896026 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:45.896005    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f25527-b950-4ba5-b02e-c991b116e4db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rqchz\" (UID: \"32f25527-b950-4ba5-b02e-c991b116e4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rqchz"
Oct 30 13:47:45.917314 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:45.917280    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66xfg\" (UniqueName: \"kubernetes.io/projected/32f25527-b950-4ba5-b02e-c991b116e4db-kube-api-access-66xfg\") pod \"maas-api-revoke-sub-3d4156f1-rqchz\" (UID: \"32f25527-b950-4ba5-b02e-c991b116e4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rqchz"
Oct 30 13:47:46.000030 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:45.999992    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rqchz"
Oct 30 13:47:46.302696 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:46.302612    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rqchz"]
Oct 30 13:47:46.306452 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:47:46.306415    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32f25527_b950_4ba5_b02e_c991b116e4db.slice/crio-8b76ae96191ff4be7432d502c9d625a2c8a20cbe4bf0d5427afd68356b642c48 WatchSource:0}: Error finding container 8b76ae96191ff4be7432d502c9d625a2c8a20cbe4bf0d5427afd68356b642c48: Status 404 returned error can't find the container with id 8b76ae96191ff4be7432d502c9d625a2c8a20cbe4bf0d5427afd68356b642c48
Oct 30 13:47:46.985160 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:46.985060    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rqchz" event={"ID":"32f25527-b950-4ba5-b02e-c991b116e4db","Type":"ContainerStarted","Data":"b0dc83f9f161bf5f018b04064b2ab5f4025a66731305f98020a8c8a82a9a31a8"}
Oct 30 13:47:46.985160 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:46.985118    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rqchz" event={"ID":"32f25527-b950-4ba5-b02e-c991b116e4db","Type":"ContainerStarted","Data":"8b76ae96191ff4be7432d502c9d625a2c8a20cbe4bf0d5427afd68356b642c48"}
Oct 30 13:47:47.027597 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:47.027542    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rqchz" podStartSLOduration=2.027525857 podStartE2EDuration="2.027525857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:47:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:47:47.025875494 +0000 UTC m=+3762.073009446" watchObservedRunningTime="2026-10-30 13:47:47.027525857 +0000 UTC m=+3762.074659804"
Oct 30 13:47:47.084873 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:47.084833    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f3d011b-3daa-4050-a42c-7d014717857d" path="/var/lib/kubelet/pods/2f3d011b-3daa-4050-a42c-7d014717857d/volumes"
Oct 30 13:47:47.992146 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:47.992110    4544 generic.go:356] "Generic (PLEG): container finished" podID="32f25527-b950-4ba5-b02e-c991b116e4db" containerID="b0dc83f9f161bf5f018b04064b2ab5f4025a66731305f98020a8c8a82a9a31a8" exitCode=0
Oct 30 13:47:47.992641 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:47.992205    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rqchz" event={"ID":"32f25527-b950-4ba5-b02e-c991b116e4db","Type":"ContainerDied","Data":"b0dc83f9f161bf5f018b04064b2ab5f4025a66731305f98020a8c8a82a9a31a8"}
Oct 30 13:47:49.148128 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:49.148101    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rqchz"
Oct 30 13:47:49.226738 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:49.226687    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32f25527-b950-4ba5-b02e-c991b116e4db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f25527-b950-4ba5-b02e-c991b116e4db-maas-api-xks-ca\") pod \"32f25527-b950-4ba5-b02e-c991b116e4db\" (UID: \"32f25527-b950-4ba5-b02e-c991b116e4db\") "
Oct 30 13:47:49.226929 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:49.226809    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32f25527-b950-4ba5-b02e-c991b116e4db-kube-api-access-66xfg\" (UniqueName: \"kubernetes.io/projected/32f25527-b950-4ba5-b02e-c991b116e4db-kube-api-access-66xfg\") pod \"32f25527-b950-4ba5-b02e-c991b116e4db\" (UID: \"32f25527-b950-4ba5-b02e-c991b116e4db\") "
Oct 30 13:47:49.226929 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:49.226865    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32f25527-b950-4ba5-b02e-c991b116e4db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f25527-b950-4ba5-b02e-c991b116e4db-maas-api-openshift-ca\") pod \"32f25527-b950-4ba5-b02e-c991b116e4db\" (UID: \"32f25527-b950-4ba5-b02e-c991b116e4db\") "
Oct 30 13:47:49.227329 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:49.227262    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32f25527-b950-4ba5-b02e-c991b116e4db-maas-api-openshift-ca" pod "32f25527-b950-4ba5-b02e-c991b116e4db" (UID: "32f25527-b950-4ba5-b02e-c991b116e4db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:47:49.229028 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:49.228999    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32f25527-b950-4ba5-b02e-c991b116e4db-maas-api-xks-ca" pod "32f25527-b950-4ba5-b02e-c991b116e4db" (UID: "32f25527-b950-4ba5-b02e-c991b116e4db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:47:49.229149 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:49.229068    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f25527-b950-4ba5-b02e-c991b116e4db-kube-api-access-66xfg" pod "32f25527-b950-4ba5-b02e-c991b116e4db" (UID: "32f25527-b950-4ba5-b02e-c991b116e4db"). InnerVolumeSpecName "kube-api-access-66xfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:47:49.327739 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:49.327697    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f25527-b950-4ba5-b02e-c991b116e4db-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:47:49.327739 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:49.327733    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66xfg\" (UniqueName: \"kubernetes.io/projected/32f25527-b950-4ba5-b02e-c991b116e4db-kube-api-access-66xfg\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:47:49.327739 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:49.327745    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f25527-b950-4ba5-b02e-c991b116e4db-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:47:50.004086 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:50.004054    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rqchz"
Oct 30 13:47:50.004299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:50.004088    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rqchz" event={"ID":"32f25527-b950-4ba5-b02e-c991b116e4db","Type":"ContainerDied","Data":"8b76ae96191ff4be7432d502c9d625a2c8a20cbe4bf0d5427afd68356b642c48"}
Oct 30 13:47:50.004299 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:50.004164    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b76ae96191ff4be7432d502c9d625a2c8a20cbe4bf0d5427afd68356b642c48"
Oct 30 13:47:59.009604 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:47:59.009565    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:47:59.412616 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:47:59.412571    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:03.245113 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:03.245073    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rqchz"]
Oct 30 13:48:03.256911 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:03.256875    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rqchz"]
Oct 30 13:48:03.293157 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:03.293117    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tvx4p"]
Oct 30 13:48:03.293635 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:03.293614    4544 cpu_manager.go:414] "containerMap: removing container" podUID="32f25527-b950-4ba5-b02e-c991b116e4db" containerName="revoke-keys"
Oct 30 13:48:03.293737 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:03.293637    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32f25527-b950-4ba5-b02e-c991b116e4db" containerName="revoke-keys"
Oct 30 13:48:03.293823 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:03.293807    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="32f25527-b950-4ba5-b02e-c991b116e4db" containerName="revoke-keys"
Oct 30 13:48:03.297292 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:03.297269    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tvx4p"
Oct 30 13:48:03.312123 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:03.312087    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:48:03.312329 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:03.312198    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:48:03.317376 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:03.317324    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tvx4p"]
Oct 30 13:48:03.375383 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:03.375335    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d4f8a9a-7238-41d4-ab06-39f44f2d11dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-tvx4p\" (UID: \"1d4f8a9a-7238-41d4-ab06-39f44f2d11dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tvx4p"
Oct 30 13:48:03.375566 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:03.375404    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d4f8a9a-7238-41d4-ab06-39f44f2d11dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-tvx4p\" (UID: \"1d4f8a9a-7238-41d4-ab06-39f44f2d11dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tvx4p"
Oct 30 13:48:03.375566 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:03.375523    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jhzg\" (UniqueName: \"kubernetes.io/projected/1d4f8a9a-7238-41d4-ab06-39f44f2d11dc-kube-api-access-7jhzg\") pod \"maas-api-revoke-sub-3d4156f1-tvx4p\" (UID: \"1d4f8a9a-7238-41d4-ab06-39f44f2d11dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tvx4p"
Oct 30 13:48:03.476505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:03.476465    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d4f8a9a-7238-41d4-ab06-39f44f2d11dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-tvx4p\" (UID: \"1d4f8a9a-7238-41d4-ab06-39f44f2d11dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tvx4p"
Oct 30 13:48:03.476706 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:03.476531    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d4f8a9a-7238-41d4-ab06-39f44f2d11dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-tvx4p\" (UID: \"1d4f8a9a-7238-41d4-ab06-39f44f2d11dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tvx4p"
Oct 30 13:48:03.476706 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:03.476561    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jhzg\" (UniqueName: \"kubernetes.io/projected/1d4f8a9a-7238-41d4-ab06-39f44f2d11dc-kube-api-access-7jhzg\") pod \"maas-api-revoke-sub-3d4156f1-tvx4p\" (UID: \"1d4f8a9a-7238-41d4-ab06-39f44f2d11dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tvx4p"
Oct 30 13:48:03.477233 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:03.477196    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d4f8a9a-7238-41d4-ab06-39f44f2d11dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-tvx4p\" (UID: \"1d4f8a9a-7238-41d4-ab06-39f44f2d11dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tvx4p"
Oct 30 13:48:03.478927 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:03.478906    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d4f8a9a-7238-41d4-ab06-39f44f2d11dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-tvx4p\" (UID: \"1d4f8a9a-7238-41d4-ab06-39f44f2d11dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tvx4p"
Oct 30 13:48:03.499924 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:03.499847    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jhzg\" (UniqueName: \"kubernetes.io/projected/1d4f8a9a-7238-41d4-ab06-39f44f2d11dc-kube-api-access-7jhzg\") pod \"maas-api-revoke-sub-3d4156f1-tvx4p\" (UID: \"1d4f8a9a-7238-41d4-ab06-39f44f2d11dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tvx4p"
Oct 30 13:48:03.610805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:03.610760    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tvx4p"
Oct 30 13:48:03.916980 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:03.916943    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tvx4p"]
Oct 30 13:48:03.918472 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:48:03.918442    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d4f8a9a_7238_41d4_ab06_39f44f2d11dc.slice/crio-a59761c72dd7dceedc2977afbdabf71c1432640ffede5f44c5a4a2765345e96e WatchSource:0}: Error finding container a59761c72dd7dceedc2977afbdabf71c1432640ffede5f44c5a4a2765345e96e: Status 404 returned error can't find the container with id a59761c72dd7dceedc2977afbdabf71c1432640ffede5f44c5a4a2765345e96e
Oct 30 13:48:04.029177 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:04.029131    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bc33589-vz7cz"]
Oct 30 13:48:04.031744 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:04.031709    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bc33589-vz7cz"]
Oct 30 13:48:04.081621 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:04.081583    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tvx4p" event={"ID":"1d4f8a9a-7238-41d4-ab06-39f44f2d11dc","Type":"ContainerStarted","Data":"a59761c72dd7dceedc2977afbdabf71c1432640ffede5f44c5a4a2765345e96e"}
Oct 30 13:48:05.085639 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:05.085603    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32f25527-b950-4ba5-b02e-c991b116e4db" path="/var/lib/kubelet/pods/32f25527-b950-4ba5-b02e-c991b116e4db/volumes"
Oct 30 13:48:05.086188 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:05.086172    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ebcb91d-6f3d-4f42-b029-0affb4abbbf9" path="/var/lib/kubelet/pods/4ebcb91d-6f3d-4f42-b029-0affb4abbbf9/volumes"
Oct 30 13:48:05.088758 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:05.088732    4544 generic.go:356] "Generic (PLEG): container finished" podID="1d4f8a9a-7238-41d4-ab06-39f44f2d11dc" containerID="989b1d7839fddf3442efc7fb12e6055b6124c7d644a2e77823dc2b976b5d1e36" exitCode=0
Oct 30 13:48:05.088905 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:05.088815    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tvx4p" event={"ID":"1d4f8a9a-7238-41d4-ab06-39f44f2d11dc","Type":"ContainerDied","Data":"989b1d7839fddf3442efc7fb12e6055b6124c7d644a2e77823dc2b976b5d1e36"}
Oct 30 13:48:06.242089 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:06.242046    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tvx4p"
Oct 30 13:48:06.407016 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:06.406920    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d4f8a9a-7238-41d4-ab06-39f44f2d11dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d4f8a9a-7238-41d4-ab06-39f44f2d11dc-maas-api-openshift-ca\") pod \"1d4f8a9a-7238-41d4-ab06-39f44f2d11dc\" (UID: \"1d4f8a9a-7238-41d4-ab06-39f44f2d11dc\") "
Oct 30 13:48:06.407016 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:06.407020    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d4f8a9a-7238-41d4-ab06-39f44f2d11dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d4f8a9a-7238-41d4-ab06-39f44f2d11dc-maas-api-xks-ca\") pod \"1d4f8a9a-7238-41d4-ab06-39f44f2d11dc\" (UID: \"1d4f8a9a-7238-41d4-ab06-39f44f2d11dc\") "
Oct 30 13:48:06.407207 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:06.407064    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d4f8a9a-7238-41d4-ab06-39f44f2d11dc-kube-api-access-7jhzg\" (UniqueName: \"kubernetes.io/projected/1d4f8a9a-7238-41d4-ab06-39f44f2d11dc-kube-api-access-7jhzg\") pod \"1d4f8a9a-7238-41d4-ab06-39f44f2d11dc\" (UID: \"1d4f8a9a-7238-41d4-ab06-39f44f2d11dc\") "
Oct 30 13:48:06.407382 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:06.407359    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d4f8a9a-7238-41d4-ab06-39f44f2d11dc-maas-api-openshift-ca" pod "1d4f8a9a-7238-41d4-ab06-39f44f2d11dc" (UID: "1d4f8a9a-7238-41d4-ab06-39f44f2d11dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:48:06.409141 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:06.409109    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d4f8a9a-7238-41d4-ab06-39f44f2d11dc-maas-api-xks-ca" pod "1d4f8a9a-7238-41d4-ab06-39f44f2d11dc" (UID: "1d4f8a9a-7238-41d4-ab06-39f44f2d11dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:48:06.409286 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:06.409153    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d4f8a9a-7238-41d4-ab06-39f44f2d11dc-kube-api-access-7jhzg" pod "1d4f8a9a-7238-41d4-ab06-39f44f2d11dc" (UID: "1d4f8a9a-7238-41d4-ab06-39f44f2d11dc"). InnerVolumeSpecName "kube-api-access-7jhzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:48:06.508173 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:06.508128    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d4f8a9a-7238-41d4-ab06-39f44f2d11dc-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:48:06.508173 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:06.508166    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d4f8a9a-7238-41d4-ab06-39f44f2d11dc-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:48:06.508173 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:06.508175    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jhzg\" (UniqueName: \"kubernetes.io/projected/1d4f8a9a-7238-41d4-ab06-39f44f2d11dc-kube-api-access-7jhzg\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:48:07.102291 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:07.102256    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tvx4p" event={"ID":"1d4f8a9a-7238-41d4-ab06-39f44f2d11dc","Type":"ContainerDied","Data":"a59761c72dd7dceedc2977afbdabf71c1432640ffede5f44c5a4a2765345e96e"}
Oct 30 13:48:07.102466 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:07.102315    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a59761c72dd7dceedc2977afbdabf71c1432640ffede5f44c5a4a2765345e96e"
Oct 30 13:48:07.102466 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:07.102344    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tvx4p"
Oct 30 13:48:15.636045 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:15.636009    4544 scope.go:122] "RemoveContainer" containerID="d23e1e88e41960458d322df59f15d9915fb67a6b166aaf5f339ef9171ead782b"
Oct 30 13:48:18.499791 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:18.499757    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:19.049447 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:19.049410    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:19.190832 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:19.190789    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:20.746729 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:20.746689    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:20.858709 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:20.858671    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:22.886835 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:22.886799    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:23.015818 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:23.015782    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:24.265247 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:24.265184    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t"]
Oct 30 13:48:24.265659 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:24.265621    4544 cpu_manager.go:414] "containerMap: removing container" podUID="1d4f8a9a-7238-41d4-ab06-39f44f2d11dc" containerName="revoke-keys"
Oct 30 13:48:24.265659 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:24.265636    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d4f8a9a-7238-41d4-ab06-39f44f2d11dc" containerName="revoke-keys"
Oct 30 13:48:24.265746 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:24.265735    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d4f8a9a-7238-41d4-ab06-39f44f2d11dc" containerName="revoke-keys"
Oct 30 13:48:24.270335 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:24.270314    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t"
Oct 30 13:48:24.291065 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:24.291030    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-f53955-serving-cert\""
Oct 30 13:48:24.291315 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:24.291118    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 13:48:24.301215 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:24.301187    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nb7kb\""
Oct 30 13:48:24.305211 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:24.305175    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t"]
Oct 30 13:48:24.347610 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:24.347579    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:24.384655 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:24.384617    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gxwm\" (UniqueName: \"kubernetes.io/projected/6603ea80-9688-4bf6-8052-3456ccd50d6a-kube-api-access-8gxwm\") pod \"maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t\" (UID: \"6603ea80-9688-4bf6-8052-3456ccd50d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t"
Oct 30 13:48:24.384841 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:24.384755    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6603ea80-9688-4bf6-8052-3456ccd50d6a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t\" (UID: \"6603ea80-9688-4bf6-8052-3456ccd50d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t"
Oct 30 13:48:24.384841 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:24.384813    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6603ea80-9688-4bf6-8052-3456ccd50d6a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t\" (UID: \"6603ea80-9688-4bf6-8052-3456ccd50d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t"
Oct 30 13:48:24.384841 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:24.384833    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6603ea80-9688-4bf6-8052-3456ccd50d6a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t\" (UID: \"6603ea80-9688-4bf6-8052-3456ccd50d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t"
Oct 30 13:48:24.474436 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:24.471481    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:24.486160 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:24.486121    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6603ea80-9688-4bf6-8052-3456ccd50d6a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t\" (UID: \"6603ea80-9688-4bf6-8052-3456ccd50d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t"
Oct 30 13:48:24.486392 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:24.486184    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6603ea80-9688-4bf6-8052-3456ccd50d6a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t\" (UID: \"6603ea80-9688-4bf6-8052-3456ccd50d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t"
Oct 30 13:48:24.486392 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:24.486207    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6603ea80-9688-4bf6-8052-3456ccd50d6a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t\" (UID: \"6603ea80-9688-4bf6-8052-3456ccd50d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t"
Oct 30 13:48:24.486392 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:24.486258    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gxwm\" (UniqueName: \"kubernetes.io/projected/6603ea80-9688-4bf6-8052-3456ccd50d6a-kube-api-access-8gxwm\") pod \"maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t\" (UID: \"6603ea80-9688-4bf6-8052-3456ccd50d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t"
Oct 30 13:48:24.486847 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:24.486823    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6603ea80-9688-4bf6-8052-3456ccd50d6a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t\" (UID: \"6603ea80-9688-4bf6-8052-3456ccd50d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t"
Oct 30 13:48:24.488772 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:24.488741    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6603ea80-9688-4bf6-8052-3456ccd50d6a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t\" (UID: \"6603ea80-9688-4bf6-8052-3456ccd50d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t"
Oct 30 13:48:24.488871 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:24.488785    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6603ea80-9688-4bf6-8052-3456ccd50d6a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t\" (UID: \"6603ea80-9688-4bf6-8052-3456ccd50d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t"
Oct 30 13:48:24.511822 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:24.511786    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gxwm\" (UniqueName: \"kubernetes.io/projected/6603ea80-9688-4bf6-8052-3456ccd50d6a-kube-api-access-8gxwm\") pod \"maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t\" (UID: \"6603ea80-9688-4bf6-8052-3456ccd50d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t"
Oct 30 13:48:24.585527 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:24.585483    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t"
Oct 30 13:48:24.615489 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:24.615450    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:24.782323 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:24.782288    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:24.910696 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:24.910662    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t"]
Oct 30 13:48:24.921147 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:24.916427    4544 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:48:25.222521 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:25.222438    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t" event={"ID":"6603ea80-9688-4bf6-8052-3456ccd50d6a","Type":"ContainerStarted","Data":"a1cc21b1211c1269fe6b10ddb885464374264a89cda8e1d2770c3fe455f6e458"}
Oct 30 13:48:26.084000 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:26.083963    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:26.229993 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:26.229949    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t" event={"ID":"6603ea80-9688-4bf6-8052-3456ccd50d6a","Type":"ContainerStarted","Data":"4bc29e3b45ec6b300d1ff489c6640904df85838e4f669601605087cc3c15d897"}
Oct 30 13:48:26.230212 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:26.230093    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t"
Oct 30 13:48:26.274530 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:26.274471    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t" podStartSLOduration=1.748820298 podStartE2EDuration="2.274454155s" totalImagesPullingTime="525.633857ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:26.273206041 +0000 UTC m=+3801.320339993" watchObservedRunningTime="2026-10-30 13:48:26.274454155 +0000 UTC m=+3801.321588107"
Oct 30 13:48:27.766864 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:27.766826    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:29.007084 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:48:29.007046    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:48:29.502916 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:29.502877    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:31.054822 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:31.054786    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:32.241085 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:32.241055    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t"
Oct 30 13:48:32.654749 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:32.654713    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:34.267576 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:34.267540    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:35.864091 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:35.864053    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:37.491635 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:37.491599    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:37.898965 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:37.898928    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:38.528631 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:38.528599    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:38.644707 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:38.644673    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:38.788554 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:38.788453    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:39.337895 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:39.337844    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:39.763901 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:39.763803    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:41.140513 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:41.140475    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:42.729151 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:42.729114    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:44.452575 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:44.452541    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:46.002911 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:46.002874    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:47.562274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:47.562171    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:49.269696 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:49.269654    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:50.952189 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:50.952153    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:52.632624 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:52.632583    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:54.249679 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:54.249640    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:55.788213 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:55.788168    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:56.028310 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:56.028159    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-66xvf"]
Oct 30 13:48:56.039769 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:56.039673    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-66xvf"]
Oct 30 13:48:57.084770 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:57.084732    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8fafc3f-2651-4647-9cb8-56a984441c97" path="/var/lib/kubelet/pods/c8fafc3f-2651-4647-9cb8-56a984441c97/volumes"
Oct 30 13:48:57.419036 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:57.418947    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:59.006615 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:48:59.006580    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:48:59.152463 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:59.152425    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:48:59.509051 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:48:59.509014    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:49:00.708657 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:00.708628    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:49:02.376570 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:02.376533    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:49:03.962305 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:03.962266    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:49:05.581289 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:05.581247    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:49:07.262253 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:07.262207    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:49:08.882270 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:08.882199    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:49:10.754617 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:10.754579    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:49:12.257134 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:12.257098    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:49:13.818388 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:13.818353    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:49:14.973143 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:14.973107    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:49:15.604601 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:15.604566    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:49:15.663007 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:15.662975    4544 scope.go:122] "RemoveContainer" containerID="488de18aa1625eaf36ab73a68985a0ed8fa71f6bfe8fad0ebf05d92c4a7d0b1b"
Oct 30 13:49:15.672239 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:15.672198    4544 scope.go:122] "RemoveContainer" containerID="c2de7b432fa7efa6a832e3aebddc87a637e0c4e6f2118477024c467febbf1c65"
Oct 30 13:49:15.680346 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:15.680319    4544 scope.go:122] "RemoveContainer" containerID="68a1037cbd27423558f5bd476cd7ee5eec3abf2f3dc55d82eea9e426a8f45379"
Oct 30 13:49:17.367418 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:17.367292    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:49:18.916346 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:18.916304    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:49:19.968328 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:19.968291    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:49:20.601752 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:20.601711    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:49:22.253666 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:22.253624    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:49:23.860713 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:23.860676    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:49:25.465623 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:25.465564    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:49:25.695982 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:25.695947    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:49:25.821172 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:25.821083    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:49:26.821486 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:26.821451    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t"]
Oct 30 13:49:26.822378 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:26.822329    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t" podUID="6603ea80-9688-4bf6-8052-3456ccd50d6a" containerName="maas-api" containerID="cri-o://4bc29e3b45ec6b300d1ff489c6640904df85838e4f669601605087cc3c15d897" gracePeriod=30
Oct 30 13:49:26.890463 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:26.890426    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:49:27.303818 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:27.303791    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t"
Oct 30 13:49:27.395156 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:27.395056    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6603ea80-9688-4bf6-8052-3456ccd50d6a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6603ea80-9688-4bf6-8052-3456ccd50d6a-metrics-certs\") pod \"6603ea80-9688-4bf6-8052-3456ccd50d6a\" (UID: \"6603ea80-9688-4bf6-8052-3456ccd50d6a\") "
Oct 30 13:49:27.395156 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:27.395126    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6603ea80-9688-4bf6-8052-3456ccd50d6a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6603ea80-9688-4bf6-8052-3456ccd50d6a-maas-api-tls\") pod \"6603ea80-9688-4bf6-8052-3456ccd50d6a\" (UID: \"6603ea80-9688-4bf6-8052-3456ccd50d6a\") "
Oct 30 13:49:27.395450 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:27.395187    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6603ea80-9688-4bf6-8052-3456ccd50d6a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6603ea80-9688-4bf6-8052-3456ccd50d6a-openshift-service-ca\") pod \"6603ea80-9688-4bf6-8052-3456ccd50d6a\" (UID: \"6603ea80-9688-4bf6-8052-3456ccd50d6a\") "
Oct 30 13:49:27.395450 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:27.395348    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6603ea80-9688-4bf6-8052-3456ccd50d6a-kube-api-access-8gxwm\" (UniqueName: \"kubernetes.io/projected/6603ea80-9688-4bf6-8052-3456ccd50d6a-kube-api-access-8gxwm\") pod \"6603ea80-9688-4bf6-8052-3456ccd50d6a\" (UID: \"6603ea80-9688-4bf6-8052-3456ccd50d6a\") "
Oct 30 13:49:27.395627 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:27.395592    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6603ea80-9688-4bf6-8052-3456ccd50d6a-openshift-service-ca" pod "6603ea80-9688-4bf6-8052-3456ccd50d6a" (UID: "6603ea80-9688-4bf6-8052-3456ccd50d6a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:49:27.395750 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:27.395689    4544 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6603ea80-9688-4bf6-8052-3456ccd50d6a-openshift-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:49:27.397175 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:27.397155    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6603ea80-9688-4bf6-8052-3456ccd50d6a-metrics-certs" pod "6603ea80-9688-4bf6-8052-3456ccd50d6a" (UID: "6603ea80-9688-4bf6-8052-3456ccd50d6a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:49:27.397425 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:27.397408    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6603ea80-9688-4bf6-8052-3456ccd50d6a-maas-api-tls" pod "6603ea80-9688-4bf6-8052-3456ccd50d6a" (UID: "6603ea80-9688-4bf6-8052-3456ccd50d6a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:49:27.397491 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:27.397438    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6603ea80-9688-4bf6-8052-3456ccd50d6a-kube-api-access-8gxwm" pod "6603ea80-9688-4bf6-8052-3456ccd50d6a" (UID: "6603ea80-9688-4bf6-8052-3456ccd50d6a"). InnerVolumeSpecName "kube-api-access-8gxwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:49:27.496406 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:27.496369    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gxwm\" (UniqueName: \"kubernetes.io/projected/6603ea80-9688-4bf6-8052-3456ccd50d6a-kube-api-access-8gxwm\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:49:27.496406 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:27.496398    4544 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6603ea80-9688-4bf6-8052-3456ccd50d6a-metrics-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:49:27.496406 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:27.496410    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6603ea80-9688-4bf6-8052-3456ccd50d6a-maas-api-tls\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:49:27.566195 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:27.566164    4544 generic.go:356] "Generic (PLEG): container finished" podID="6603ea80-9688-4bf6-8052-3456ccd50d6a" containerID="4bc29e3b45ec6b300d1ff489c6640904df85838e4f669601605087cc3c15d897" exitCode=0
Oct 30 13:49:27.566195 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:27.566207    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t" event={"ID":"6603ea80-9688-4bf6-8052-3456ccd50d6a","Type":"ContainerDied","Data":"4bc29e3b45ec6b300d1ff489c6640904df85838e4f669601605087cc3c15d897"}
Oct 30 13:49:27.566450 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:27.566208    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t"
Oct 30 13:49:27.566450 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:27.566262    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t" event={"ID":"6603ea80-9688-4bf6-8052-3456ccd50d6a","Type":"ContainerDied","Data":"a1cc21b1211c1269fe6b10ddb885464374264a89cda8e1d2770c3fe455f6e458"}
Oct 30 13:49:27.566450 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:27.566285    4544 scope.go:122] "RemoveContainer" containerID="4bc29e3b45ec6b300d1ff489c6640904df85838e4f669601605087cc3c15d897"
Oct 30 13:49:27.571798 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:27.571767    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:49:27.576333 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:27.576315    4544 scope.go:122] "RemoveContainer" containerID="4bc29e3b45ec6b300d1ff489c6640904df85838e4f669601605087cc3c15d897"
Oct 30 13:49:27.576624 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:49:27.576605    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bc29e3b45ec6b300d1ff489c6640904df85838e4f669601605087cc3c15d897\": container with ID starting with 4bc29e3b45ec6b300d1ff489c6640904df85838e4f669601605087cc3c15d897 not found: ID does not exist" containerID="4bc29e3b45ec6b300d1ff489c6640904df85838e4f669601605087cc3c15d897"
Oct 30 13:49:27.576676 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:27.576634    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bc29e3b45ec6b300d1ff489c6640904df85838e4f669601605087cc3c15d897"} err="failed to get container status \"4bc29e3b45ec6b300d1ff489c6640904df85838e4f669601605087cc3c15d897\": rpc error: code = NotFound desc = could not find container \"4bc29e3b45ec6b300d1ff489c6640904df85838e4f669601605087cc3c15d897\": container with ID starting with 4bc29e3b45ec6b300d1ff489c6640904df85838e4f669601605087cc3c15d897 not found: ID does not exist"
Oct 30 13:49:27.618417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:27.618378    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t"]
Oct 30 13:49:27.629957 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:27.629875    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t"]
Oct 30 13:49:27.684458 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:27.684378    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:49:28.216026 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:28.215978    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:49:28.580438 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:28.580400    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:49:29.005425 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:49:29.005338    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:49:29.084691 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:29.084657    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6603ea80-9688-4bf6-8052-3456ccd50d6a" path="/var/lib/kubelet/pods/6603ea80-9688-4bf6-8052-3456ccd50d6a/volumes"
Oct 30 13:49:30.235521 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:30.235474    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f53955-6f678d7686-f6z9t" podUID="6603ea80-9688-4bf6-8052-3456ccd50d6a" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.1.107:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:49:30.377525 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:30.377489    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:49:30.518098 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:30.517988    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:49:34.137766 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:34.137717    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:49:34.263699 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:34.263661    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:49:34.537618 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:34.537533    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:49:35.866075 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:35.866036    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tvx4p"]
Oct 30 13:49:35.877559 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:35.877529    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tvx4p"]
Oct 30 13:49:35.914453 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:35.914417    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g6lnw"]
Oct 30 13:49:35.914863 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:35.914850    4544 cpu_manager.go:414] "containerMap: removing container" podUID="6603ea80-9688-4bf6-8052-3456ccd50d6a" containerName="maas-api"
Oct 30 13:49:35.914909 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:35.914865    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6603ea80-9688-4bf6-8052-3456ccd50d6a" containerName="maas-api"
Oct 30 13:49:35.914950 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:35.914933    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="6603ea80-9688-4bf6-8052-3456ccd50d6a" containerName="maas-api"
Oct 30 13:49:35.919345 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:35.919322    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g6lnw"
Oct 30 13:49:35.937063 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:35.937034    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:49:35.937238 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:35.937169    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:49:35.940074 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:35.940040    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g6lnw"]
Oct 30 13:49:35.972862 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:35.972825    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttnp7\" (UniqueName: \"kubernetes.io/projected/92c9f740-bed1-4e48-a5b9-121ca74df46f-kube-api-access-ttnp7\") pod \"maas-api-revoke-sub-3d4156f1-g6lnw\" (UID: \"92c9f740-bed1-4e48-a5b9-121ca74df46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g6lnw"
Oct 30 13:49:35.973048 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:35.972886    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92c9f740-bed1-4e48-a5b9-121ca74df46f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-g6lnw\" (UID: \"92c9f740-bed1-4e48-a5b9-121ca74df46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g6lnw"
Oct 30 13:49:35.973048 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:35.972987    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92c9f740-bed1-4e48-a5b9-121ca74df46f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-g6lnw\" (UID: \"92c9f740-bed1-4e48-a5b9-121ca74df46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g6lnw"
Oct 30 13:49:36.074658 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:36.074610    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92c9f740-bed1-4e48-a5b9-121ca74df46f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-g6lnw\" (UID: \"92c9f740-bed1-4e48-a5b9-121ca74df46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g6lnw"
Oct 30 13:49:36.074855 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:36.074690    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92c9f740-bed1-4e48-a5b9-121ca74df46f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-g6lnw\" (UID: \"92c9f740-bed1-4e48-a5b9-121ca74df46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g6lnw"
Oct 30 13:49:36.074855 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:36.074788    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttnp7\" (UniqueName: \"kubernetes.io/projected/92c9f740-bed1-4e48-a5b9-121ca74df46f-kube-api-access-ttnp7\") pod \"maas-api-revoke-sub-3d4156f1-g6lnw\" (UID: \"92c9f740-bed1-4e48-a5b9-121ca74df46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g6lnw"
Oct 30 13:49:36.075492 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:36.075470    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92c9f740-bed1-4e48-a5b9-121ca74df46f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-g6lnw\" (UID: \"92c9f740-bed1-4e48-a5b9-121ca74df46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g6lnw"
Oct 30 13:49:36.077283 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:36.077254    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92c9f740-bed1-4e48-a5b9-121ca74df46f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-g6lnw\" (UID: \"92c9f740-bed1-4e48-a5b9-121ca74df46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g6lnw"
Oct 30 13:49:36.098581 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:36.098546    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttnp7\" (UniqueName: \"kubernetes.io/projected/92c9f740-bed1-4e48-a5b9-121ca74df46f-kube-api-access-ttnp7\") pod \"maas-api-revoke-sub-3d4156f1-g6lnw\" (UID: \"92c9f740-bed1-4e48-a5b9-121ca74df46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g6lnw"
Oct 30 13:49:36.232362 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:36.232263    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g6lnw"
Oct 30 13:49:36.535317 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:36.535210    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g6lnw"]
Oct 30 13:49:36.538831 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:49:36.538792    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c9f740_bed1_4e48_a5b9_121ca74df46f.slice/crio-097172540d0f69055e2674866a33665e8d0c9cbc7cb682cc820403d272d94334 WatchSource:0}: Error finding container 097172540d0f69055e2674866a33665e8d0c9cbc7cb682cc820403d272d94334: Status 404 returned error can't find the container with id 097172540d0f69055e2674866a33665e8d0c9cbc7cb682cc820403d272d94334
Oct 30 13:49:36.638791 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:36.638754    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g6lnw" event={"ID":"92c9f740-bed1-4e48-a5b9-121ca74df46f","Type":"ContainerStarted","Data":"097172540d0f69055e2674866a33665e8d0c9cbc7cb682cc820403d272d94334"}
Oct 30 13:49:37.086274 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:37.086212    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d4f8a9a-7238-41d4-ab06-39f44f2d11dc" path="/var/lib/kubelet/pods/1d4f8a9a-7238-41d4-ab06-39f44f2d11dc/volumes"
Oct 30 13:49:37.645413 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:37.645374    4544 generic.go:356] "Generic (PLEG): container finished" podID="92c9f740-bed1-4e48-a5b9-121ca74df46f" containerID="5a8959f3b803d5426b6d6faa9a063de498317e6bf1324648f670c90c8e4ebb01" exitCode=0
Oct 30 13:49:37.645629 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:37.645460    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g6lnw" event={"ID":"92c9f740-bed1-4e48-a5b9-121ca74df46f","Type":"ContainerDied","Data":"5a8959f3b803d5426b6d6faa9a063de498317e6bf1324648f670c90c8e4ebb01"}
Oct 30 13:49:38.798766 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:38.798733    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g6lnw"
Oct 30 13:49:38.903003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:38.902915    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92c9f740-bed1-4e48-a5b9-121ca74df46f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92c9f740-bed1-4e48-a5b9-121ca74df46f-maas-api-openshift-ca\") pod \"92c9f740-bed1-4e48-a5b9-121ca74df46f\" (UID: \"92c9f740-bed1-4e48-a5b9-121ca74df46f\") "
Oct 30 13:49:38.903003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:38.902958    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92c9f740-bed1-4e48-a5b9-121ca74df46f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92c9f740-bed1-4e48-a5b9-121ca74df46f-maas-api-xks-ca\") pod \"92c9f740-bed1-4e48-a5b9-121ca74df46f\" (UID: \"92c9f740-bed1-4e48-a5b9-121ca74df46f\") "
Oct 30 13:49:38.903300 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:38.903022    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92c9f740-bed1-4e48-a5b9-121ca74df46f-kube-api-access-ttnp7\" (UniqueName: \"kubernetes.io/projected/92c9f740-bed1-4e48-a5b9-121ca74df46f-kube-api-access-ttnp7\") pod \"92c9f740-bed1-4e48-a5b9-121ca74df46f\" (UID: \"92c9f740-bed1-4e48-a5b9-121ca74df46f\") "
Oct 30 13:49:38.903405 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:38.903378    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92c9f740-bed1-4e48-a5b9-121ca74df46f-maas-api-openshift-ca" pod "92c9f740-bed1-4e48-a5b9-121ca74df46f" (UID: "92c9f740-bed1-4e48-a5b9-121ca74df46f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:49:38.905209 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:38.905177    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92c9f740-bed1-4e48-a5b9-121ca74df46f-maas-api-xks-ca" pod "92c9f740-bed1-4e48-a5b9-121ca74df46f" (UID: "92c9f740-bed1-4e48-a5b9-121ca74df46f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:49:38.905209 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:38.905191    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92c9f740-bed1-4e48-a5b9-121ca74df46f-kube-api-access-ttnp7" pod "92c9f740-bed1-4e48-a5b9-121ca74df46f" (UID: "92c9f740-bed1-4e48-a5b9-121ca74df46f"). InnerVolumeSpecName "kube-api-access-ttnp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:49:39.003891 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:39.003854    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92c9f740-bed1-4e48-a5b9-121ca74df46f-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:49:39.003891 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:39.003886    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92c9f740-bed1-4e48-a5b9-121ca74df46f-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:49:39.003891 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:39.003896    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttnp7\" (UniqueName: \"kubernetes.io/projected/92c9f740-bed1-4e48-a5b9-121ca74df46f-kube-api-access-ttnp7\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:49:39.657035 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:39.657007    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g6lnw"
Oct 30 13:49:39.657272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:39.657001    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g6lnw" event={"ID":"92c9f740-bed1-4e48-a5b9-121ca74df46f","Type":"ContainerDied","Data":"097172540d0f69055e2674866a33665e8d0c9cbc7cb682cc820403d272d94334"}
Oct 30 13:49:39.657272 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:39.657135    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="097172540d0f69055e2674866a33665e8d0c9cbc7cb682cc820403d272d94334"
Oct 30 13:49:45.028915 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:45.028876    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8c4l4"]
Oct 30 13:49:45.040843 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:45.040811    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8c4l4"]
Oct 30 13:49:45.085672 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:45.085634    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43d7da87-2345-40ed-a5cb-96c2438eaf5a" path="/var/lib/kubelet/pods/43d7da87-2345-40ed-a5cb-96c2438eaf5a/volumes"
Oct 30 13:49:56.339752 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:56.339720    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:49:58.330778 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:58.330734    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:49:59.007076 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:49:59.007041    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:49:59.644701 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:49:59.644664    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:50:02.486813 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:50:02.486775    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:50:04.150476 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:50:04.150441    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:50:12.605770 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:50:12.605725    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:50:15.707655 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:50:15.707620    4544 scope.go:122] "RemoveContainer" containerID="7df008c94d72b07ed1f2e5ef297bb6d020196d254c18e536c0189d6245a6f935"
Oct 30 13:50:29.007827 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:50:29.007735    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:50:45.061277 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:50:45.061235    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xwvvp"]
Oct 30 13:50:45.072700 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:50:45.072659    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xwvvp"]
Oct 30 13:50:45.085003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:50:45.084966    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="819366a4-ff59-41e8-a51a-51dba3404a76" path="/var/lib/kubelet/pods/819366a4-ff59-41e8-a51a-51dba3404a76/volumes"
Oct 30 13:50:56.051913 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:50:56.051875    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6jqc6"]
Oct 30 13:50:56.063242 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:50:56.063194    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6jqc6"]
Oct 30 13:50:57.084824 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:50:57.084789    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="365c34a6-53b4-4f52-bc89-3408a579937a" path="/var/lib/kubelet/pods/365c34a6-53b4-4f52-bc89-3408a579937a/volumes"
Oct 30 13:50:59.006788 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:50:59.006751    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:50:59.791782 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:50:59.791749    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:51:15.732748 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:15.732694    4544 scope.go:122] "RemoveContainer" containerID="fc3d77339479044de1242eafb71000ec238d759941526d6bd6e404a8e6c57bed"
Oct 30 13:51:17.033120 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:17.033072    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-t8skv"]
Oct 30 13:51:17.043172 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:17.043137    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-t8skv"]
Oct 30 13:51:17.161426 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:17.161198    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0344ce9-437c-4203-ac7a-dd3036521b2c" path="/var/lib/kubelet/pods/a0344ce9-437c-4203-ac7a-dd3036521b2c/volumes"
Oct 30 13:51:18.030455 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:18.030412    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vx8q2"]
Oct 30 13:51:18.031631 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:18.031602    4544 cpu_manager.go:414] "containerMap: removing container" podUID="92c9f740-bed1-4e48-a5b9-121ca74df46f" containerName="revoke-keys"
Oct 30 13:51:18.031817 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:18.031803    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92c9f740-bed1-4e48-a5b9-121ca74df46f" containerName="revoke-keys"
Oct 30 13:51:18.032194 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:18.032177    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="92c9f740-bed1-4e48-a5b9-121ca74df46f" containerName="revoke-keys"
Oct 30 13:51:18.038124 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:18.038102    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vx8q2"
Oct 30 13:51:18.055001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:18.054962    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vx8q2"]
Oct 30 13:51:18.064940 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:18.064917    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:51:18.065058 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:18.064947    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:51:18.140849 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:18.140814    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a6b8940-4819-4258-801d-e38ba52c39f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-vx8q2\" (UID: \"5a6b8940-4819-4258-801d-e38ba52c39f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vx8q2"
Oct 30 13:51:18.140849 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:18.140855    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c8qk\" (UniqueName: \"kubernetes.io/projected/5a6b8940-4819-4258-801d-e38ba52c39f2-kube-api-access-6c8qk\") pod \"maas-api-revoke-sub-d576e68b-vx8q2\" (UID: \"5a6b8940-4819-4258-801d-e38ba52c39f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vx8q2"
Oct 30 13:51:18.141083 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:18.140939    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a6b8940-4819-4258-801d-e38ba52c39f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-vx8q2\" (UID: \"5a6b8940-4819-4258-801d-e38ba52c39f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vx8q2"
Oct 30 13:51:18.242683 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:18.242646    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a6b8940-4819-4258-801d-e38ba52c39f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-vx8q2\" (UID: \"5a6b8940-4819-4258-801d-e38ba52c39f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vx8q2"
Oct 30 13:51:18.242683 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:18.242688    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c8qk\" (UniqueName: \"kubernetes.io/projected/5a6b8940-4819-4258-801d-e38ba52c39f2-kube-api-access-6c8qk\") pod \"maas-api-revoke-sub-d576e68b-vx8q2\" (UID: \"5a6b8940-4819-4258-801d-e38ba52c39f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vx8q2"
Oct 30 13:51:18.242928 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:18.242723    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a6b8940-4819-4258-801d-e38ba52c39f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-vx8q2\" (UID: \"5a6b8940-4819-4258-801d-e38ba52c39f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vx8q2"
Oct 30 13:51:18.243347 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:18.243319    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a6b8940-4819-4258-801d-e38ba52c39f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-vx8q2\" (UID: \"5a6b8940-4819-4258-801d-e38ba52c39f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vx8q2"
Oct 30 13:51:18.245066 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:18.245039    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a6b8940-4819-4258-801d-e38ba52c39f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-vx8q2\" (UID: \"5a6b8940-4819-4258-801d-e38ba52c39f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vx8q2"
Oct 30 13:51:18.265698 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:18.265668    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c8qk\" (UniqueName: \"kubernetes.io/projected/5a6b8940-4819-4258-801d-e38ba52c39f2-kube-api-access-6c8qk\") pod \"maas-api-revoke-sub-d576e68b-vx8q2\" (UID: \"5a6b8940-4819-4258-801d-e38ba52c39f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vx8q2"
Oct 30 13:51:18.350958 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:18.350918    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vx8q2"
Oct 30 13:51:18.656123 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:18.656037    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vx8q2"]
Oct 30 13:51:18.660330 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:51:18.660303    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a6b8940_4819_4258_801d_e38ba52c39f2.slice/crio-fa24bdfa8eb73adb688288b91ac92af3e802ac9f5fd7b389edf5d4d78884afdf WatchSource:0}: Error finding container fa24bdfa8eb73adb688288b91ac92af3e802ac9f5fd7b389edf5d4d78884afdf: Status 404 returned error can't find the container with id fa24bdfa8eb73adb688288b91ac92af3e802ac9f5fd7b389edf5d4d78884afdf
Oct 30 13:51:19.216348 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:19.216260    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vx8q2" event={"ID":"5a6b8940-4819-4258-801d-e38ba52c39f2","Type":"ContainerStarted","Data":"e5c19ea63167f15cbcd0e428cbef6ef97c6de3d29e87fa41b0d60c5e58e03c51"}
Oct 30 13:51:19.216348 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:19.216317    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vx8q2" event={"ID":"5a6b8940-4819-4258-801d-e38ba52c39f2","Type":"ContainerStarted","Data":"fa24bdfa8eb73adb688288b91ac92af3e802ac9f5fd7b389edf5d4d78884afdf"}
Oct 30 13:51:19.260001 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:19.259820    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vx8q2" podStartSLOduration=1.259800977 podStartE2EDuration="1.259800977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:51:19.258417015 +0000 UTC m=+3974.305550967" watchObservedRunningTime="2026-10-30 13:51:19.259800977 +0000 UTC m=+3974.306934933"
Oct 30 13:51:20.222637 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:20.222601    4544 generic.go:356] "Generic (PLEG): container finished" podID="5a6b8940-4819-4258-801d-e38ba52c39f2" containerID="e5c19ea63167f15cbcd0e428cbef6ef97c6de3d29e87fa41b0d60c5e58e03c51" exitCode=0
Oct 30 13:51:20.223088 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:20.222697    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vx8q2" event={"ID":"5a6b8940-4819-4258-801d-e38ba52c39f2","Type":"ContainerDied","Data":"e5c19ea63167f15cbcd0e428cbef6ef97c6de3d29e87fa41b0d60c5e58e03c51"}
Oct 30 13:51:21.373309 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:21.373280    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vx8q2"
Oct 30 13:51:21.476492 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:21.476459    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a6b8940-4819-4258-801d-e38ba52c39f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a6b8940-4819-4258-801d-e38ba52c39f2-maas-api-xks-ca\") pod \"5a6b8940-4819-4258-801d-e38ba52c39f2\" (UID: \"5a6b8940-4819-4258-801d-e38ba52c39f2\") "
Oct 30 13:51:21.476693 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:21.476585    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a6b8940-4819-4258-801d-e38ba52c39f2-kube-api-access-6c8qk\" (UniqueName: \"kubernetes.io/projected/5a6b8940-4819-4258-801d-e38ba52c39f2-kube-api-access-6c8qk\") pod \"5a6b8940-4819-4258-801d-e38ba52c39f2\" (UID: \"5a6b8940-4819-4258-801d-e38ba52c39f2\") "
Oct 30 13:51:21.476693 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:21.476636    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a6b8940-4819-4258-801d-e38ba52c39f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a6b8940-4819-4258-801d-e38ba52c39f2-maas-api-openshift-ca\") pod \"5a6b8940-4819-4258-801d-e38ba52c39f2\" (UID: \"5a6b8940-4819-4258-801d-e38ba52c39f2\") "
Oct 30 13:51:21.477079 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:21.477054    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a6b8940-4819-4258-801d-e38ba52c39f2-maas-api-openshift-ca" pod "5a6b8940-4819-4258-801d-e38ba52c39f2" (UID: "5a6b8940-4819-4258-801d-e38ba52c39f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:51:21.478719 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:21.478658    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a6b8940-4819-4258-801d-e38ba52c39f2-maas-api-xks-ca" pod "5a6b8940-4819-4258-801d-e38ba52c39f2" (UID: "5a6b8940-4819-4258-801d-e38ba52c39f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:51:21.478719 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:21.478704    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a6b8940-4819-4258-801d-e38ba52c39f2-kube-api-access-6c8qk" pod "5a6b8940-4819-4258-801d-e38ba52c39f2" (UID: "5a6b8940-4819-4258-801d-e38ba52c39f2"). InnerVolumeSpecName "kube-api-access-6c8qk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:51:21.578292 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:21.578248    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c8qk\" (UniqueName: \"kubernetes.io/projected/5a6b8940-4819-4258-801d-e38ba52c39f2-kube-api-access-6c8qk\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:51:21.578292 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:21.578294    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a6b8940-4819-4258-801d-e38ba52c39f2-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:51:21.578292 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:21.578320    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a6b8940-4819-4258-801d-e38ba52c39f2-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:51:22.233721 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:22.233688    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vx8q2"
Oct 30 13:51:22.233721 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:22.233706    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vx8q2" event={"ID":"5a6b8940-4819-4258-801d-e38ba52c39f2","Type":"ContainerDied","Data":"fa24bdfa8eb73adb688288b91ac92af3e802ac9f5fd7b389edf5d4d78884afdf"}
Oct 30 13:51:22.233978 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:22.233750    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa24bdfa8eb73adb688288b91ac92af3e802ac9f5fd7b389edf5d4d78884afdf"
Oct 30 13:51:29.005295 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:51:29.005257    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:51:34.463417 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:34.463359    4544 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7grvt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:51:34.464185 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:34.464142    4544 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt" podUID="c16d5364-17da-4fef-993d-9ac8a77f56a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:51:34.758075 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:34.757993    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z4z5"]
Oct 30 13:51:34.758852 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:34.758820    4544 cpu_manager.go:414] "containerMap: removing container" podUID="5a6b8940-4819-4258-801d-e38ba52c39f2" containerName="revoke-keys"
Oct 30 13:51:34.758852 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:34.758849    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a6b8940-4819-4258-801d-e38ba52c39f2" containerName="revoke-keys"
Oct 30 13:51:34.759070 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:34.758941    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a6b8940-4819-4258-801d-e38ba52c39f2" containerName="revoke-keys"
Oct 30 13:51:34.762410 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:34.762388    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z4z5"
Oct 30 13:51:34.777627 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:34.777601    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:51:34.789848 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:34.789816    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:51:34.792775 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:34.792738    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z4z5"]
Oct 30 13:51:34.848863 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:34.848825    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49d0fb91-0749-4365-9257-c14ba79f85d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4z4z5\" (UID: \"49d0fb91-0749-4365-9257-c14ba79f85d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z4z5"
Oct 30 13:51:34.848863 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:34.848864    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n657\" (UniqueName: \"kubernetes.io/projected/49d0fb91-0749-4365-9257-c14ba79f85d0-kube-api-access-4n657\") pod \"maas-api-revoke-sub-d2bc93bc-4z4z5\" (UID: \"49d0fb91-0749-4365-9257-c14ba79f85d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z4z5"
Oct 30 13:51:34.849087 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:34.849062    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49d0fb91-0749-4365-9257-c14ba79f85d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4z4z5\" (UID: \"49d0fb91-0749-4365-9257-c14ba79f85d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z4z5"
Oct 30 13:51:34.950050 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:34.950010    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49d0fb91-0749-4365-9257-c14ba79f85d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4z4z5\" (UID: \"49d0fb91-0749-4365-9257-c14ba79f85d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z4z5"
Oct 30 13:51:34.950287 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:34.950062    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49d0fb91-0749-4365-9257-c14ba79f85d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4z4z5\" (UID: \"49d0fb91-0749-4365-9257-c14ba79f85d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z4z5"
Oct 30 13:51:34.950287 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:34.950193    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n657\" (UniqueName: \"kubernetes.io/projected/49d0fb91-0749-4365-9257-c14ba79f85d0-kube-api-access-4n657\") pod \"maas-api-revoke-sub-d2bc93bc-4z4z5\" (UID: \"49d0fb91-0749-4365-9257-c14ba79f85d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z4z5"
Oct 30 13:51:34.950738 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:34.950712    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49d0fb91-0749-4365-9257-c14ba79f85d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4z4z5\" (UID: \"49d0fb91-0749-4365-9257-c14ba79f85d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z4z5"
Oct 30 13:51:34.952424 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:34.952401    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49d0fb91-0749-4365-9257-c14ba79f85d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4z4z5\" (UID: \"49d0fb91-0749-4365-9257-c14ba79f85d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z4z5"
Oct 30 13:51:34.973619 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:34.973592    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n657\" (UniqueName: \"kubernetes.io/projected/49d0fb91-0749-4365-9257-c14ba79f85d0-kube-api-access-4n657\") pod \"maas-api-revoke-sub-d2bc93bc-4z4z5\" (UID: \"49d0fb91-0749-4365-9257-c14ba79f85d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z4z5"
Oct 30 13:51:35.078248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:35.078204    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z4z5"
Oct 30 13:51:35.383750 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:35.383671    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z4z5"]
Oct 30 13:51:35.387059 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:51:35.387030    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d0fb91_0749_4365_9257_c14ba79f85d0.slice/crio-307354f4c8cbdd83f58cdfb649a9eb41ca3761fcf3cb47bcd76cdb3fcd5207c3 WatchSource:0}: Error finding container 307354f4c8cbdd83f58cdfb649a9eb41ca3761fcf3cb47bcd76cdb3fcd5207c3: Status 404 returned error can't find the container with id 307354f4c8cbdd83f58cdfb649a9eb41ca3761fcf3cb47bcd76cdb3fcd5207c3
Oct 30 13:51:36.341095 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:36.341056    4544 generic.go:356] "Generic (PLEG): container finished" podID="49d0fb91-0749-4365-9257-c14ba79f85d0" containerID="60f44775cf6854212693786dc78bc83a10680087d1bf01d1a409993810fd144c" exitCode=0
Oct 30 13:51:36.341556 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:36.341146    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z4z5" event={"ID":"49d0fb91-0749-4365-9257-c14ba79f85d0","Type":"ContainerDied","Data":"60f44775cf6854212693786dc78bc83a10680087d1bf01d1a409993810fd144c"}
Oct 30 13:51:36.341556 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:36.341234    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z4z5" event={"ID":"49d0fb91-0749-4365-9257-c14ba79f85d0","Type":"ContainerStarted","Data":"307354f4c8cbdd83f58cdfb649a9eb41ca3761fcf3cb47bcd76cdb3fcd5207c3"}
Oct 30 13:51:37.485334 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:37.485306    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z4z5"
Oct 30 13:51:37.575690 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:37.575653    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49d0fb91-0749-4365-9257-c14ba79f85d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49d0fb91-0749-4365-9257-c14ba79f85d0-maas-api-xks-ca\") pod \"49d0fb91-0749-4365-9257-c14ba79f85d0\" (UID: \"49d0fb91-0749-4365-9257-c14ba79f85d0\") "
Oct 30 13:51:37.575871 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:37.575711    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49d0fb91-0749-4365-9257-c14ba79f85d0-kube-api-access-4n657\" (UniqueName: \"kubernetes.io/projected/49d0fb91-0749-4365-9257-c14ba79f85d0-kube-api-access-4n657\") pod \"49d0fb91-0749-4365-9257-c14ba79f85d0\" (UID: \"49d0fb91-0749-4365-9257-c14ba79f85d0\") "
Oct 30 13:51:37.575871 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:37.575732    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49d0fb91-0749-4365-9257-c14ba79f85d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49d0fb91-0749-4365-9257-c14ba79f85d0-maas-api-openshift-ca\") pod \"49d0fb91-0749-4365-9257-c14ba79f85d0\" (UID: \"49d0fb91-0749-4365-9257-c14ba79f85d0\") "
Oct 30 13:51:37.576195 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:37.576170    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49d0fb91-0749-4365-9257-c14ba79f85d0-maas-api-openshift-ca" pod "49d0fb91-0749-4365-9257-c14ba79f85d0" (UID: "49d0fb91-0749-4365-9257-c14ba79f85d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:51:37.577848 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:37.577829    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49d0fb91-0749-4365-9257-c14ba79f85d0-maas-api-xks-ca" pod "49d0fb91-0749-4365-9257-c14ba79f85d0" (UID: "49d0fb91-0749-4365-9257-c14ba79f85d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:51:37.577941 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:37.577922    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49d0fb91-0749-4365-9257-c14ba79f85d0-kube-api-access-4n657" pod "49d0fb91-0749-4365-9257-c14ba79f85d0" (UID: "49d0fb91-0749-4365-9257-c14ba79f85d0"). InnerVolumeSpecName "kube-api-access-4n657". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:51:37.677363 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:37.677267    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49d0fb91-0749-4365-9257-c14ba79f85d0-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:51:37.677363 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:37.677311    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n657\" (UniqueName: \"kubernetes.io/projected/49d0fb91-0749-4365-9257-c14ba79f85d0-kube-api-access-4n657\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:51:37.677363 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:37.677321    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49d0fb91-0749-4365-9257-c14ba79f85d0-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:51:38.352496 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:38.352468    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z4z5"
Oct 30 13:51:38.352496 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:38.352480    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z4z5" event={"ID":"49d0fb91-0749-4365-9257-c14ba79f85d0","Type":"ContainerDied","Data":"307354f4c8cbdd83f58cdfb649a9eb41ca3761fcf3cb47bcd76cdb3fcd5207c3"}
Oct 30 13:51:38.352746 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:38.352528    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="307354f4c8cbdd83f58cdfb649a9eb41ca3761fcf3cb47bcd76cdb3fcd5207c3"
Oct 30 13:51:39.030120 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:39.030079    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bp7f9"]
Oct 30 13:51:39.041927 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:39.041896    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bp7f9"]
Oct 30 13:51:39.086388 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:39.086346    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e287f86b-364b-4a4f-bcd1-4a99c695d1da" path="/var/lib/kubelet/pods/e287f86b-364b-4a4f-bcd1-4a99c695d1da/volumes"
Oct 30 13:51:46.590931 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:46.590891    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:51:48.260233 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:48.260182    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:51:51.038376 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:51.038331    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:51:51.171893 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:51.171857    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:51:52.386313 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:52.386272    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q"]
Oct 30 13:51:52.387088 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:52.386661    4544 cpu_manager.go:414] "containerMap: removing container" podUID="49d0fb91-0749-4365-9257-c14ba79f85d0" containerName="revoke-keys"
Oct 30 13:51:52.387088 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:52.386672    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49d0fb91-0749-4365-9257-c14ba79f85d0" containerName="revoke-keys"
Oct 30 13:51:52.387088 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:52.386765    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="49d0fb91-0749-4365-9257-c14ba79f85d0" containerName="revoke-keys"
Oct 30 13:51:52.389650 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:52.389632    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q"
Oct 30 13:51:52.416443 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:52.416397    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-bf43af-serving-cert\""
Oct 30 13:51:52.416443 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:52.416397    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 13:51:52.416689 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:52.416489    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nb7kb\""
Oct 30 13:51:52.420965 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:52.420929    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q"]
Oct 30 13:51:52.483661 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:52.483619    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:51:52.523792 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:52.523748    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/968756d1-6e26-4f17-aea1-f534a8b1330b-metrics-certs\") pod \"maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q\" (UID: \"968756d1-6e26-4f17-aea1-f534a8b1330b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q"
Oct 30 13:51:52.523792 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:52.523790    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/968756d1-6e26-4f17-aea1-f534a8b1330b-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q\" (UID: \"968756d1-6e26-4f17-aea1-f534a8b1330b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q"
Oct 30 13:51:52.524061 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:52.523955    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/968756d1-6e26-4f17-aea1-f534a8b1330b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q\" (UID: \"968756d1-6e26-4f17-aea1-f534a8b1330b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q"
Oct 30 13:51:52.524061 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:52.523990    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv8z2\" (UniqueName: \"kubernetes.io/projected/968756d1-6e26-4f17-aea1-f534a8b1330b-kube-api-access-tv8z2\") pod \"maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q\" (UID: \"968756d1-6e26-4f17-aea1-f534a8b1330b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q"
Oct 30 13:51:52.611570 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:52.611522    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:51:52.625593 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:52.625536    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/968756d1-6e26-4f17-aea1-f534a8b1330b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q\" (UID: \"968756d1-6e26-4f17-aea1-f534a8b1330b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q"
Oct 30 13:51:52.625593 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:52.625580    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv8z2\" (UniqueName: \"kubernetes.io/projected/968756d1-6e26-4f17-aea1-f534a8b1330b-kube-api-access-tv8z2\") pod \"maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q\" (UID: \"968756d1-6e26-4f17-aea1-f534a8b1330b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q"
Oct 30 13:51:52.625902 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:52.625663    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/968756d1-6e26-4f17-aea1-f534a8b1330b-metrics-certs\") pod \"maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q\" (UID: \"968756d1-6e26-4f17-aea1-f534a8b1330b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q"
Oct 30 13:51:52.625902 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:52.625682    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/968756d1-6e26-4f17-aea1-f534a8b1330b-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q\" (UID: \"968756d1-6e26-4f17-aea1-f534a8b1330b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q"
Oct 30 13:51:52.626476 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:52.626448    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/968756d1-6e26-4f17-aea1-f534a8b1330b-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q\" (UID: \"968756d1-6e26-4f17-aea1-f534a8b1330b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q"
Oct 30 13:51:52.628215 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:52.628190    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/968756d1-6e26-4f17-aea1-f534a8b1330b-metrics-certs\") pod \"maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q\" (UID: \"968756d1-6e26-4f17-aea1-f534a8b1330b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q"
Oct 30 13:51:52.628339 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:52.628249    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/968756d1-6e26-4f17-aea1-f534a8b1330b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q\" (UID: \"968756d1-6e26-4f17-aea1-f534a8b1330b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q"
Oct 30 13:51:52.649797 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:52.649714    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv8z2\" (UniqueName: \"kubernetes.io/projected/968756d1-6e26-4f17-aea1-f534a8b1330b-kube-api-access-tv8z2\") pod \"maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q\" (UID: \"968756d1-6e26-4f17-aea1-f534a8b1330b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q"
Oct 30 13:51:52.702631 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:52.702565    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q"
Oct 30 13:51:52.901851 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:52.901689    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:51:53.020277 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:53.020239    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q"]
Oct 30 13:51:53.024971 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:51:53.024939    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968756d1_6e26_4f17_aea1_f534a8b1330b.slice/crio-c7030bdf89506fee25f17505a0e45cc325e4d04806f7d4378876afcff24eb17f WatchSource:0}: Error finding container c7030bdf89506fee25f17505a0e45cc325e4d04806f7d4378876afcff24eb17f: Status 404 returned error can't find the container with id c7030bdf89506fee25f17505a0e45cc325e4d04806f7d4378876afcff24eb17f
Oct 30 13:51:53.089316 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:53.089280    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:51:53.448211 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:53.448172    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q" event={"ID":"968756d1-6e26-4f17-aea1-f534a8b1330b","Type":"ContainerStarted","Data":"d8f1e25ac012b80e1256b82f3ca043b380749d7be561194046a6f69093c7fa29"}
Oct 30 13:51:53.448651 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:53.448253    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q"
Oct 30 13:51:53.448651 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:53.448279    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q" event={"ID":"968756d1-6e26-4f17-aea1-f534a8b1330b","Type":"ContainerStarted","Data":"c7030bdf89506fee25f17505a0e45cc325e4d04806f7d4378876afcff24eb17f"}
Oct 30 13:51:53.492703 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:53.492637    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q" podStartSLOduration=1.20280271 podStartE2EDuration="1.492615885s" totalImagesPullingTime="289.813175ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:51:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:51:53.489798279 +0000 UTC m=+4008.536932213" watchObservedRunningTime="2026-10-30 13:51:53.492615885 +0000 UTC m=+4008.539749839"
Oct 30 13:51:54.460774 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:54.460737    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:51:56.080405 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:56.080371    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:51:57.672878 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:57.672843    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:51:59.005451 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:51:59.005412    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:51:59.033284 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:59.033240    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xc4b8"]
Oct 30 13:51:59.044686 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:59.044655    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xc4b8"]
Oct 30 13:51:59.087303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:59.087237    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f42cebea-cc55-4430-b6ff-c1abf0b0c47c" path="/var/lib/kubelet/pods/f42cebea-cc55-4430-b6ff-c1abf0b0c47c/volumes"
Oct 30 13:51:59.374348 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:59.374311    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:51:59.459428 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:59.459397    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q"
Oct 30 13:51:59.873069 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:51:59.873031    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:00.167981 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:00.167892    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:00.922704 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:00.922660    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:02.841624 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:02.841586    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:04.384621 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:04.384589    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:05.923765 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:05.923708    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:06.055538 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:06.055484    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:06.690872 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:06.690838    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:06.888094 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:06.888058    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:07.013542 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:07.013444    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:07.766943 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:07.766905    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:08.438188 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:08.438151    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:09.477029 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:09.476991    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:11.045625 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:11.045588    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:12.698936 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:12.698895    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:14.339150 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:14.339107    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:15.771768 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:15.771730    4544 scope.go:122] "RemoveContainer" containerID="65eb362ec3fabb22525be56d1cb3259fb6be095fba87b9159af99cbb25198230"
Oct 30 13:52:15.780777 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:15.780750    4544 scope.go:122] "RemoveContainer" containerID="52320b68fd5bfcb4c81647212b3376634c319d43af9553461d13742d1d48f524"
Oct 30 13:52:15.789084 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:15.789058    4544 scope.go:122] "RemoveContainer" containerID="b3c40ecaea0831886f4cc1d14da88e5be80e1f9273c3ebdbe07352e99550dac3"
Oct 30 13:52:15.883235 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:15.883189    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:17.027905 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:17.027863    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pdgdk"]
Oct 30 13:52:17.039539 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:17.039508    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pdgdk"]
Oct 30 13:52:17.084762 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:17.084729    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5f2b7a6-bdb6-4096-92c9-ed10b209c297" path="/var/lib/kubelet/pods/b5f2b7a6-bdb6-4096-92c9-ed10b209c297/volumes"
Oct 30 13:52:17.444808 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:17.444768    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:18.990158 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:18.990127    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:20.647955 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:20.647910    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:22.183775 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:22.183738    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:23.724119 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:23.724089    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:25.561630 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:25.561589    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:27.100039 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:27.100005    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:28.723843 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:28.723805    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:29.007454 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:52:29.007365    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:52:30.332942 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:30.332870    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:31.910641 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:31.910600    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:33.655985 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:33.655941    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:35.286076 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:35.286039    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:36.848817 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:36.848781    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:38.460973 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:38.460940    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:39.092006 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:39.091966    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:40.222882 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:40.222843    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:41.940087 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:41.940050    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:43.683416 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:43.683380    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:44.127844 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:44.127811    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:45.206875 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:45.206842    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:46.769995 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:46.769952    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:48.326360 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:48.326326    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:49.856638 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:49.856604    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:49.995851 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:49.995812    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:50.260366 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:50.260256    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:51.325109 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:51.325069    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q"]
Oct 30 13:52:51.325645 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:51.325294    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q" podUID="968756d1-6e26-4f17-aea1-f534a8b1330b" containerName="maas-api" containerID="cri-o://d8f1e25ac012b80e1256b82f3ca043b380749d7be561194046a6f69093c7fa29" gracePeriod=30
Oct 30 13:52:51.394473 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:51.394434    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:51.871972 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:51.871934    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:51.980936 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:51.980898    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:52.554101 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:52.554061    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:52.595198 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:52.595174    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q"
Oct 30 13:52:52.696976 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:52.696878    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/968756d1-6e26-4f17-aea1-f534a8b1330b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/968756d1-6e26-4f17-aea1-f534a8b1330b-maas-api-tls\") pod \"968756d1-6e26-4f17-aea1-f534a8b1330b\" (UID: \"968756d1-6e26-4f17-aea1-f534a8b1330b\") "
Oct 30 13:52:52.696976 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:52.696943    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/968756d1-6e26-4f17-aea1-f534a8b1330b-kube-api-access-tv8z2\" (UniqueName: \"kubernetes.io/projected/968756d1-6e26-4f17-aea1-f534a8b1330b-kube-api-access-tv8z2\") pod \"968756d1-6e26-4f17-aea1-f534a8b1330b\" (UID: \"968756d1-6e26-4f17-aea1-f534a8b1330b\") "
Oct 30 13:52:52.697248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:52.697027    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/968756d1-6e26-4f17-aea1-f534a8b1330b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/968756d1-6e26-4f17-aea1-f534a8b1330b-openshift-service-ca\") pod \"968756d1-6e26-4f17-aea1-f534a8b1330b\" (UID: \"968756d1-6e26-4f17-aea1-f534a8b1330b\") "
Oct 30 13:52:52.697248 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:52.697093    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/968756d1-6e26-4f17-aea1-f534a8b1330b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/968756d1-6e26-4f17-aea1-f534a8b1330b-metrics-certs\") pod \"968756d1-6e26-4f17-aea1-f534a8b1330b\" (UID: \"968756d1-6e26-4f17-aea1-f534a8b1330b\") "
Oct 30 13:52:52.697463 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:52.697434    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/968756d1-6e26-4f17-aea1-f534a8b1330b-openshift-service-ca" pod "968756d1-6e26-4f17-aea1-f534a8b1330b" (UID: "968756d1-6e26-4f17-aea1-f534a8b1330b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:52:52.699137 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:52.699108    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/968756d1-6e26-4f17-aea1-f534a8b1330b-maas-api-tls" pod "968756d1-6e26-4f17-aea1-f534a8b1330b" (UID: "968756d1-6e26-4f17-aea1-f534a8b1330b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:52:52.699341 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:52.699159    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/968756d1-6e26-4f17-aea1-f534a8b1330b-metrics-certs" pod "968756d1-6e26-4f17-aea1-f534a8b1330b" (UID: "968756d1-6e26-4f17-aea1-f534a8b1330b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:52:52.699341 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:52.699199    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/968756d1-6e26-4f17-aea1-f534a8b1330b-kube-api-access-tv8z2" pod "968756d1-6e26-4f17-aea1-f534a8b1330b" (UID: "968756d1-6e26-4f17-aea1-f534a8b1330b"). InnerVolumeSpecName "kube-api-access-tv8z2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:52:52.777588 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:52.777550    4544 generic.go:356] "Generic (PLEG): container finished" podID="968756d1-6e26-4f17-aea1-f534a8b1330b" containerID="d8f1e25ac012b80e1256b82f3ca043b380749d7be561194046a6f69093c7fa29" exitCode=0
Oct 30 13:52:52.777782 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:52.777600    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q"
Oct 30 13:52:52.777782 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:52.777637    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q" event={"ID":"968756d1-6e26-4f17-aea1-f534a8b1330b","Type":"ContainerDied","Data":"d8f1e25ac012b80e1256b82f3ca043b380749d7be561194046a6f69093c7fa29"}
Oct 30 13:52:52.777782 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:52.777698    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q" event={"ID":"968756d1-6e26-4f17-aea1-f534a8b1330b","Type":"ContainerDied","Data":"c7030bdf89506fee25f17505a0e45cc325e4d04806f7d4378876afcff24eb17f"}
Oct 30 13:52:52.777782 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:52.777723    4544 scope.go:122] "RemoveContainer" containerID="d8f1e25ac012b80e1256b82f3ca043b380749d7be561194046a6f69093c7fa29"
Oct 30 13:52:52.787428 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:52.787408    4544 scope.go:122] "RemoveContainer" containerID="d8f1e25ac012b80e1256b82f3ca043b380749d7be561194046a6f69093c7fa29"
Oct 30 13:52:52.787770 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:52:52.787746    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8f1e25ac012b80e1256b82f3ca043b380749d7be561194046a6f69093c7fa29\": container with ID starting with d8f1e25ac012b80e1256b82f3ca043b380749d7be561194046a6f69093c7fa29 not found: ID does not exist" containerID="d8f1e25ac012b80e1256b82f3ca043b380749d7be561194046a6f69093c7fa29"
Oct 30 13:52:52.787823 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:52.787778    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8f1e25ac012b80e1256b82f3ca043b380749d7be561194046a6f69093c7fa29"} err="failed to get container status \"d8f1e25ac012b80e1256b82f3ca043b380749d7be561194046a6f69093c7fa29\": rpc error: code = NotFound desc = could not find container \"d8f1e25ac012b80e1256b82f3ca043b380749d7be561194046a6f69093c7fa29\": container with ID starting with d8f1e25ac012b80e1256b82f3ca043b380749d7be561194046a6f69093c7fa29 not found: ID does not exist"
Oct 30 13:52:52.798638 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:52.798610    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv8z2\" (UniqueName: \"kubernetes.io/projected/968756d1-6e26-4f17-aea1-f534a8b1330b-kube-api-access-tv8z2\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:52:52.798638 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:52.798637    4544 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/968756d1-6e26-4f17-aea1-f534a8b1330b-openshift-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:52:52.798793 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:52.798647    4544 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/968756d1-6e26-4f17-aea1-f534a8b1330b-metrics-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:52:52.798793 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:52.798659    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/968756d1-6e26-4f17-aea1-f534a8b1330b-maas-api-tls\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:52:52.827809 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:52.827777    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q"]
Oct 30 13:52:52.832819 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:52.832792    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bf43af-66fff98859-gjg6q"]
Oct 30 13:52:53.084968 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:53.084933    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="968756d1-6e26-4f17-aea1-f534a8b1330b" path="/var/lib/kubelet/pods/968756d1-6e26-4f17-aea1-f534a8b1330b/volumes"
Oct 30 13:52:59.006946 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:52:59.006903    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:52:59.050906 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:59.050873    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:59.182669 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:59.182632    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:59.413935 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:59.413904    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:52:59.996007 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:52:59.995965    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:53:00.285188 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:00.285085    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:53:04.063041 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:04.063001    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:53:15.818868 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:15.818837    4544 scope.go:122] "RemoveContainer" containerID="60df2f8573c538cbf6f695db502a8415207f53c4ef96bc94086b083fe7b25e10"
Oct 30 13:53:15.827497 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:15.827475    4544 scope.go:122] "RemoveContainer" containerID="29b963f73c49aca65547d01f31a028404a3116e500cc21d99097aa1c583d54f6"
Oct 30 13:53:15.836006 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:15.835983    4544 scope.go:122] "RemoveContainer" containerID="904c33f8901ba551fc1114948cdbaaa610ad009c0e22acdcf50aa9743735202f"
Oct 30 13:53:18.172737 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:18.172643    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:53:20.996798 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:20.996757    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:53:21.344536 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:21.344493    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:53:22.223754 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:22.223717    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:53:27.329245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:27.321207    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:53:29.006968 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:53:29.006923    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:53:37.775169 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:37.775132    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-q9r5z"]
Oct 30 13:53:37.775686 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:37.775551    4544 cpu_manager.go:414] "containerMap: removing container" podUID="968756d1-6e26-4f17-aea1-f534a8b1330b" containerName="maas-api"
Oct 30 13:53:37.775686 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:37.775562    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="968756d1-6e26-4f17-aea1-f534a8b1330b" containerName="maas-api"
Oct 30 13:53:37.775686 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:37.775641    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="968756d1-6e26-4f17-aea1-f534a8b1330b" containerName="maas-api"
Oct 30 13:53:37.780208 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:37.780186    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-q9r5z"
Oct 30 13:53:37.800000 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:37.799966    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-q9r5z"]
Oct 30 13:53:37.809962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:37.809927    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:53:37.810124 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:37.810046    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:53:37.897736 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:37.897698    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbm7x\" (UniqueName: \"kubernetes.io/projected/c0047dab-616a-4c43-bc58-83e8ef2d04a3-kube-api-access-lbm7x\") pod \"maas-api-revoke-sub-452b6d7b-q9r5z\" (UID: \"c0047dab-616a-4c43-bc58-83e8ef2d04a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-q9r5z"
Oct 30 13:53:37.897952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:37.897746    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0047dab-616a-4c43-bc58-83e8ef2d04a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-q9r5z\" (UID: \"c0047dab-616a-4c43-bc58-83e8ef2d04a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-q9r5z"
Oct 30 13:53:37.897952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:37.897769    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0047dab-616a-4c43-bc58-83e8ef2d04a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-q9r5z\" (UID: \"c0047dab-616a-4c43-bc58-83e8ef2d04a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-q9r5z"
Oct 30 13:53:37.998863 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:37.998821    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbm7x\" (UniqueName: \"kubernetes.io/projected/c0047dab-616a-4c43-bc58-83e8ef2d04a3-kube-api-access-lbm7x\") pod \"maas-api-revoke-sub-452b6d7b-q9r5z\" (UID: \"c0047dab-616a-4c43-bc58-83e8ef2d04a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-q9r5z"
Oct 30 13:53:37.999081 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:37.998875    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0047dab-616a-4c43-bc58-83e8ef2d04a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-q9r5z\" (UID: \"c0047dab-616a-4c43-bc58-83e8ef2d04a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-q9r5z"
Oct 30 13:53:37.999081 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:37.998966    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0047dab-616a-4c43-bc58-83e8ef2d04a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-q9r5z\" (UID: \"c0047dab-616a-4c43-bc58-83e8ef2d04a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-q9r5z"
Oct 30 13:53:37.999677 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:37.999645    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0047dab-616a-4c43-bc58-83e8ef2d04a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-q9r5z\" (UID: \"c0047dab-616a-4c43-bc58-83e8ef2d04a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-q9r5z"
Oct 30 13:53:38.001399 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:38.001375    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0047dab-616a-4c43-bc58-83e8ef2d04a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-q9r5z\" (UID: \"c0047dab-616a-4c43-bc58-83e8ef2d04a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-q9r5z"
Oct 30 13:53:38.022545 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:38.022506    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbm7x\" (UniqueName: \"kubernetes.io/projected/c0047dab-616a-4c43-bc58-83e8ef2d04a3-kube-api-access-lbm7x\") pod \"maas-api-revoke-sub-452b6d7b-q9r5z\" (UID: \"c0047dab-616a-4c43-bc58-83e8ef2d04a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-q9r5z"
Oct 30 13:53:38.093834 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:38.093788    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-q9r5z"
Oct 30 13:53:38.394595 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:38.394510    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-q9r5z"]
Oct 30 13:53:38.398587 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:53:38.398559    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0047dab_616a_4c43_bc58_83e8ef2d04a3.slice/crio-ddbd4de8ee8d175fa3609e731569ed16cfa51ca4210577d77662328217d101c0 WatchSource:0}: Error finding container ddbd4de8ee8d175fa3609e731569ed16cfa51ca4210577d77662328217d101c0: Status 404 returned error can't find the container with id ddbd4de8ee8d175fa3609e731569ed16cfa51ca4210577d77662328217d101c0
Oct 30 13:53:39.073042 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:39.073007    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-q9r5z" event={"ID":"c0047dab-616a-4c43-bc58-83e8ef2d04a3","Type":"ContainerStarted","Data":"e60285d9ce10fcebb6c8644b0080c38147375e725dc11102af3f4d7be35361ba"}
Oct 30 13:53:39.073506 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:39.073076    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-q9r5z" event={"ID":"c0047dab-616a-4c43-bc58-83e8ef2d04a3","Type":"ContainerStarted","Data":"ddbd4de8ee8d175fa3609e731569ed16cfa51ca4210577d77662328217d101c0"}
Oct 30 13:53:39.115918 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:39.115852    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-q9r5z" podStartSLOduration=2.1158287590000002 podStartE2EDuration="2.115828759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:53:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:53:39.113725884 +0000 UTC m=+4114.160859837" watchObservedRunningTime="2026-10-30 13:53:39.115828759 +0000 UTC m=+4114.162962716"
Oct 30 13:53:40.091743 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:40.091702    4544 generic.go:356] "Generic (PLEG): container finished" podID="c0047dab-616a-4c43-bc58-83e8ef2d04a3" containerID="e60285d9ce10fcebb6c8644b0080c38147375e725dc11102af3f4d7be35361ba" exitCode=0
Oct 30 13:53:40.092279 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:40.091797    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-q9r5z" event={"ID":"c0047dab-616a-4c43-bc58-83e8ef2d04a3","Type":"ContainerDied","Data":"e60285d9ce10fcebb6c8644b0080c38147375e725dc11102af3f4d7be35361ba"}
Oct 30 13:53:42.040810 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:42.040437    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-q9r5z"
Oct 30 13:53:42.181552 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:42.180564    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0047dab-616a-4c43-bc58-83e8ef2d04a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0047dab-616a-4c43-bc58-83e8ef2d04a3-maas-api-xks-ca\") pod \"c0047dab-616a-4c43-bc58-83e8ef2d04a3\" (UID: \"c0047dab-616a-4c43-bc58-83e8ef2d04a3\") "
Oct 30 13:53:42.181552 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:42.180643    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0047dab-616a-4c43-bc58-83e8ef2d04a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0047dab-616a-4c43-bc58-83e8ef2d04a3-maas-api-openshift-ca\") pod \"c0047dab-616a-4c43-bc58-83e8ef2d04a3\" (UID: \"c0047dab-616a-4c43-bc58-83e8ef2d04a3\") "
Oct 30 13:53:42.181552 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:42.180759    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0047dab-616a-4c43-bc58-83e8ef2d04a3-kube-api-access-lbm7x\" (UniqueName: \"kubernetes.io/projected/c0047dab-616a-4c43-bc58-83e8ef2d04a3-kube-api-access-lbm7x\") pod \"c0047dab-616a-4c43-bc58-83e8ef2d04a3\" (UID: \"c0047dab-616a-4c43-bc58-83e8ef2d04a3\") "
Oct 30 13:53:42.191148 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:42.190765    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0047dab-616a-4c43-bc58-83e8ef2d04a3-kube-api-access-lbm7x" pod "c0047dab-616a-4c43-bc58-83e8ef2d04a3" (UID: "c0047dab-616a-4c43-bc58-83e8ef2d04a3"). InnerVolumeSpecName "kube-api-access-lbm7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:53:42.204244 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:42.199992    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0047dab-616a-4c43-bc58-83e8ef2d04a3-maas-api-xks-ca" pod "c0047dab-616a-4c43-bc58-83e8ef2d04a3" (UID: "c0047dab-616a-4c43-bc58-83e8ef2d04a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:53:42.206350 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:42.204604    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0047dab-616a-4c43-bc58-83e8ef2d04a3-maas-api-openshift-ca" pod "c0047dab-616a-4c43-bc58-83e8ef2d04a3" (UID: "c0047dab-616a-4c43-bc58-83e8ef2d04a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:53:42.337519 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:42.334566    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-q9r5z" event={"ID":"c0047dab-616a-4c43-bc58-83e8ef2d04a3","Type":"ContainerDied","Data":"ddbd4de8ee8d175fa3609e731569ed16cfa51ca4210577d77662328217d101c0"}
Oct 30 13:53:42.337519 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:42.334635    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddbd4de8ee8d175fa3609e731569ed16cfa51ca4210577d77662328217d101c0"
Oct 30 13:53:42.337519 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:42.334709    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-q9r5z"
Oct 30 13:53:42.395526 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:42.339172    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0047dab-616a-4c43-bc58-83e8ef2d04a3-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:53:42.395909 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:42.395874    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbm7x\" (UniqueName: \"kubernetes.io/projected/c0047dab-616a-4c43-bc58-83e8ef2d04a3-kube-api-access-lbm7x\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:53:42.396077 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:42.396058    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0047dab-616a-4c43-bc58-83e8ef2d04a3-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:53:49.075971 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:49.075938    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:53:53.106245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:53.105983    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:53:53.223407 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:53.223362    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:53:53.828245 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:53.816705    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:53:55.280759 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:55.280711    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jpwjf"]
Oct 30 13:53:55.285239 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:55.281933    4544 cpu_manager.go:414] "containerMap: removing container" podUID="c0047dab-616a-4c43-bc58-83e8ef2d04a3" containerName="revoke-keys"
Oct 30 13:53:55.301360 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:55.301322    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0047dab-616a-4c43-bc58-83e8ef2d04a3" containerName="revoke-keys"
Oct 30 13:53:55.301806 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:55.301771    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0047dab-616a-4c43-bc58-83e8ef2d04a3" containerName="revoke-keys"
Oct 30 13:53:55.314516 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:55.314485    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jpwjf"]
Oct 30 13:53:55.321256 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:55.314838    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jpwjf"
Oct 30 13:53:55.412608 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:55.411547    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:53:55.412608 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:55.411639    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:53:55.480064 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:55.479093    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afed0f2e-0ee6-46a8-83c3-426bbdba58a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-jpwjf\" (UID: \"afed0f2e-0ee6-46a8-83c3-426bbdba58a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jpwjf"
Oct 30 13:53:55.480064 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:55.479183    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afed0f2e-0ee6-46a8-83c3-426bbdba58a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-jpwjf\" (UID: \"afed0f2e-0ee6-46a8-83c3-426bbdba58a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jpwjf"
Oct 30 13:53:55.480064 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:55.479296    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk4cn\" (UniqueName: \"kubernetes.io/projected/afed0f2e-0ee6-46a8-83c3-426bbdba58a7-kube-api-access-kk4cn\") pod \"maas-api-revoke-sub-3e31a506-jpwjf\" (UID: \"afed0f2e-0ee6-46a8-83c3-426bbdba58a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jpwjf"
Oct 30 13:53:55.592393 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:55.580821    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk4cn\" (UniqueName: \"kubernetes.io/projected/afed0f2e-0ee6-46a8-83c3-426bbdba58a7-kube-api-access-kk4cn\") pod \"maas-api-revoke-sub-3e31a506-jpwjf\" (UID: \"afed0f2e-0ee6-46a8-83c3-426bbdba58a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jpwjf"
Oct 30 13:53:55.592393 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:55.580908    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afed0f2e-0ee6-46a8-83c3-426bbdba58a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-jpwjf\" (UID: \"afed0f2e-0ee6-46a8-83c3-426bbdba58a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jpwjf"
Oct 30 13:53:55.592393 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:55.580978    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afed0f2e-0ee6-46a8-83c3-426bbdba58a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-jpwjf\" (UID: \"afed0f2e-0ee6-46a8-83c3-426bbdba58a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jpwjf"
Oct 30 13:53:55.605447 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:55.601142    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afed0f2e-0ee6-46a8-83c3-426bbdba58a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-jpwjf\" (UID: \"afed0f2e-0ee6-46a8-83c3-426bbdba58a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jpwjf"
Oct 30 13:53:55.635932 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:55.606351    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afed0f2e-0ee6-46a8-83c3-426bbdba58a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-jpwjf\" (UID: \"afed0f2e-0ee6-46a8-83c3-426bbdba58a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jpwjf"
Oct 30 13:53:55.681259 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:55.669941    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk4cn\" (UniqueName: \"kubernetes.io/projected/afed0f2e-0ee6-46a8-83c3-426bbdba58a7-kube-api-access-kk4cn\") pod \"maas-api-revoke-sub-3e31a506-jpwjf\" (UID: \"afed0f2e-0ee6-46a8-83c3-426bbdba58a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jpwjf"
Oct 30 13:53:55.781734 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:55.781455    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jpwjf"
Oct 30 13:53:56.661260 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:56.642078    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jpwjf"]
Oct 30 13:53:57.793413 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:57.793366    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jpwjf" event={"ID":"afed0f2e-0ee6-46a8-83c3-426bbdba58a7","Type":"ContainerStarted","Data":"c20e497cb118aa1ac2e09972b40af0c049e49fdb69151ccb84c807f11757a2fb"}
Oct 30 13:53:57.794018 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:57.793441    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jpwjf" event={"ID":"afed0f2e-0ee6-46a8-83c3-426bbdba58a7","Type":"ContainerStarted","Data":"1b86deb349a74e46eab052300044f3b7f676ba27cfc0b01d8e67cdb5f6fb112c"}
Oct 30 13:53:57.838066 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:57.837983    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jpwjf" podStartSLOduration=2.837961159 podStartE2EDuration="2.837961159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:53:57.834738744 +0000 UTC m=+4132.881872698" watchObservedRunningTime="2026-10-30 13:53:57.837961159 +0000 UTC m=+4132.885095114"
Oct 30 13:53:58.812097 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:58.812060    4544 generic.go:356] "Generic (PLEG): container finished" podID="afed0f2e-0ee6-46a8-83c3-426bbdba58a7" containerID="c20e497cb118aa1ac2e09972b40af0c049e49fdb69151ccb84c807f11757a2fb" exitCode=0
Oct 30 13:53:58.812961 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:53:58.812278    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jpwjf" event={"ID":"afed0f2e-0ee6-46a8-83c3-426bbdba58a7","Type":"ContainerDied","Data":"c20e497cb118aa1ac2e09972b40af0c049e49fdb69151ccb84c807f11757a2fb"}
Oct 30 13:53:59.244091 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:53:59.244041    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:54:00.294964 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:00.294920    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:54:00.497303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:00.496980    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jpwjf"
Oct 30 13:54:00.530860 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:00.530816    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afed0f2e-0ee6-46a8-83c3-426bbdba58a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afed0f2e-0ee6-46a8-83c3-426bbdba58a7-maas-api-xks-ca\") pod \"afed0f2e-0ee6-46a8-83c3-426bbdba58a7\" (UID: \"afed0f2e-0ee6-46a8-83c3-426bbdba58a7\") "
Oct 30 13:54:00.535374 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:00.535308    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afed0f2e-0ee6-46a8-83c3-426bbdba58a7-maas-api-xks-ca" pod "afed0f2e-0ee6-46a8-83c3-426bbdba58a7" (UID: "afed0f2e-0ee6-46a8-83c3-426bbdba58a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:54:00.635737 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:00.635688    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afed0f2e-0ee6-46a8-83c3-426bbdba58a7-kube-api-access-kk4cn\" (UniqueName: \"kubernetes.io/projected/afed0f2e-0ee6-46a8-83c3-426bbdba58a7-kube-api-access-kk4cn\") pod \"afed0f2e-0ee6-46a8-83c3-426bbdba58a7\" (UID: \"afed0f2e-0ee6-46a8-83c3-426bbdba58a7\") "
Oct 30 13:54:00.635953 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:00.635914    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afed0f2e-0ee6-46a8-83c3-426bbdba58a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afed0f2e-0ee6-46a8-83c3-426bbdba58a7-maas-api-openshift-ca\") pod \"afed0f2e-0ee6-46a8-83c3-426bbdba58a7\" (UID: \"afed0f2e-0ee6-46a8-83c3-426bbdba58a7\") "
Oct 30 13:54:00.636300 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:00.636272    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afed0f2e-0ee6-46a8-83c3-426bbdba58a7-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:54:00.636300 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:00.636291    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afed0f2e-0ee6-46a8-83c3-426bbdba58a7-maas-api-openshift-ca" pod "afed0f2e-0ee6-46a8-83c3-426bbdba58a7" (UID: "afed0f2e-0ee6-46a8-83c3-426bbdba58a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:54:00.637908 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:00.637875    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afed0f2e-0ee6-46a8-83c3-426bbdba58a7-kube-api-access-kk4cn" pod "afed0f2e-0ee6-46a8-83c3-426bbdba58a7" (UID: "afed0f2e-0ee6-46a8-83c3-426bbdba58a7"). InnerVolumeSpecName "kube-api-access-kk4cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:00.736834 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:00.736729    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afed0f2e-0ee6-46a8-83c3-426bbdba58a7-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:54:00.736834 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:00.736770    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk4cn\" (UniqueName: \"kubernetes.io/projected/afed0f2e-0ee6-46a8-83c3-426bbdba58a7-kube-api-access-kk4cn\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:54:00.870211 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:00.870172    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jpwjf" event={"ID":"afed0f2e-0ee6-46a8-83c3-426bbdba58a7","Type":"ContainerDied","Data":"1b86deb349a74e46eab052300044f3b7f676ba27cfc0b01d8e67cdb5f6fb112c"}
Oct 30 13:54:00.870211 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:00.870213    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jpwjf"
Oct 30 13:54:00.870501 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:00.870271    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b86deb349a74e46eab052300044f3b7f676ba27cfc0b01d8e67cdb5f6fb112c"
Oct 30 13:54:15.870322 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:15.870284    4544 scope.go:122] "RemoveContainer" containerID="989b1d7839fddf3442efc7fb12e6055b6124c7d644a2e77823dc2b976b5d1e36"
Oct 30 13:54:15.878988 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:15.878963    4544 scope.go:122] "RemoveContainer" containerID="b0dc83f9f161bf5f018b04064b2ab5f4025a66731305f98020a8c8a82a9a31a8"
Oct 30 13:54:15.886890 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:15.886866    4544 scope.go:122] "RemoveContainer" containerID="10e5d3c768fe0360acf0c36235b1259c8b55f99ddfed74c55262d15f29e6f7a3"
Oct 30 13:54:17.985528 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:17.985491    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c32af363-tn4j2"]
Oct 30 13:54:17.985964 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:17.985939    4544 cpu_manager.go:414] "containerMap: removing container" podUID="afed0f2e-0ee6-46a8-83c3-426bbdba58a7" containerName="revoke-keys"
Oct 30 13:54:17.985964 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:17.985952    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afed0f2e-0ee6-46a8-83c3-426bbdba58a7" containerName="revoke-keys"
Oct 30 13:54:17.986062 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:17.986050    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="afed0f2e-0ee6-46a8-83c3-426bbdba58a7" containerName="revoke-keys"
Oct 30 13:54:17.990797 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:17.990774    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32af363-tn4j2"
Oct 30 13:54:18.010618 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:18.010580    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c32af363-tn4j2"]
Oct 30 13:54:18.019749 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:18.019716    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:54:18.019937 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:18.019849    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:54:18.074124 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:18.074086    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83342704-e62b-4c3c-b896-033307b1e58c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c32af363-tn4j2\" (UID: \"83342704-e62b-4c3c-b896-033307b1e58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32af363-tn4j2"
Oct 30 13:54:18.074379 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:18.074161    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cfnb\" (UniqueName: \"kubernetes.io/projected/83342704-e62b-4c3c-b896-033307b1e58c-kube-api-access-2cfnb\") pod \"maas-api-revoke-sub-c32af363-tn4j2\" (UID: \"83342704-e62b-4c3c-b896-033307b1e58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32af363-tn4j2"
Oct 30 13:54:18.074379 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:18.074359    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83342704-e62b-4c3c-b896-033307b1e58c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c32af363-tn4j2\" (UID: \"83342704-e62b-4c3c-b896-033307b1e58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32af363-tn4j2"
Oct 30 13:54:18.175404 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:18.175359    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83342704-e62b-4c3c-b896-033307b1e58c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c32af363-tn4j2\" (UID: \"83342704-e62b-4c3c-b896-033307b1e58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32af363-tn4j2"
Oct 30 13:54:18.175625 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:18.175439    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cfnb\" (UniqueName: \"kubernetes.io/projected/83342704-e62b-4c3c-b896-033307b1e58c-kube-api-access-2cfnb\") pod \"maas-api-revoke-sub-c32af363-tn4j2\" (UID: \"83342704-e62b-4c3c-b896-033307b1e58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32af363-tn4j2"
Oct 30 13:54:18.175625 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:18.175488    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83342704-e62b-4c3c-b896-033307b1e58c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c32af363-tn4j2\" (UID: \"83342704-e62b-4c3c-b896-033307b1e58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32af363-tn4j2"
Oct 30 13:54:18.176116 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:18.176077    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83342704-e62b-4c3c-b896-033307b1e58c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c32af363-tn4j2\" (UID: \"83342704-e62b-4c3c-b896-033307b1e58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32af363-tn4j2"
Oct 30 13:54:18.177946 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:18.177922    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83342704-e62b-4c3c-b896-033307b1e58c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c32af363-tn4j2\" (UID: \"83342704-e62b-4c3c-b896-033307b1e58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32af363-tn4j2"
Oct 30 13:54:18.199323 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:18.199284    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cfnb\" (UniqueName: \"kubernetes.io/projected/83342704-e62b-4c3c-b896-033307b1e58c-kube-api-access-2cfnb\") pod \"maas-api-revoke-sub-c32af363-tn4j2\" (UID: \"83342704-e62b-4c3c-b896-033307b1e58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32af363-tn4j2"
Oct 30 13:54:18.303848 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:18.303744    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32af363-tn4j2"
Oct 30 13:54:18.604872 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:18.604832    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c32af363-tn4j2"]
Oct 30 13:54:18.609097 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:54:18.609066    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83342704_e62b_4c3c_b896_033307b1e58c.slice/crio-b4b923d5ccd0c0cd5b34ed3447ba8d613a68180a73861375bb1c4d7f318710fe WatchSource:0}: Error finding container b4b923d5ccd0c0cd5b34ed3447ba8d613a68180a73861375bb1c4d7f318710fe: Status 404 returned error can't find the container with id b4b923d5ccd0c0cd5b34ed3447ba8d613a68180a73861375bb1c4d7f318710fe
Oct 30 13:54:19.000733 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:19.000693    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32af363-tn4j2" event={"ID":"83342704-e62b-4c3c-b896-033307b1e58c","Type":"ContainerStarted","Data":"8efd6f5f0b3bf0d7a579964d43d3d8cf0eb45bf1c5289b78901456fb96a6d502"}
Oct 30 13:54:19.001166 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:19.000753    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32af363-tn4j2" event={"ID":"83342704-e62b-4c3c-b896-033307b1e58c","Type":"ContainerStarted","Data":"b4b923d5ccd0c0cd5b34ed3447ba8d613a68180a73861375bb1c4d7f318710fe"}
Oct 30 13:54:19.043608 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:19.043542    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32af363-tn4j2" podStartSLOduration=2.043520438 podStartE2EDuration="2.043520438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:54:19.041059968 +0000 UTC m=+4154.088193922" watchObservedRunningTime="2026-10-30 13:54:19.043520438 +0000 UTC m=+4154.090654392"
Oct 30 13:54:20.007456 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:20.007419    4544 generic.go:356] "Generic (PLEG): container finished" podID="83342704-e62b-4c3c-b896-033307b1e58c" containerID="8efd6f5f0b3bf0d7a579964d43d3d8cf0eb45bf1c5289b78901456fb96a6d502" exitCode=0
Oct 30 13:54:20.007878 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:20.007504    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32af363-tn4j2" event={"ID":"83342704-e62b-4c3c-b896-033307b1e58c","Type":"ContainerDied","Data":"8efd6f5f0b3bf0d7a579964d43d3d8cf0eb45bf1c5289b78901456fb96a6d502"}
Oct 30 13:54:21.156100 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:21.156067    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32af363-tn4j2"
Oct 30 13:54:21.308063 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:21.307951    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83342704-e62b-4c3c-b896-033307b1e58c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83342704-e62b-4c3c-b896-033307b1e58c-maas-api-xks-ca\") pod \"83342704-e62b-4c3c-b896-033307b1e58c\" (UID: \"83342704-e62b-4c3c-b896-033307b1e58c\") "
Oct 30 13:54:21.308063 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:21.308016    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83342704-e62b-4c3c-b896-033307b1e58c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83342704-e62b-4c3c-b896-033307b1e58c-maas-api-openshift-ca\") pod \"83342704-e62b-4c3c-b896-033307b1e58c\" (UID: \"83342704-e62b-4c3c-b896-033307b1e58c\") "
Oct 30 13:54:21.308408 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:21.308148    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83342704-e62b-4c3c-b896-033307b1e58c-kube-api-access-2cfnb\" (UniqueName: \"kubernetes.io/projected/83342704-e62b-4c3c-b896-033307b1e58c-kube-api-access-2cfnb\") pod \"83342704-e62b-4c3c-b896-033307b1e58c\" (UID: \"83342704-e62b-4c3c-b896-033307b1e58c\") "
Oct 30 13:54:21.308482 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:21.308453    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83342704-e62b-4c3c-b896-033307b1e58c-maas-api-openshift-ca" pod "83342704-e62b-4c3c-b896-033307b1e58c" (UID: "83342704-e62b-4c3c-b896-033307b1e58c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:54:21.310300 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:21.310267    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83342704-e62b-4c3c-b896-033307b1e58c-maas-api-xks-ca" pod "83342704-e62b-4c3c-b896-033307b1e58c" (UID: "83342704-e62b-4c3c-b896-033307b1e58c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:54:21.310300 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:21.310291    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83342704-e62b-4c3c-b896-033307b1e58c-kube-api-access-2cfnb" pod "83342704-e62b-4c3c-b896-033307b1e58c" (UID: "83342704-e62b-4c3c-b896-033307b1e58c"). InnerVolumeSpecName "kube-api-access-2cfnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:21.408907 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:21.408865    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83342704-e62b-4c3c-b896-033307b1e58c-maas-api-xks-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:54:21.408907 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:21.408901    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83342704-e62b-4c3c-b896-033307b1e58c-maas-api-openshift-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:54:21.409148 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:21.408920    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cfnb\" (UniqueName: \"kubernetes.io/projected/83342704-e62b-4c3c-b896-033307b1e58c-kube-api-access-2cfnb\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:54:22.020770 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:22.020733    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32af363-tn4j2"
Oct 30 13:54:22.020770 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:22.020758    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32af363-tn4j2" event={"ID":"83342704-e62b-4c3c-b896-033307b1e58c","Type":"ContainerDied","Data":"b4b923d5ccd0c0cd5b34ed3447ba8d613a68180a73861375bb1c4d7f318710fe"}
Oct 30 13:54:22.021007 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:22.020806    4544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4b923d5ccd0c0cd5b34ed3447ba8d613a68180a73861375bb1c4d7f318710fe"
Oct 30 13:54:29.005788 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:54:29.005745    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:54:30.566363 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:30.566321    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-l4qbp"]
Oct 30 13:54:30.566777 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:30.566760    4544 cpu_manager.go:414] "containerMap: removing container" podUID="83342704-e62b-4c3c-b896-033307b1e58c" containerName="revoke-keys"
Oct 30 13:54:30.566777 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:30.566772    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83342704-e62b-4c3c-b896-033307b1e58c" containerName="revoke-keys"
Oct 30 13:54:30.566855 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:30.566845    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="83342704-e62b-4c3c-b896-033307b1e58c" containerName="revoke-keys"
Oct 30 13:54:30.570368 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:30.570346    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-l4qbp"
Oct 30 13:54:30.603972 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:30.603939    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 13:54:30.604258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:30.603999    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nb7kb\""
Oct 30 13:54:30.604258 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:30.604069    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 13:54:30.617656 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:30.617617    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-l4qbp"]
Oct 30 13:54:30.705158 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:30.705101    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxctd\" (UniqueName: \"kubernetes.io/projected/2b923499-fdc6-4b49-a7b6-309da9441981-kube-api-access-sxctd\") pod \"maas-api-5c48877c75-l4qbp\" (UID: \"2b923499-fdc6-4b49-a7b6-309da9441981\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-l4qbp"
Oct 30 13:54:30.705402 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:30.705251    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b923499-fdc6-4b49-a7b6-309da9441981-maas-api-tls\") pod \"maas-api-5c48877c75-l4qbp\" (UID: \"2b923499-fdc6-4b49-a7b6-309da9441981\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-l4qbp"
Oct 30 13:54:30.705402 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:30.705283    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b923499-fdc6-4b49-a7b6-309da9441981-metrics-certs\") pod \"maas-api-5c48877c75-l4qbp\" (UID: \"2b923499-fdc6-4b49-a7b6-309da9441981\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-l4qbp"
Oct 30 13:54:30.705402 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:30.705319    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b923499-fdc6-4b49-a7b6-309da9441981-openshift-service-ca\") pod \"maas-api-5c48877c75-l4qbp\" (UID: \"2b923499-fdc6-4b49-a7b6-309da9441981\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-l4qbp"
Oct 30 13:54:30.806492 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:30.806430    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b923499-fdc6-4b49-a7b6-309da9441981-metrics-certs\") pod \"maas-api-5c48877c75-l4qbp\" (UID: \"2b923499-fdc6-4b49-a7b6-309da9441981\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-l4qbp"
Oct 30 13:54:30.806779 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:30.806511    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b923499-fdc6-4b49-a7b6-309da9441981-openshift-service-ca\") pod \"maas-api-5c48877c75-l4qbp\" (UID: \"2b923499-fdc6-4b49-a7b6-309da9441981\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-l4qbp"
Oct 30 13:54:30.806779 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:30.806644    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxctd\" (UniqueName: \"kubernetes.io/projected/2b923499-fdc6-4b49-a7b6-309da9441981-kube-api-access-sxctd\") pod \"maas-api-5c48877c75-l4qbp\" (UID: \"2b923499-fdc6-4b49-a7b6-309da9441981\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-l4qbp"
Oct 30 13:54:30.806779 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:30.806692    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b923499-fdc6-4b49-a7b6-309da9441981-maas-api-tls\") pod \"maas-api-5c48877c75-l4qbp\" (UID: \"2b923499-fdc6-4b49-a7b6-309da9441981\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-l4qbp"
Oct 30 13:54:30.807415 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:30.807374    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b923499-fdc6-4b49-a7b6-309da9441981-openshift-service-ca\") pod \"maas-api-5c48877c75-l4qbp\" (UID: \"2b923499-fdc6-4b49-a7b6-309da9441981\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-l4qbp"
Oct 30 13:54:30.809515 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:30.809487    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b923499-fdc6-4b49-a7b6-309da9441981-maas-api-tls\") pod \"maas-api-5c48877c75-l4qbp\" (UID: \"2b923499-fdc6-4b49-a7b6-309da9441981\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-l4qbp"
Oct 30 13:54:30.809787 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:30.809754    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b923499-fdc6-4b49-a7b6-309da9441981-metrics-certs\") pod \"maas-api-5c48877c75-l4qbp\" (UID: \"2b923499-fdc6-4b49-a7b6-309da9441981\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-l4qbp"
Oct 30 13:54:30.829662 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:30.829625    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxctd\" (UniqueName: \"kubernetes.io/projected/2b923499-fdc6-4b49-a7b6-309da9441981-kube-api-access-sxctd\") pod \"maas-api-5c48877c75-l4qbp\" (UID: \"2b923499-fdc6-4b49-a7b6-309da9441981\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-l4qbp"
Oct 30 13:54:30.883654 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:30.883621    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-l4qbp"
Oct 30 13:54:31.187833 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:31.187752    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-l4qbp"]
Oct 30 13:54:31.191946 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:54:31.191917    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b923499_fdc6_4b49_a7b6_309da9441981.slice/crio-914b2d41dc235f8f6d2b69ab6e3a3584b219062cd0835f208afd047bd08b5665 WatchSource:0}: Error finding container 914b2d41dc235f8f6d2b69ab6e3a3584b219062cd0835f208afd047bd08b5665: Status 404 returned error can't find the container with id 914b2d41dc235f8f6d2b69ab6e3a3584b219062cd0835f208afd047bd08b5665
Oct 30 13:54:31.193333 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:31.193316    4544 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:54:32.071887 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:32.071786    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-l4qbp" event={"ID":"2b923499-fdc6-4b49-a7b6-309da9441981","Type":"ContainerStarted","Data":"7ef65b45f9848ece458401f788ac6827c58f4eb2d014539ee485589738740bf2"}
Oct 30 13:54:32.071887 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:32.071884    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-l4qbp" event={"ID":"2b923499-fdc6-4b49-a7b6-309da9441981","Type":"ContainerStarted","Data":"914b2d41dc235f8f6d2b69ab6e3a3584b219062cd0835f208afd047bd08b5665"}
Oct 30 13:54:32.072372 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:32.071915    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-l4qbp"
Oct 30 13:54:32.115142 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:32.115070    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-l4qbp" podStartSLOduration=1.716459452 podStartE2EDuration="2.115051334s" totalImagesPullingTime="398.591882ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:54:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:54:32.11338749 +0000 UTC m=+4167.160521443" watchObservedRunningTime="2026-10-30 13:54:32.115051334 +0000 UTC m=+4167.162185292"
Oct 30 13:54:38.083119 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:38.083079    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-l4qbp"
Oct 30 13:54:38.639551 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:38.639491    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-pmbjx"]
Oct 30 13:54:38.643686 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:38.643655    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-pmbjx"
Oct 30 13:54:38.664066 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:38.664029    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-pmbjx"]
Oct 30 13:54:38.794064 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:38.794007    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75af1b1d-2f4c-46b2-bde6-966faa6e2723-metrics-certs\") pod \"maas-api-595cc648c-pmbjx\" (UID: \"75af1b1d-2f4c-46b2-bde6-966faa6e2723\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-pmbjx"
Oct 30 13:54:38.794266 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:38.794171    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75af1b1d-2f4c-46b2-bde6-966faa6e2723-maas-api-tls\") pod \"maas-api-595cc648c-pmbjx\" (UID: \"75af1b1d-2f4c-46b2-bde6-966faa6e2723\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-pmbjx"
Oct 30 13:54:38.794266 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:38.794249    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwc87\" (UniqueName: \"kubernetes.io/projected/75af1b1d-2f4c-46b2-bde6-966faa6e2723-kube-api-access-rwc87\") pod \"maas-api-595cc648c-pmbjx\" (UID: \"75af1b1d-2f4c-46b2-bde6-966faa6e2723\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-pmbjx"
Oct 30 13:54:38.794418 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:38.794330    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75af1b1d-2f4c-46b2-bde6-966faa6e2723-openshift-service-ca\") pod \"maas-api-595cc648c-pmbjx\" (UID: \"75af1b1d-2f4c-46b2-bde6-966faa6e2723\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-pmbjx"
Oct 30 13:54:38.895452 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:38.895359    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75af1b1d-2f4c-46b2-bde6-966faa6e2723-maas-api-tls\") pod \"maas-api-595cc648c-pmbjx\" (UID: \"75af1b1d-2f4c-46b2-bde6-966faa6e2723\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-pmbjx"
Oct 30 13:54:38.895452 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:38.895411    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwc87\" (UniqueName: \"kubernetes.io/projected/75af1b1d-2f4c-46b2-bde6-966faa6e2723-kube-api-access-rwc87\") pod \"maas-api-595cc648c-pmbjx\" (UID: \"75af1b1d-2f4c-46b2-bde6-966faa6e2723\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-pmbjx"
Oct 30 13:54:38.895452 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:38.895436    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75af1b1d-2f4c-46b2-bde6-966faa6e2723-openshift-service-ca\") pod \"maas-api-595cc648c-pmbjx\" (UID: \"75af1b1d-2f4c-46b2-bde6-966faa6e2723\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-pmbjx"
Oct 30 13:54:38.895759 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:38.895508    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75af1b1d-2f4c-46b2-bde6-966faa6e2723-metrics-certs\") pod \"maas-api-595cc648c-pmbjx\" (UID: \"75af1b1d-2f4c-46b2-bde6-966faa6e2723\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-pmbjx"
Oct 30 13:54:38.896213 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:38.896187    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75af1b1d-2f4c-46b2-bde6-966faa6e2723-openshift-service-ca\") pod \"maas-api-595cc648c-pmbjx\" (UID: \"75af1b1d-2f4c-46b2-bde6-966faa6e2723\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-pmbjx"
Oct 30 13:54:38.898121 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:38.898098    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75af1b1d-2f4c-46b2-bde6-966faa6e2723-metrics-certs\") pod \"maas-api-595cc648c-pmbjx\" (UID: \"75af1b1d-2f4c-46b2-bde6-966faa6e2723\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-pmbjx"
Oct 30 13:54:38.898182 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:38.898145    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75af1b1d-2f4c-46b2-bde6-966faa6e2723-maas-api-tls\") pod \"maas-api-595cc648c-pmbjx\" (UID: \"75af1b1d-2f4c-46b2-bde6-966faa6e2723\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-pmbjx"
Oct 30 13:54:38.918962 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:38.918923    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwc87\" (UniqueName: \"kubernetes.io/projected/75af1b1d-2f4c-46b2-bde6-966faa6e2723-kube-api-access-rwc87\") pod \"maas-api-595cc648c-pmbjx\" (UID: \"75af1b1d-2f4c-46b2-bde6-966faa6e2723\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-pmbjx"
Oct 30 13:54:38.958831 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:38.958792    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-pmbjx"
Oct 30 13:54:39.034439 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:39.034149    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g6lnw"]
Oct 30 13:54:39.039071 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:39.039018    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g6lnw"]
Oct 30 13:54:39.084639 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:39.084604    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92c9f740-bed1-4e48-a5b9-121ca74df46f" path="/var/lib/kubelet/pods/92c9f740-bed1-4e48-a5b9-121ca74df46f/volumes"
Oct 30 13:54:39.274349 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:39.274262    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-pmbjx"]
Oct 30 13:54:39.277737 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:54:39.277705    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75af1b1d_2f4c_46b2_bde6_966faa6e2723.slice/crio-ed11dce00cff9fd0db47c5965e9292bb80f7882d2609e3e64b079bbc099bf4a2 WatchSource:0}: Error finding container ed11dce00cff9fd0db47c5965e9292bb80f7882d2609e3e64b079bbc099bf4a2: Status 404 returned error can't find the container with id ed11dce00cff9fd0db47c5965e9292bb80f7882d2609e3e64b079bbc099bf4a2
Oct 30 13:54:40.123364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:40.123314    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-pmbjx" event={"ID":"75af1b1d-2f4c-46b2-bde6-966faa6e2723","Type":"ContainerStarted","Data":"708bf129c71cba5a2ac95b35ceb30673a003fc8a94d15ca191767436c985aad0"}
Oct 30 13:54:40.123938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:40.123388    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-pmbjx" event={"ID":"75af1b1d-2f4c-46b2-bde6-966faa6e2723","Type":"ContainerStarted","Data":"ed11dce00cff9fd0db47c5965e9292bb80f7882d2609e3e64b079bbc099bf4a2"}
Oct 30 13:54:40.123938 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:40.123451    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-pmbjx"
Oct 30 13:54:40.170501 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:40.170432    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-pmbjx" podStartSLOduration=1.835290206 podStartE2EDuration="2.170408995s" totalImagesPullingTime="335.118789ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:54:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:54:40.166315592 +0000 UTC m=+4175.213449556" watchObservedRunningTime="2026-10-30 13:54:40.170408995 +0000 UTC m=+4175.217542949"
Oct 30 13:54:46.152993 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:46.152962    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-pmbjx"
Oct 30 13:54:46.238273 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:46.235857    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-l4qbp"]
Oct 30 13:54:46.238273 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:46.236140    4544 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-l4qbp" podUID="2b923499-fdc6-4b49-a7b6-309da9441981" containerName="maas-api" containerID="cri-o://7ef65b45f9848ece458401f788ac6827c58f4eb2d014539ee485589738740bf2" gracePeriod=30
Oct 30 13:54:46.511543 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:46.511514    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-l4qbp"
Oct 30 13:54:46.612848 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:46.612805    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b923499-fdc6-4b49-a7b6-309da9441981-kube-api-access-sxctd\" (UniqueName: \"kubernetes.io/projected/2b923499-fdc6-4b49-a7b6-309da9441981-kube-api-access-sxctd\") pod \"2b923499-fdc6-4b49-a7b6-309da9441981\" (UID: \"2b923499-fdc6-4b49-a7b6-309da9441981\") "
Oct 30 13:54:46.613067 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:46.612901    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b923499-fdc6-4b49-a7b6-309da9441981-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b923499-fdc6-4b49-a7b6-309da9441981-metrics-certs\") pod \"2b923499-fdc6-4b49-a7b6-309da9441981\" (UID: \"2b923499-fdc6-4b49-a7b6-309da9441981\") "
Oct 30 13:54:46.613067 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:46.612953    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b923499-fdc6-4b49-a7b6-309da9441981-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b923499-fdc6-4b49-a7b6-309da9441981-maas-api-tls\") pod \"2b923499-fdc6-4b49-a7b6-309da9441981\" (UID: \"2b923499-fdc6-4b49-a7b6-309da9441981\") "
Oct 30 13:54:46.613067 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:46.612972    4544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b923499-fdc6-4b49-a7b6-309da9441981-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b923499-fdc6-4b49-a7b6-309da9441981-openshift-service-ca\") pod \"2b923499-fdc6-4b49-a7b6-309da9441981\" (UID: \"2b923499-fdc6-4b49-a7b6-309da9441981\") "
Oct 30 13:54:46.613442 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:46.613419    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b923499-fdc6-4b49-a7b6-309da9441981-openshift-service-ca" pod "2b923499-fdc6-4b49-a7b6-309da9441981" (UID: "2b923499-fdc6-4b49-a7b6-309da9441981"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:54:46.615216 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:46.615185    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b923499-fdc6-4b49-a7b6-309da9441981-kube-api-access-sxctd" pod "2b923499-fdc6-4b49-a7b6-309da9441981" (UID: "2b923499-fdc6-4b49-a7b6-309da9441981"). InnerVolumeSpecName "kube-api-access-sxctd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:46.615346 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:46.615250    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b923499-fdc6-4b49-a7b6-309da9441981-maas-api-tls" pod "2b923499-fdc6-4b49-a7b6-309da9441981" (UID: "2b923499-fdc6-4b49-a7b6-309da9441981"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:54:46.615346 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:46.615293    4544 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b923499-fdc6-4b49-a7b6-309da9441981-metrics-certs" pod "2b923499-fdc6-4b49-a7b6-309da9441981" (UID: "2b923499-fdc6-4b49-a7b6-309da9441981"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:54:46.714269 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:46.714158    4544 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b923499-fdc6-4b49-a7b6-309da9441981-metrics-certs\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:54:46.714269 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:46.714194    4544 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b923499-fdc6-4b49-a7b6-309da9441981-maas-api-tls\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:54:46.714269 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:46.714205    4544 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b923499-fdc6-4b49-a7b6-309da9441981-openshift-service-ca\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:54:46.714269 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:46.714230    4544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxctd\" (UniqueName: \"kubernetes.io/projected/2b923499-fdc6-4b49-a7b6-309da9441981-kube-api-access-sxctd\") on node \"ip-10-0-143-38.ec2.internal\" DevicePath \"\""
Oct 30 13:54:47.191306 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:47.191270    4544 generic.go:356] "Generic (PLEG): container finished" podID="2b923499-fdc6-4b49-a7b6-309da9441981" containerID="7ef65b45f9848ece458401f788ac6827c58f4eb2d014539ee485589738740bf2" exitCode=0
Oct 30 13:54:47.191789 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:47.191312    4544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-l4qbp"
Oct 30 13:54:47.191789 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:47.191348    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-l4qbp" event={"ID":"2b923499-fdc6-4b49-a7b6-309da9441981","Type":"ContainerDied","Data":"7ef65b45f9848ece458401f788ac6827c58f4eb2d014539ee485589738740bf2"}
Oct 30 13:54:47.191789 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:47.191401    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-l4qbp" event={"ID":"2b923499-fdc6-4b49-a7b6-309da9441981","Type":"ContainerDied","Data":"914b2d41dc235f8f6d2b69ab6e3a3584b219062cd0835f208afd047bd08b5665"}
Oct 30 13:54:47.191789 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:47.191424    4544 scope.go:122] "RemoveContainer" containerID="7ef65b45f9848ece458401f788ac6827c58f4eb2d014539ee485589738740bf2"
Oct 30 13:54:47.200265 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:47.200248    4544 scope.go:122] "RemoveContainer" containerID="7ef65b45f9848ece458401f788ac6827c58f4eb2d014539ee485589738740bf2"
Oct 30 13:54:47.200531 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:54:47.200510    4544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ef65b45f9848ece458401f788ac6827c58f4eb2d014539ee485589738740bf2\": container with ID starting with 7ef65b45f9848ece458401f788ac6827c58f4eb2d014539ee485589738740bf2 not found: ID does not exist" containerID="7ef65b45f9848ece458401f788ac6827c58f4eb2d014539ee485589738740bf2"
Oct 30 13:54:47.200591 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:47.200538    4544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ef65b45f9848ece458401f788ac6827c58f4eb2d014539ee485589738740bf2"} err="failed to get container status \"7ef65b45f9848ece458401f788ac6827c58f4eb2d014539ee485589738740bf2\": rpc error: code = NotFound desc = could not find container \"7ef65b45f9848ece458401f788ac6827c58f4eb2d014539ee485589738740bf2\": container with ID starting with 7ef65b45f9848ece458401f788ac6827c58f4eb2d014539ee485589738740bf2 not found: ID does not exist"
Oct 30 13:54:47.255584 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:47.255549    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-l4qbp"]
Oct 30 13:54:47.267445 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:47.267404    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-l4qbp"]
Oct 30 13:54:49.096253 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:49.095289    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b923499-fdc6-4b49-a7b6-309da9441981" path="/var/lib/kubelet/pods/2b923499-fdc6-4b49-a7b6-309da9441981/volumes"
Oct 30 13:54:58.675297 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:58.675256    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:54:58.786505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:54:58.786471    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:54:59.006308 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:54:59.006197    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:55:00.363505 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:55:00.363468    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:55:29.136247 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:55:29.136163    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:55:40.845952 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:55:40.845912    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75d8bff59-8hz8s_9af0033c-76ac-4cc4-852f-3a3c190778bb/manager/0.log"
Oct 30 13:55:41.569731 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:55:41.569689    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-rvsjg_50683686-80a0-4425-8052-104dcf94a8aa/manager/0.log"
Oct 30 13:55:42.077020 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:55:42.076981    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c4494f88b-7l28c_36d7a70c-cbe6-481a-a8ba-7e2a8511f59d/copy-manifests/0.log"
Oct 30 13:55:42.115890 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:55:42.115851    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c4494f88b-7l28c_36d7a70c-cbe6-481a-a8ba-7e2a8511f59d/manager/0.log"
Oct 30 13:55:42.590149 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:55:42.590101    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-ssc54_509fe526-3fb6-4afa-807c-25028c1160b7/proxy/0.log"
Oct 30 13:55:43.428967 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:55:43.428925    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-pmbjx_75af1b1d-2f4c-46b2-bde6-966faa6e2723/maas-api/0.log"
Oct 30 13:55:43.957442 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:55:43.957397    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:55:44.247955 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:55:44.247867    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-jpwjf_afed0f2e-0ee6-46a8-83c3-426bbdba58a7/revoke-keys/0.log"
Oct 30 13:55:44.442729 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:55:44.442695    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-q9r5z_c0047dab-616a-4c43-bc58-83e8ef2d04a3/revoke-keys/0.log"
Oct 30 13:55:44.832303 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:55:44.832272    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c32af363-tn4j2_83342704-e62b-4c3c-b896-033307b1e58c/revoke-keys/0.log"
Oct 30 13:55:45.035899 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:55:45.035858    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-4z4z5_49d0fb91-0749-4365-9257-c14ba79f85d0/revoke-keys/0.log"
Oct 30 13:55:45.229013 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:55:45.228928    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-vx8q2_5a6b8940-4819-4258-801d-e38ba52c39f2/revoke-keys/0.log"
Oct 30 13:55:45.634166 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:55:45.634134    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-4l4tc_7472d739-738d-481d-9845-ff8959593b6a/postgres/0.log"
Oct 30 13:55:47.576130 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:55:47.576090    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-nlk4n_8714d903-8aa8-4b34-ab60-943c82d814c3/manager/0.log"
Oct 30 13:55:47.796364 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:55:47.796323    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-n6qxn_a7c27cba-095a-446a-bf7c-acab18ce8bca/manager/0.log"
Oct 30 13:55:48.658914 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:55:48.658878    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-689fp_ebca0355-cba2-4a9d-8a7c-81036e479ca8/manager/0.log"
Oct 30 13:55:49.688718 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:55:49.688681    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-dlbtt_1e8d89e6-3301-45d8-8e9d-4629f6bad974/discovery/0.log"
Oct 30 13:55:50.113726 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:55:50.113697    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cbd4cb4f6-fl8vc_3eb6d5b4-6fbb-4ae8-9359-833f745cf6c2/kube-auth-proxy/0.log"
Oct 30 13:55:50.530805 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:55:50.530726    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-52t9q_7e9c9db5-5873-40d8-94bb-d827c7698714/istio-proxy/0.log"
Oct 30 13:55:50.739134 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:55:50.739103    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-7grvt_c16d5364-17da-4fef-993d-9ac8a77f56a5/istio-proxy/0.log"
Oct 30 13:55:50.947157 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:55:50.947103    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-9cf5j_61006021-7c0a-4e04-bb5a-c8b2079b3a3e/istio-proxy/0.log"
Oct 30 13:55:51.365431 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:55:51.365397    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-fdnx5_bfb6ceaf-36c4-4a7e-8d4d-72efeecef675/istio-proxy/0.log"
Oct 30 13:55:52.413403 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:55:52.413375    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-8lgfp_097cd592-eba6-4113-9944-3cdbd57400f4/payload-pre-processing/0.log"
Oct 30 13:55:52.619407 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:55:52.619376    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-wrtq4_7568a93f-5fdb-444f-86e7-b600ab7c7a78/payload-processing/0.log"
Oct 30 13:55:53.878821 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:55:53.878789    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6_4a23bc66-d828-423f-a2a1-e803aef031d1/main/0.log"
Oct 30 13:55:54.076145 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:55:54.076114    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-bd6w2_c0251641-4f60-4eae-8970-dbc5f652936b/main/0.log"
Oct 30 13:55:54.686111 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:55:54.686072    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88_62b23320-6cd9-4cbb-b58e-cf8de8a931a8/main/0.log"
Oct 30 13:55:59.006095 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:55:59.006054    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:56:00.515199 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:00.515169    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:56:06.516458 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:06.516368    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75d8bff59-8hz8s_9af0033c-76ac-4cc4-852f-3a3c190778bb/manager/0.log"
Oct 30 13:56:07.189055 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:07.189019    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-rvsjg_50683686-80a0-4425-8052-104dcf94a8aa/manager/0.log"
Oct 30 13:56:07.683861 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:07.683830    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c4494f88b-7l28c_36d7a70c-cbe6-481a-a8ba-7e2a8511f59d/copy-manifests/0.log"
Oct 30 13:56:07.720184 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:07.720154    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c4494f88b-7l28c_36d7a70c-cbe6-481a-a8ba-7e2a8511f59d/manager/0.log"
Oct 30 13:56:08.209422 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:08.209379    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-ssc54_509fe526-3fb6-4afa-807c-25028c1160b7/proxy/0.log"
Oct 30 13:56:08.798361 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:08.798317    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-pmbjx_75af1b1d-2f4c-46b2-bde6-966faa6e2723/maas-api/0.log"
Oct 30 13:56:09.601658 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:09.601629    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-jpwjf_afed0f2e-0ee6-46a8-83c3-426bbdba58a7/revoke-keys/0.log"
Oct 30 13:56:09.792261 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:09.792203    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-q9r5z_c0047dab-616a-4c43-bc58-83e8ef2d04a3/revoke-keys/0.log"
Oct 30 13:56:10.199897 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:10.199862    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c32af363-tn4j2_83342704-e62b-4c3c-b896-033307b1e58c/revoke-keys/0.log"
Oct 30 13:56:10.409937 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:10.409912    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-4z4z5_49d0fb91-0749-4365-9257-c14ba79f85d0/revoke-keys/0.log"
Oct 30 13:56:10.608610 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:10.608580    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-vx8q2_5a6b8940-4819-4258-801d-e38ba52c39f2/revoke-keys/0.log"
Oct 30 13:56:11.022268 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:11.022168    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-4l4tc_7472d739-738d-481d-9845-ff8959593b6a/postgres/0.log"
Oct 30 13:56:12.484507 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:12.484470    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-nlk4n_8714d903-8aa8-4b34-ab60-943c82d814c3/manager/0.log"
Oct 30 13:56:12.699894 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:12.699858    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-n6qxn_a7c27cba-095a-446a-bf7c-acab18ce8bca/manager/0.log"
Oct 30 13:56:13.605433 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:13.605398    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-689fp_ebca0355-cba2-4a9d-8a7c-81036e479ca8/manager/0.log"
Oct 30 13:56:14.432313 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:14.432279    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-dlbtt_1e8d89e6-3301-45d8-8e9d-4629f6bad974/discovery/0.log"
Oct 30 13:56:14.850598 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:14.850575    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cbd4cb4f6-fl8vc_3eb6d5b4-6fbb-4ae8-9359-833f745cf6c2/kube-auth-proxy/0.log"
Oct 30 13:56:15.261919 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:15.261835    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-52t9q_7e9c9db5-5873-40d8-94bb-d827c7698714/istio-proxy/0.log"
Oct 30 13:56:15.470623 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:15.470592    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-7grvt_c16d5364-17da-4fef-993d-9ac8a77f56a5/istio-proxy/0.log"
Oct 30 13:56:15.680208 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:15.680155    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-9cf5j_61006021-7c0a-4e04-bb5a-c8b2079b3a3e/istio-proxy/0.log"
Oct 30 13:56:15.925102 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:15.925068    4544 scope.go:122] "RemoveContainer" containerID="5a8959f3b803d5426b6d6faa9a063de498317e6bf1324648f670c90c8e4ebb01"
Oct 30 13:56:16.107197 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:16.107140    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-fdnx5_bfb6ceaf-36c4-4a7e-8d4d-72efeecef675/istio-proxy/0.log"
Oct 30 13:56:17.109521 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:17.109408    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-8lgfp_097cd592-eba6-4113-9944-3cdbd57400f4/payload-pre-processing/0.log"
Oct 30 13:56:17.315025 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:17.314995    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-wrtq4_7568a93f-5fdb-444f-86e7-b600ab7c7a78/payload-processing/0.log"
Oct 30 13:56:18.118606 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:18.118573    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-bdtq6_4a23bc66-d828-423f-a2a1-e803aef031d1/main/0.log"
Oct 30 13:56:18.317237 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:18.317194    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-bd6w2_c0251641-4f60-4eae-8970-dbc5f652936b/main/0.log"
Oct 30 13:56:18.927131 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:18.927097    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjd88_62b23320-6cd9-4cbb-b58e-cf8de8a931a8/main/0.log"
Oct 30 13:56:22.054402 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:22.054363    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vx8q2"]
Oct 30 13:56:22.066008 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:22.065971    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vx8q2"]
Oct 30 13:56:23.085198 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:23.085161    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a6b8940-4819-4258-801d-e38ba52c39f2" path="/var/lib/kubelet/pods/5a6b8940-4819-4258-801d-e38ba52c39f2/volumes"
Oct 30 13:56:29.005710 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:56:29.005666    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:56:38.045957 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:38.045920    4544 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z4z5"]
Oct 30 13:56:38.057365 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:38.057335    4544 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z4z5"]
Oct 30 13:56:39.085453 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:39.085408    4544 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49d0fb91-0749-4365-9257-c14ba79f85d0" path="/var/lib/kubelet/pods/49d0fb91-0749-4365-9257-c14ba79f85d0/volumes"
Oct 30 13:56:53.886515 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:53.886486    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75d8bff59-8hz8s_9af0033c-76ac-4cc4-852f-3a3c190778bb/manager/0.log"
Oct 30 13:56:54.194164 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:54.194071    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75d8bff59-8hz8s_9af0033c-76ac-4cc4-852f-3a3c190778bb/manager/0.log"
Oct 30 13:56:54.987635 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:54.987604    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-wrtq4_7568a93f-5fdb-444f-86e7-b600ab7c7a78/payload-processing/0.log"
Oct 30 13:56:55.197419 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:56:55.197386    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-wrtq4_7568a93f-5fdb-444f-86e7-b600ab7c7a78/payload-processing/0.log"
Oct 30 13:56:59.005597 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:56:59.005561    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:57:00.663406 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:00.663372    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:57:01.629864 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:01.629829    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:57:01.761036 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:01.761001    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:57:29.007345 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:57:29.007298    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:57:38.055997 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:38.055952    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75d8bff59-8hz8s_9af0033c-76ac-4cc4-852f-3a3c190778bb/manager/0.log"
Oct 30 13:57:38.367003 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:38.366950    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75d8bff59-8hz8s_9af0033c-76ac-4cc4-852f-3a3c190778bb/manager/0.log"
Oct 30 13:57:39.113432 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:39.113402    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-wrtq4_7568a93f-5fdb-444f-86e7-b600ab7c7a78/payload-processing/0.log"
Oct 30 13:57:39.307687 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:39.307654    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-wrtq4_7568a93f-5fdb-444f-86e7-b600ab7c7a78/payload-processing/0.log"
Oct 30 13:57:39.655807 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:39.655758    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:57:39.782615 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:39.782579    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:57:44.290133 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:44.290096    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xd7lj_b9736332-b3de-4fda-8372-880e2196a136/global-pull-secret-syncer/0.log"
Oct 30 13:57:44.918004 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:44.917974    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log"
Oct 30 13:57:45.565108 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:45.565070    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-38.ec2.internal_6d2044c13bd181c94b6dd5acd1654d5c/haproxy/0.log"
Oct 30 13:57:51.655859 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:51.655768    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-nlk4n_8714d903-8aa8-4b34-ab60-943c82d814c3/manager/0.log"
Oct 30 13:57:51.784659 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:51.784624    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-n6qxn_a7c27cba-095a-446a-bf7c-acab18ce8bca/manager/0.log"
Oct 30 13:57:52.411331 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:52.411294    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-689fp_ebca0355-cba2-4a9d-8a7c-81036e479ca8/manager/0.log"
Oct 30 13:57:53.625253 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:53.625198    4544 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cz4kh/perf-node-gather-daemonset-4jsbj"]
Oct 30 13:57:53.625728 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:53.625671    4544 cpu_manager.go:414] "containerMap: removing container" podUID="2b923499-fdc6-4b49-a7b6-309da9441981" containerName="maas-api"
Oct 30 13:57:53.625728 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:53.625685    4544 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b923499-fdc6-4b49-a7b6-309da9441981" containerName="maas-api"
Oct 30 13:57:53.625823 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:53.625755    4544 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b923499-fdc6-4b49-a7b6-309da9441981" containerName="maas-api"
Oct 30 13:57:53.628903 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:53.628884    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-4jsbj"
Oct 30 13:57:53.656600 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:53.656566    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cz4kh/perf-node-gather-daemonset-4jsbj"]
Oct 30 13:57:53.728498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:53.728442    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cz4kh\"/\"kube-root-ca.crt\""
Oct 30 13:57:53.728498 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:53.728488    4544 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cz4kh\"/\"openshift-service-ca.crt\""
Oct 30 13:57:53.739894 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:53.739862    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bf37429e-10ee-48b5-bc12-6334f12a69d6-podres\") pod \"perf-node-gather-daemonset-4jsbj\" (UID: \"bf37429e-10ee-48b5-bc12-6334f12a69d6\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-4jsbj"
Oct 30 13:57:53.740082 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:53.739986    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nmdh\" (UniqueName: \"kubernetes.io/projected/bf37429e-10ee-48b5-bc12-6334f12a69d6-kube-api-access-2nmdh\") pod \"perf-node-gather-daemonset-4jsbj\" (UID: \"bf37429e-10ee-48b5-bc12-6334f12a69d6\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-4jsbj"
Oct 30 13:57:53.740082 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:53.740025    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bf37429e-10ee-48b5-bc12-6334f12a69d6-proc\") pod \"perf-node-gather-daemonset-4jsbj\" (UID: \"bf37429e-10ee-48b5-bc12-6334f12a69d6\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-4jsbj"
Oct 30 13:57:53.740082 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:53.740048    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf37429e-10ee-48b5-bc12-6334f12a69d6-lib-modules\") pod \"perf-node-gather-daemonset-4jsbj\" (UID: \"bf37429e-10ee-48b5-bc12-6334f12a69d6\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-4jsbj"
Oct 30 13:57:53.740082 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:53.740070    4544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf37429e-10ee-48b5-bc12-6334f12a69d6-sys\") pod \"perf-node-gather-daemonset-4jsbj\" (UID: \"bf37429e-10ee-48b5-bc12-6334f12a69d6\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-4jsbj"
Oct 30 13:57:53.740435 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:53.740419    4544 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cz4kh\"/\"default-dockercfg-2bzw5\""
Oct 30 13:57:53.840881 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:53.840839    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nmdh\" (UniqueName: \"kubernetes.io/projected/bf37429e-10ee-48b5-bc12-6334f12a69d6-kube-api-access-2nmdh\") pod \"perf-node-gather-daemonset-4jsbj\" (UID: \"bf37429e-10ee-48b5-bc12-6334f12a69d6\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-4jsbj"
Oct 30 13:57:53.841072 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:53.840888    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bf37429e-10ee-48b5-bc12-6334f12a69d6-proc\") pod \"perf-node-gather-daemonset-4jsbj\" (UID: \"bf37429e-10ee-48b5-bc12-6334f12a69d6\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-4jsbj"
Oct 30 13:57:53.841072 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:53.840921    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf37429e-10ee-48b5-bc12-6334f12a69d6-lib-modules\") pod \"perf-node-gather-daemonset-4jsbj\" (UID: \"bf37429e-10ee-48b5-bc12-6334f12a69d6\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-4jsbj"
Oct 30 13:57:53.841072 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:53.840957    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf37429e-10ee-48b5-bc12-6334f12a69d6-sys\") pod \"perf-node-gather-daemonset-4jsbj\" (UID: \"bf37429e-10ee-48b5-bc12-6334f12a69d6\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-4jsbj"
Oct 30 13:57:53.841072 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:53.841013    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bf37429e-10ee-48b5-bc12-6334f12a69d6-proc\") pod \"perf-node-gather-daemonset-4jsbj\" (UID: \"bf37429e-10ee-48b5-bc12-6334f12a69d6\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-4jsbj"
Oct 30 13:57:53.841072 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:53.841029    4544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bf37429e-10ee-48b5-bc12-6334f12a69d6-podres\") pod \"perf-node-gather-daemonset-4jsbj\" (UID: \"bf37429e-10ee-48b5-bc12-6334f12a69d6\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-4jsbj"
Oct 30 13:57:53.841307 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:53.841076    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf37429e-10ee-48b5-bc12-6334f12a69d6-lib-modules\") pod \"perf-node-gather-daemonset-4jsbj\" (UID: \"bf37429e-10ee-48b5-bc12-6334f12a69d6\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-4jsbj"
Oct 30 13:57:53.841307 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:53.841094    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf37429e-10ee-48b5-bc12-6334f12a69d6-sys\") pod \"perf-node-gather-daemonset-4jsbj\" (UID: \"bf37429e-10ee-48b5-bc12-6334f12a69d6\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-4jsbj"
Oct 30 13:57:53.841307 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:53.841101    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bf37429e-10ee-48b5-bc12-6334f12a69d6-podres\") pod \"perf-node-gather-daemonset-4jsbj\" (UID: \"bf37429e-10ee-48b5-bc12-6334f12a69d6\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-4jsbj"
Oct 30 13:57:53.864231 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:53.864183    4544 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nmdh\" (UniqueName: \"kubernetes.io/projected/bf37429e-10ee-48b5-bc12-6334f12a69d6-kube-api-access-2nmdh\") pod \"perf-node-gather-daemonset-4jsbj\" (UID: \"bf37429e-10ee-48b5-bc12-6334f12a69d6\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-4jsbj"
Oct 30 13:57:53.944383 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:53.944288    4544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-4jsbj"
Oct 30 13:57:54.245540 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:54.245450    4544 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cz4kh/perf-node-gather-daemonset-4jsbj"]
Oct 30 13:57:54.248836 ip-10-0-143-38 kubenswrapper[4544]: W1030 13:57:54.248799    4544 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbf37429e_10ee_48b5_bc12_6334f12a69d6.slice/crio-a994e8f57a409027fcb4d8eab8c743904e8c94d2d558646dc2666acaa14425f4 WatchSource:0}: Error finding container a994e8f57a409027fcb4d8eab8c743904e8c94d2d558646dc2666acaa14425f4: Status 404 returned error can't find the container with id a994e8f57a409027fcb4d8eab8c743904e8c94d2d558646dc2666acaa14425f4
Oct 30 13:57:55.259293 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:55.259257    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-4jsbj" event={"ID":"bf37429e-10ee-48b5-bc12-6334f12a69d6","Type":"ContainerStarted","Data":"daf1132841bf76a205fa85152419107fc81abba2012840ea51bd9424950e24a8"}
Oct 30 13:57:55.259846 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:55.259315    4544 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-4jsbj" event={"ID":"bf37429e-10ee-48b5-bc12-6334f12a69d6","Type":"ContainerStarted","Data":"a994e8f57a409027fcb4d8eab8c743904e8c94d2d558646dc2666acaa14425f4"}
Oct 30 13:57:55.259846 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:55.259431    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-4jsbj"
Oct 30 13:57:55.302460 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:55.302393    4544 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-4jsbj" podStartSLOduration=2.302373548 podStartE2EDuration="2.302373548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:57:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:57:55.300635108 +0000 UTC m=+4370.347769062" watchObservedRunningTime="2026-10-30 13:57:55.302373548 +0000 UTC m=+4370.349507502"
Oct 30 13:57:55.705731 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:55.705698    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kg8q6_e9723bc6-9b25-43ec-b210-7e231e8bc8af/node-exporter/0.log"
Oct 30 13:57:55.843778 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:55.843744    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kg8q6_e9723bc6-9b25-43ec-b210-7e231e8bc8af/kube-rbac-proxy/0.log"
Oct 30 13:57:55.960370 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:55.960279    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kg8q6_e9723bc6-9b25-43ec-b210-7e231e8bc8af/init-textfile/0.log"
Oct 30 13:57:57.609841 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:57.609809    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-2w7nn_b3b4295c-1ef2-400d-a7e6-f2cc60f39cb6/prometheus-operator/0.log"
Oct 30 13:57:57.723620 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:57.723587    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-2w7nn_b3b4295c-1ef2-400d-a7e6-f2cc60f39cb6/kube-rbac-proxy/0.log"
Oct 30 13:57:57.840079 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:57:57.840051    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-qvzmt_0f5859be-f23e-4aa3-8f6b-df7c77f1caa4/prometheus-operator-admission-webhook/0.log"
Oct 30 13:57:59.005922 ip-10-0-143-38 kubenswrapper[4544]: E1030 13:57:59.005882    4544 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q6dlw" containerName="extractor"
Oct 30 13:58:00.766618 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:58:00.766581    4544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sprl_abe27783-da2a-46b9-81f6-cf551787b1bc/konnectivity-agent/0.log" limit=1024
Oct 30 13:58:01.274762 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:58:01.274734    4544 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-4jsbj"
Oct 30 13:58:02.290935 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:58:02.290902    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-69c8c76d5f-247nt_b44fb3f6-1691-4588-8a15-d0a0afe25dd1/console/0.log"
Oct 30 13:58:02.433785 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:58:02.433747    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-km7rm_28d577e6-7b12-40a1-98b4-4bc41023ec17/download-server/0.log"
Oct 30 13:58:06.270936 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:58:06.270903    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xmn5t_a36ceaac-5967-4321-802e-42f4c4b37032/dns/0.log"
Oct 30 13:58:06.393532 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:58:06.393500    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xmn5t_a36ceaac-5967-4321-802e-42f4c4b37032/kube-rbac-proxy/0.log"
Oct 30 13:58:06.761328 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:58:06.761292    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v68vl_7b607752-d51b-4781-a57f-99e95883d236/dns-node-resolver/0.log"
Oct 30 13:58:07.857975 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:58:07.857937    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5cc44fbdc6-z6lmh_4df72e77-b759-4a2a-b9c1-808f30a3a1ae/registry/0.log"
Oct 30 13:58:08.199280 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:58:08.199158    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v9khv_2eef9c4d-7079-47ad-9fad-c89c16ac69b4/node-ca/0.log"
Oct 30 13:58:08.384241 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:58:08.384187    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-dlbtt_1e8d89e6-3301-45d8-8e9d-4629f6bad974/discovery/0.log"
Oct 30 13:58:08.475814 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:58:08.475722    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cbd4cb4f6-fl8vc_3eb6d5b4-6fbb-4ae8-9359-833f745cf6c2/kube-auth-proxy/0.log"
Oct 30 13:58:08.602426 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:58:08.602398    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-52t9q_7e9c9db5-5873-40d8-94bb-d827c7698714/istio-proxy/0.log"
Oct 30 13:58:08.664394 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:58:08.664355    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-7grvt_c16d5364-17da-4fef-993d-9ac8a77f56a5/istio-proxy/0.log"
Oct 30 13:58:08.735160 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:58:08.735068    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-9cf5j_61006021-7c0a-4e04-bb5a-c8b2079b3a3e/istio-proxy/0.log"
Oct 30 13:58:08.834056 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:58:08.834019    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-fdnx5_bfb6ceaf-36c4-4a7e-8d4d-72efeecef675/istio-proxy/0.log"
Oct 30 13:58:09.139396 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:58:09.139365    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-8lgfp_097cd592-eba6-4113-9944-3cdbd57400f4/payload-pre-processing/0.log"
Oct 30 13:58:09.201925 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:58:09.201896    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-wrtq4_7568a93f-5fdb-444f-86e7-b600ab7c7a78/payload-processing/0.log"
Oct 30 13:58:11.881492 ip-10-0-143-38 kubenswrapper[4544]: I1030 13:58:11.881454    4544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-dlbtt_1e8d89e6-3301-45d8-8e9d-4629f6bad974/discovery/0.log"